SPI-Birds Network and Database: Pipelines 

               

 Welcome to the pipeline repository for the SPI-Birds Network and Database. Follow the links above to visit our website, contact us via e-mail or Twitter. This README contains all the details required to work with the pipelines, including workflow guidelines for developers. 

SPI-Birds pipeline: Introduction (for the general user)

Welcome to the SPI-Birds pipeline package. This section of the README will give you an introduction on how to load the package, how to find out details about each pipeline, and how to use the package for creating bird data following the SPI-Birds community data standard and generating standard quality checks.

remotes::install_github("SPI-Birds/pipelines")
library(pipelines)

This will install all pipelines and quality check code on your computer and attach our pipeline package into your session of R. Individual pipelines are build as separate functions for each data owner (where one data owner can administer multiple populations). Each function is given the name format_X() where X is the letter code for the data owner. The codes for different data owners and corresponding populations are described in the SPI-Birds standard protocol. Note in cases where a data owner administers one population, the unique 3 letter population ID code and the data owner code are identical.

To process each set of primary data into the structure described in the SPI-Birds standard protocol it is often necessary to make assumptions about how each variable is interpreted. All assumptions made during the pipeline process are described in the help documentation for a given function. This can be accessed using the ? in R. For example, to read about the assumptions made when processing data from the NIOO, you can use the code:

?format_NIOO

Run the pipelines for yourself

Each set of primary data is in a slightly different format. Therefore, to run all pipelines successfully, your system will require additional software and drivers (in addition to R).

Setup your computer to run pipelines

Pipelines for some populations require additional software and drivers. Setup instructions describe how to install the required software for both a Windows 10 64-bit operating system and Mac OSX. The setup procedure should be similar for other Windows 64-bit systems. If you are unsure which version of Windows is running on your computer, check 'System Type' in 'System Information'. To run the pipelines for all populations a users system must have:

• Microsoft Access Driver (/.mdb, /.accdb) (Windows only)
• Python 3
• Python libraries pandas and pypxlib

Note Users running Mac OSX will not be able to run pipelines with primary data stored in Microsoft Access format without purchasing paid drivers.

---

Windows 10 64-bit

Microsoft Access Driver

Firstly, you must check that you are running a 64-bit version of R. Open an R session and see whether you have 64-bit or 32-bit installation.

If you do not have a 64-bit version you will need to install one here.

Once you have a 64-bit version of R, search for 'ODBC' in the Windows taskbar. There will be two version (32-bit and 64-bit) select the 64-bit version. This will open the 'ODBC Data Source Administrator (64-bit)' window.

In the new window check for 'Microsoft Access Driver'. If this already exists you can skip to the Python stage.

If 'Microsoft Access Driver' does not exist click 'Add' to install a new driver on your system.

Select 'Microsoft Access Driver (/.mdb, /.accdb)' and click finish.

If 'Microsoft Access Driver (/.mdb, /.accdb)' does not appear, you will need to download the 64-bit driver here

In the next window, you must add a 'Data Source Name'. Everything else can be left blank.

Check if this driver is installed and recognised by R using the function odbcListDrivers() from the odbc package. Note that you will need to open a new session of R before the driver will appear.

Python 3

To install Python, we recommend using the Anaconda distribution. Make sure to download the 3.X version of Python. The Anaconda distribution comes with some libraries (including pandas) pre-loaded.

Once installed, open the 'Anaconda prompt' and type:

pip install pypxlib

This will install the pypxlib library on your system.

Restart your computer before running the pipelines.

MikTex

To generate the pdf quality check report on Windows you will need to have installed MikTex. If MikTex is not installed, only the html version of the quality check report can be created.

An alternative LaTeX distribution that works well in R is TinyTeX.

Mac

Microsoft Access Driver

At present, no free Microsoft Access Driver is available for Mac.

As a consequence, the pipelines package currently does not run pipelines requiring a Microsoft Access Driver on Mac OSX (the affected pipelines are skipped and a information message displayed when attempting to run on Mac).

Python 3 for Mac

The following notes detail how to set up the python environment on MacOS, including necessary libraries:

• Install Anaconda 3.X (this was last tested with 3.8)

• Check your default python version by opening terminal and typing:python3 --version(This should return Python 3.X.Y)

• Check that pip is available by typing pip3 --version in the terminal

• Update pip by typing python3 -m pip install --user --upgrade pip in the terminal (You have to use the --user argument as permission may be denied otherwise)

• Open RStudio and load the reticulate package: library(reticulate)

• Check which version of python reticulate is linked to: py_config() (Required python libraries need to be installed into this virtual environment)

• Install pandas library from within R: py_install("pandas")

• Install pypxlib library from within R: py_install("pypxlib", pip = TRUE) (Since this is an external library hosted on GitHub, you need to specify installation via pip)

• Check that both libraries are now available:

  reticulate::py_module_available("pandas")
  reticulate::py_module_available("pypxlib")

  (Both commands should return TRUE)

With this setup, python should be good to go for extracting paradox databases. (Note that when you install Anaconda, the r-reticulate environment should already be present. If that is not the case, you may have to first generate the environment and link it to RStudio).

Pdf compilation on Mac

At present, the pipelines package does not create pdf outputs when run on a Mac. This is work in progress and will be changed in the future.

Troubleshooting

If you are still unable to run the pipelines following these setup instructions try these troubleshooting techniques:

• Restart your computer before running pipelines to ensure R recognises the newly installed software and drivers.

• If R on Windows does not recognise Python's pandas module, try installing it using reticulate::py_install("pandas").

• Download the newest version of R here.

• Update all R packages.

format_NIOO(output_type = "R")

NIOO_NetherlandsInstituteOfEcology_Netherlands

The naming convention of each of these files is described below.

The .standard_format folder

In addition to folders for each data owner, the data folder contains the most recent output of all pipelines in the standard format, including an archiving folder. When a data request is made, this version of the standard format can be filtered to meet a given data request (see Data requests below). This is more efficient than re-running pipelines for each data request.

ABC_PrimaryData_Greattit.csv
ABC_PrimaryData_Bluetit.csv

ABC_FieldProtocol.docx

Code of all pipelines is stored in the /R folder of the pipelines repository. Every pipeline file should follow the naming convention format_X.R, where X is the data owner code. More details on the structure of pipeline code can be found below.

Note Commits should ideally be distinct blocks of changes with a concise header and detailed description. See some commit best practices here.

• To make commits more easily readable/searchable you should include an emoji at the start of each commit message following these dev guidelines. For example, if you find some typos in the code your commit would be ':pencil2: Fix typo in format_XXX() function'.

• When you have finished working for a day, push your branch to the remote (git push -u origin new_branch_name the first time; git push afterwards).

Build the pipeline

• In your new branch, create the new file format_X.R in the /R folder, where X is the data owner code.

• This file should contain one parent function (format_X()) and at least 4 internal functions for each of the four tables in the standard format:

  • create_brood_X()
  • create_capture_X()
  • create_individual_X()
  • create_location_X()

• format_X() should always take 4 standard arguments:

  • path: The location of the folder where primary data are stored. Can be left blank and R will prompt you to find the folder.
  • PopID: The population code(s) for the populations where you want to run pipelines. Relevant for data owners that administer multiple populations.
  • Species: The species code(s) for the species you want to use (e.g. PARMAJ for great tit). See the SPI-Birds standard protocol for all species codes.
  • output_type: Should the data be output in R or as separate .csv files

• These arguments are all documented under pipeline_params in the zzz.R file within /R.

• Every function should be documented using roxygen2. The 'Details' section of the documentation should be used to record any decisions that were made during the pipeline construction that may be relevant for the data owner, users, or other developers.

• Once a pipeline is finished, add information about the pipeline to pop_codes.csv and pop_species_combos.csv in the /inst/extdata folder.

• If your pipeline works with a new species, also include this species in species_codes.csv in the /inst/extdata folder.

Note: We recommend you look at other pipelines as a guide.

Create unit tests

Every pipeline should have a set of unit tests in the /test/testthat folder using the testthat package.

• The unit testing code for each pipeline should be stored in a file test-XXX.R, where XXX is the data owner code. The file should start with an option to skip if the data path is missing. It should then run the corresponding format_XXX() function for the pipeline, followed by the required tests. Unit tests should ensure that primary data has been properly converted to the standard format. This will usually involve comparing the values for a range of different individuals in the standard format (e.g. different species, different sex) to those that would be expected from the primary data. In other words, these tests require some manual inspection of the primary data to determine the correct output expected for each individual.

• Each pipeline should undergo five sets of unit tests:

  • Test standard format structure. Have the four tables Brood_data, Capture_data, Individual_data, and Location_data been created.
  • Test brood data.
  • Test capture data.
  • Test individual data.
  • Test location data.

• See examples from completed pipelines to better understand the structure of unit testing code.

test_pipeline()

Once you have finished the pipeline and written relevant unit tests you should make sure these tests pass.

• Firstly, run unit tests just for your new pipeline. In the console type test_pipeline(filter = "XXX"), where XXX is the data owner code of your new pipeline.

• Once your pipeline passes the relevant tests next run the unit tests for all existing pipelines by removing the filter argument: test_pipeline(). This can be time consuming and a bit annoying, but it is important to regularly test all pipelines in case old code has broken due to e.g. package updates.

• If one or more tests fail you can fix them and trouble shoot using the filter argument as shown above. To test more than one pipeline simultaneously use test_pipeline(filter = "XXX|YYY"), where XXX and YYY are two different data owner codes.

devtools::check()

Once your branch is passing all unit tests you should next check the package structure. This will more broadly check things like the documentation, check for unreadable characters, ensure all the code can be loaded. This will not re-run the pipeline unit tests, which are skipped at this stage.

• You can check the code using devtools::check() or Ctrl/Cmd + Shift + E to run the checks in the build window.

• Currently, the output of devtools::check() should include 2 known notes:

Imports includes 27 non-default packages.
Importing from so many packages makes the package vulnerable to any of
them becoming unavailable.  Move as many as possible to Suggests and
use conditionally.

Package dependencies are discussed in more detail below.

• Any other ERRORS, WARNINGS, or NOTES must be fixed before continuing! Pull requests will not be accepted otherwise.

Tips for passing devtools::check()

"no visible binding for global variable"

This will often occur when working with dplyr code. All references to columns in a data frame should be prefixed by .data$.

"no visible global function"

All functions except those in the base package should have the package namespace explicitly stated (e.g. stats::na.omit).

"Undocumented arguments in documentation object 'XXX'"

The function XXX includes an argument that is not documented with @param in the roxygen2 documentation code. Check for spelling mistakes!

"Documented arguments not in \usage in documentation object 'XXX'"

The function XXX includes documentation for an argument in @param in the roxygen2 documentation code that does not appear in the function. Check for spelling mistakes!

"Found the following file with non-ASCII characters"

Packages can only include ASCII characters. You can check the character types being used in a line of code with stringi::stri_enc_mark(). For example:

#Will return ASCII
stringi::stri_enc_mark("ABC")

#Will return UTF-8
stringi::stri_enc_mark("是")

Watch out for cases where slanted quotation marks are used (‘’) instead of straight ones ('')! Slanted quotation marks can often be introduced when copying text from outside R, but they are NOT ASCII.

If a non-ASCII character must be used, it can be encoded with unicode \uxxxx.

Create a pull request

Once your pipeline is stable and passes all tests and checks it should be reviewed by other developers.

• Push your finished pipeline to the remote (git push).

• Visit the pipelines repository and open a pull request.

• Request a reviewer. It is also good to let the person know directly so they don't miss it.

Note One key aspect of the code review should also be to test the pipelines on both Mac OSX and Windows.

Note The pull request should not be merged until after the data owner confirmation.

• Once the pipeline is stable it can be updated to 'finished' in the Google Sheet.

Data owner confirmation

The code review should ensure that there are no major bug or oversights. At this point, we can contact the data owner to discuss the pipeline.

• Explain all decisions that were made in the pipeline.

• Confirm any columns/data that were unclear/uncertain.

• Ask about any other data that appear to be missing (e.g. capture data, nest box coordinates).

• At this point, some changes may be needed to incorporate data owner input. If changes are made to the pipeline code it's important that unit tests and checks are run on the code again.

• Record data owner approval in the Google Sheet.

Merge and quality check

• Once a pipeline has approval from both the data owner and code reviewer the pull request can be merged.

• At this point the working branch can be deleted from the remote and local.

Note Remember to pull the newest version of the master branch at this point, it will include the new pipeline.

• Run quality_check() on the standard format output from the pipeline. Send the quality check report and the standard format data to the data owner to help them improve data integrity. See more details on the quality check below.

• Contact Antica to update the populations as 'available' on the website.

Update and archive

• Every time a new pipeline is finished (or an old pipeline updated) we should update and archive the .standard_format folder on the N drive. More about archiving below.