Make selective downloading more robust

[claraqin]

Improvements should be made to the sequence data downloading functions in utils.R to make them more robust. Currently they assume a standard file naming structure, which means that they may not be robust to minute changes in naming conventions.

Suggestions from @lstanish :

In my experience the best way to ensure you are filtering for the correct sequence data is to start by downloading the metadata and doing some table joining and filtering. Here’s some example code using neonUtilities that combines the respective sequencing data table (16S or ITS) with the raw data files table. What you end up with is a data.frame containing the rawDataFIleNames for just the 16S or ITS data

library(neonUtilities)
library(plyr)
library(dplyr)

mmgL1 <- loadByProduct('DP1.10108.001', package = 'expanded', check.size = F) # output is a list of each metadata file

# extract lists into data. frames
seq16S <- mmgL1$mmg_soilMarkerGeneSequencing_16S
seqITS <- mmgL1$mmg_soilMarkerGeneSequencing_ITS
raw <- mmgL1$mmg_soilRawDataFiles

# double check that strings are of class character and not factor/logical

# Join 16S metadata
raw16S <- raw[-grep("ITS", raw$rawDataFileName), ]
joined16S <- left_join(raw16S, seq16S, by=c('dnaSampleID', 'sequencerRunID', 'internalLabID'))
joined16S <- joined16S[!is.na(joined16S$uid.y), ]

# Join ITS metadata
rawITS <- raw[-grep("16S", raw$rawDataFileName), ]
joinedITS <- left_join(rawITS, seqITS, by=c('dnaSampleID', 'sequencerRunID', 'internalLabID'))
joinedITS <- joinedITS[!is.na(joinedITS$uid.y), ]

From here you can subset the data to include only the run or samples of interest.

[lstanish]

@claraqin Questions on modifying the metadata downloading code:

• should we design the code to work for aquatic data products? It currently only works for soils
• there are a bunch of preset values being passed to the metadata download function: what is the format of these preset values and which ones are required?
• do you want to retain the functionality to save the metadata locally, and if so, which tables do you want to be retained?

[claraqin]

Sorry for the slow response.

• No need to design the code to work for the aquatic data products
• The preset_values can be found in params.R. You should only need to specify the ones that are mentioned in the arguments of downloadRawSequenceData() and downloadSequenceMetadata()
• Ultimately it would be better if we didn't have to save the metadata locally, and if that could instead be given as an option in the arguments of the function. I saved the metadata locally because (1) it was the only way I knew how to pass information to other functions for downloading the raw sequence data, and (2) it would be used later for downloading soil data. If you could make the "save to local" functionality optional, then I could redesign (2) so that it doesn't rely on locally saved metadata either.

[lstanish]

@claraqin I created a new version of the metadata downloading function, called downloadSequenceMetadataRev(). Did a bit of testing, but would be great for you and/or others in the group to also test and provide feedback.

Will shift to working on a revised sequence data downloading function that uses neonUtilities.

[lstanish]

@claraqin Thanks for the code additions, the new functionality is great! Here's a list of the most recent updates:

• added in DNA extraction metadata to output, which will make it easier to join with the soil metadata downstream
• tweaked the code to validate sequencingRuns to avoid unexpected behavior
• users can now download 16S and ITS together by setting targetGene='all'; set this as default I did a bit of testing with a small dataset, we should test on a larger dataset before finalizing

[lstanish]

@claraqin one other question: at some point do you want to clean up the older versions of functions in utils.R? Would be good to update the name of this function so that it isn't a 'rev' anymore. No rush, just wanted to write it down so we don't forget!

[claraqin]

Hi Lee,

Thanks for bringing this up! I'll clean up the older versions in my next commit. And sorry for the slow response – I don't know how to receive notifications for Issue thread replies. I'm making a note to figure that out too.

Clara

[claraqin]

I think this has been resolved as of the most recent commit, which gives downloadSequenceMetadataRev the ability to handle tarballs.