ENH: add get-metadata action

bokulich-lab / q2-fondue

Functions for reproducibly Obtaining and Normalizing Data re-Used from Elsewhere

BSD 3-Clause "New" or "Revised" License

20 stars 6 forks source link

ENH: add get-metadata action #5

Closed misialq closed 3 years ago

misialq commented 3 years ago

Introduces get-metadata action for fetching SRA metadata using Entrezpy.

accepts a list of study ids for which metadata will be fetched
supports a multi-threaded mode
outputs a SRAMetadata artifact containing a TSV file with aggregated metadata

misialq commented 3 years ago

@LenaFloerl would you be open to testing this action with some of your studies? Let me know - I'd be happy to provide you with more info :)

misialq commented 3 years ago

Thanks for trying it out @adamovanja! That's odd, when I'm trying that everything just hangs, i.e. it hangs whenever you give it a random, nonsensical ID. How long did you wait for that output?

adamovanja commented 3 years ago

This accession number is not really random - it actually exists, see here in ena browser. Hence, I was wondering why it did not work. The output appears shortly after input with the flag --verbose (2 min) but then the remaining execution hangs.

misialq commented 3 years ago

On the "random" accession: good point - I assumed it's "fake" because when I tried with a proper fake ID the result was the same (it was not responding). In this case though, I'll investigate a bit more, thanks! For now, my guess is that the response it gets for that ID must just be somehow different than it expects... 😨

misialq commented 3 years ago

All should work now. Some highlights of the latest update:

to circumvent the issue of the API not responding when given a random, non-existent accession ID I introduced ID validation step before fetching the actual metadata; it uses a new ESearch client to validate the IDs and aborts if some IDs have issues (displays a msg to the user with more details)
in the case of a run set containing more runs than the requested one, only data for that single run will be extracted
added some more tests
added docstrings in most relevant places