pubmed/data-extraction/extraction Download Updates

[lbeckman314]

This issue has two proposed additions to the wget approach as outlined in the pubmed/data-extraction/extraction README.

---

1) Add a Zsh compatible wget command to the download instructions.

Current Version (Zsh Error).

wget --cut-dirs 3 -r ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/*.xml.gz
zsh: no matches found: ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/*.xml.gz

Current Version (Zsh compatible).

wget --cut-dirs 3 -r 'ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/*.xml.gz'
# Success!

---

2) Add a parallel wget command to the download instructions.

Parallel Version

parallel --bar 'wget --cut-dirs 3 -r ftp://ftp.ncbi.nih.gov/pubmed/baseline/pubmed19n$(printf "%04d" {1}).xml.gz' ::: $(seq 972)

Parallel Version (Zsh compatible)

parallel --bar 'wget --cut-dirs 3 -r -A pubmed19n$(printf "%04d" {1}).xml.gz ftp://ftp.ncbi.nih.gov/pubmed/baseline/' ::: $(seq 972)

Cons:

• Jerky output due to parallel operations.'
• Have to keep track of number of files (currently 972).

Pros:

• Nice progress bar (with estimated time remaining!) due to the --bar flag passed to parallel.
• Faster? No benchmarks have been run on this, so not definitive. But requests to the ncbi host can be made without having to wait for each individual operation to complete.

---

Let me know if you have any suggestions. If these aren't suited for adoption, no worries!

Thanks for making a great resource!

liam