search

IDR / idr.openmicroscopy.org

Source for the IDR static website.
https://idr.openmicroscopy.org/about
Creative Commons Attribution 4.0 International
4 stars 16 forks source link

Irregular Download Path Usage #158

Closed roshankern closed 2 years ago

roshankern commented 2 years ago

I trying to download just the first plate for idr0013-screenA and am struggling to use the aspera download client to complete this partial download. The IDR download tutoral mentions to remove the leading /uod/idr/filesets/<idrNNN>-<author>-<description>/ when submitting a partial download command. However, the download paths for idr0013 do not follow the /uod/idr/filesets/<idrNNN>-<author>-<description>/ format.

For example, plate LT0001_02 lists its relative path as ../screens/LT0001_02--ex2005_11_16--sp2005_02_17--tt17--c3.screen. Attempting to run ascp -TQ -l40m -P 33001 -i "path/to/asperaweb_id_dsa.openssh" idr0013@fasp.ebi.ac.uk:screens/LT0001_02--ex2005_11_16--sp2005_02_17--tt17--c3.screen /tmp/data/ returns Session Stop (Error: Server aborted session: No such file or directory).

How should I use this relative path to complete a partial download? Thanks!

roshankern commented 2 years ago

CC @gwaygenomics to keep you in the loop

gwaybio commented 2 years ago

Thanks Roshan! Indeed, it would be helpful to know if there are any more specific instructions on how to troubleshoot misconstructed download paths

pwalczysko commented 2 years ago

@roshankern Thank you for your comment. The study idr0013 uses a screen format. This means that the .screen files are just textfiles which are read at the moment of importing the Plates into OMERO (and thus IDR). The .screen textfiles are pointing to the image files and these can in turn be downloaded by aspera. Your workflow to get these image files could be:

  1. Study the .screen file https://github.com/IDR/idr-metadata/blob/cb649f6e65b5cf1690eb4f29804eae234fc2dece/idr0013-neumann-mitocheck/screens/LT0001_02--ex2005_11_16--sp2005_02_17--tt17--c3.screen which you attempted to download
  2. Take the line https://github.com/IDR/idr-metadata/blob/cb649f6e65b5cf1690eb4f29804eae234fc2dece/idr0013-neumann-mitocheck/screens/LT0001_02--ex2005_11_16--sp2005_02_17--tt17--c3.screen#L10
  3. Copy the path to the image file, which is /uod/idr/filesets/idr0013-neumann-mitocheck/20150916-mitocheck-analysis/mitocheck/LT0001_02--ex2005_11_16--sp2005_02_17--tt17--c3/hdf5/00001_01.ch5 from that line.
  4. Adjust the path you just copied to remove the /uod/idr/filesets/idrxxxx-xxxx-xxxx motif.
  5. Insert the adjusted path to the image file into your aspera command, which gives you

ascp -TQ -l40m -P 33001 -i "path/to/asperaweb_id_dsa.openssh" idr0013@fasp.ebi.ac.uk:20150916-mitocheck-analysis/mitocheck/LT0001_02--ex2005_11_16--sp2005_02_17--tt17--c3/hdf5/00001_01.ch5
  1. You can write a script which automates the workflow 1. - 6. above, parsing the .screen file(s) and creating the aspera commands to download more image files.

We will fix the documentation on the IDR page https://idr.openmicroscopy.org/about/download.html as it is oversimplistic for the case of the particular study download you were attempting (the doc is correct for most of the studies though). I have opened an issue for that https://github.com/IDR/idr.openmicroscopy.org/issues/159 Sorry about that. Petr, OME Team

roshankern commented 2 years ago

Works perfectly, thank you for the prompt response!

gwaybio commented 2 years ago

Wonderful! Agreed - Thanks @pwalczysko for the very prompt reply :)

sbesson commented 2 years ago

Thanks all, closing this in favor of #159 which captures the remain documentation work