search

sdss / sdss_access

Product to dynamically build and download filepaths to SDSS data products
http://sdss-access.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
14 stars 4 forks source link

sdss_access isn't always downloading data #5

Closed havok2063 closed 3 years ago

havok2063 commented 5 years ago

Description sdss_access sometimes downloads the data and sometimes it does not. I am trying to download all files for 7443-12703 for DR15. It says everything finished ok but it only created the empty folder structure. No files are present.

I checked the log files in the /tmp/sdss_access/ directory. There are no errors and the logs indicate everything is fine.

Here is an example task

 {u'destination': u'/Users/Brian/Work/sdss/sas/dr15/manga/spectro/redux/v2_4_3/7443/stack/manga-7443-12701-LOGCUBE.fits.gz',
  u'exists': None,
  u'location': u'manga/spectro/redux/v2_4_3/7443/stack/manga-7443-12701-LOGCUBE.fits.gz',
  u'source': u'rsync://data.sdss.org/dr15/manga/spectro/redux/v2_4_3/7443/stack/manga-7443-12701-LOGCUBE.fits.gz'}

Expected Behaviour The files are downloaded

Steps To Recreate

rsync = RsyncAccess(public=True, release='dr15')
rsync.remote()
rsync.add('mangacube', plate=7443, ifu=12701, drpver='v2_4_3')
rsync.add('mangarss', plate=7443, ifu=12701, drpver='v2_4_3')
rsync.add('mangadap5', plate=7443, ifu=12701, drpver='v2_4_3', dapver='2.2.1', mode='*', daptype='*GAU-MILESHC')
rsync.add('mangaimage', plate=7443, ifu=12701, drpver='v2_4_3', dir3d='stack')
rsync.set_stream()
rsync.commit()
SDSS_ACCESS> syncing... please wait
SDSS_ACCESS> Done!

Additional Context I previously ran this same set of code for 8485-1901 and it ran fine.

havok2063 commented 5 years ago

It looks like it eventually finished but it took away longer than when the code reported it was done. Almost 20 minutes. This time an error was displayed in one of the stream logs. And I noticed that the other logs eventually displayed the sent stream statistics. So maybe this means that the code isn't exactly waiting until the files are downloaded before reporting DONE. 

rsync error: timeout in data send/receive (code 30) at io.c(200) [sender=3.0.6]
rsync: connection unexpectedly closed (47763908 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [receiver=3.1.2]
rsync: connection unexpectedly closed (368 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [generator=3.1.2]
joelbrownstein commented 5 years ago

The multiple streams are parallel processes, so it is supposed to wait for the last stream to complete. I will try to understand how it could be broken (i.e sometimes it waits but not always?).

havok2063 commented 3 years ago

Not sure this is relevant anymore. Closing this for now. If it comes up again, we can open a new issue.