search

nansencenter / django-geo-spaas

GeoDjango apps for satellite data management in Geo-Scientific Platform as a Service
GNU General Public License v3.0
20 stars 6 forks source link

We should implement a thredds crawler ingestor command in geospaas.nansat_ingestor #40

Closed mortenwh closed 5 years ago

mortenwh commented 5 years ago

Tested on Sentinel-2 data from the Norwegian ground segment, e.g.:

./manage.py ingest_thredds_crawl http://nbstds.met.no/thredds/catalog/NBS/S2B/2018/07/catalog.html --date 2018/07/17

or

./manage.py ingest_thredds_crawl http://nbstds.met.no/thredds/catalog/NBS/S2B/2018/07/17/catalog.html --filename S2B_MSIL1C_20180717T095029_N0206_R079_T34VFM_20180717T115424.nc

The latter is much faster...

akorosov commented 5 years ago 
./manage.py ingest_thredds_crawl http://nbstds.met.no/thredds/catalog/NBS/S2B/2018/07/17/catalog.html --filename S2B_MSIL1C_20180717T095029_N0206_R079_T34VFM_20180717T115424.nc

This one didn't work for me. It find nothing:

(base) root@0294d0ad74aa:/src# project/manage.py ingest_thredds_crawl http://nbstds.met.no/thredds/catalog/NBS/S2B/2018/07/17/catalog.html --filename S2B_MSIL1C_20180717T095029_N0206_R079_T34VFM_20180717T115424.nc
2019-02-13 13:55:24,061 - [INFO] Crawling: http://nbstds.met.no/thredds/catalog/NBS/S2B/2018/07/17/catalog.html
2019-02-13 13:55:48,434 - [INFO] Ignoring dataset based on 'selects'.  ID: nbs/S2B/2018/07/17/S2B_MSIL1C_20180717T144749_N0206_R082_T35XML_20180717T195515.nc

..... many filenames come here .....

2019-02-13 13:55:48,483 - [INFO] Ignoring dataset based on 'selects'.  ID: nbs/S2B/2018/07/17/S2B_MSIL1C_20180717T095029_N0206_R079_T34VCH_20180717T133928.nc
HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 140212812998400:
  #000: H5F.c line 509 in H5Fopen(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #001: H5Fint.c line 1400 in H5F__open(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #002: H5Fint.c line 1548 in H5F_open(): unable to open file: name = 'http://nbstds.met.no/thredds/dodsC/NBS/S2B/2018/07/17/S2B_MSIL1C_20180717T095029_N0206_R079_T34VFM_20180717T115424.nc', tent_flags = 0
    major: File accessibilty
    minor: Unable to open file
  #003: H5FD.c line 734 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object
Successfully added metadata of 0 datasets
akorosov commented 5 years ago

The same with another option. After some file which look OK:

2019-02-13 14:10:43,343 - [INFO] Ignoring dataset based on 'selects'.  ID: nbs/S2B/2018/07/01/S2B_MSIL1C_20180701T093039_N0206_R136_T35VMH_20180701T113337.nc
2019-02-13 14:10:43,344 - [INFO] Ignoring dataset based on 'selects'.  ID: nbs/S2B/2018/07/01/S2B_MSIL1C_20180701T093039_N0206_R136_T35VMF_20180701T113337.nc
2019-02-13 14:10:43,344 - [INFO] Ignoring dataset based on 'selects'.  ID: nbs/S2B/2018/07/01/S2B_MSIL1C_20180701T093039_N0206_R136_T35VME_20180701T113337.nc
2019-02-13 14:10:43,344 - [INFO] Ignoring dataset based on 'selects'.  ID: nbs/S2B/2018/07/01/S2B_MSIL1C_20180701T093039_N0206_R136_T35VLE_20180701T113337.nc
2019-02-13 14:10:43,344 - [INFO] Ignoring dataset based on 'selects'.  ID: nbs/S2B/2018/07/01/S2B_MSIL1C_20180701T093039_N0206_R136_T35VLD_20180701T113337.nc

Come many errors:

HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 140197406467840:
  #000: H5F.c line 509 in H5Fopen(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #001: H5Fint.c line 1400 in H5F__open(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #002: H5Fint.c line 1548 in H5F_open(): unable to open file: name = 'http://nbstds.met.no/thredds/dodsC/NBS/S2B/2018/07/17/S2B_MSIL1C_20180717T144749_N0206_R082_T35XML_20180717T195515.nc', tent_flags = 0
    major: File accessibilty
    minor: Unable to open file
  #003: H5FD.c line 734 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object
HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 140197406467840:
  #000: H5F.c line 509 in H5Fopen(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #001: H5Fint.c line 1400 in H5F__open(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #002: H5Fint.c line 1548 in H5F_open(): unable to open file: name = 'http://nbstds.met.no/thredds/dodsC/NBS/S2B/2018/07/17/S2B_MSIL1C_20180717T144749_N0206_R082_T35XMK_20180717T195515.nc', tent_flags = 0
    major: File accessibilty
    minor: Unable to open file
  #003: H5FD.c line 734 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object
HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 140197406467840:
  #000: H5F.c line 509 in H5Fopen(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #001: H5Fint.c line 1400 in H5F__open(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #002: H5Fint.c line 1548 in H5F_open(): unable to open file: name = 'http://nbstds.met.no/thredds/dodsC/NBS/S2B/2018/07/17/S2B_MSIL1C_20180717T144749_N0206_R082_T33XXL_20180717T195515.nc', tent_flags = 0
    major: File accessibilty
    minor: Unable to open file
  #003: H5FD.c line 734 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object
HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 140197406467840:
  #000: H5F.c line 509 in H5Fopen(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #001: H5Fint.c line 1400 in H5F__open(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #002: H5Fint.c line 1548 in H5F_open(): unable to open file: name = 'http://nbstds.met.no/thredds/dodsC/NBS/S2B/2018/07/17/S2B_MSIL1C_20180717T144749_N0206_R082_T33XXK_20180717T195515.nc', tent_flags = 0
    major: File accessibilty
    minor: Unable to open file
  #003: H5FD.c line 734 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object

Maybe a timeout should be added before ds, cr = NansatDataset.objects.get_or_create(url) to not crash the opendap server ?

mortenwh commented 5 years ago

Those errors is not a problem - it still works..

ninsbl commented 2 years ago

Hei, I have been using thredds_crawler towards NBS too, but for me that regularly causes 502 or 504 errors. Did you notice that too? I tried to address thatin: https://github.com/ioos/thredds_crawler/pull/29

But I am not sure if the library is actively maintained.... I saw that @akorosov contributed to it earlier. Do you know more or an alternative to thredds_crawler if that is no longer actively maintained (there is also anonther un-commented PR from may)? Maybe that is a dead horse?

mortenwh commented 2 years ago

I think the problem is on the nbs server-side. They are struggling with thredds at the moment.

On Fri, Oct 1, 2021 at 10:57 AM Stefan Blumentrath @.***> wrote:

Hei, I have been using thredds_crawler towards NBS too, but for me that regularly causes 502 or 504 errors. Did you notice that too? I tried to address thatin: ioos/thredds_crawler#29 https://github.com/ioos/thredds_crawler/pull/29

But I am not sure if the library is actively maintained.... I saw that @akorosov https://github.com/akorosov contributed to it earlier. Do you know more or an alternative to thredds_crawler if that is no longer actively maintained (there is also anonther un-commented PR from may)? Maybe that is a dead horse?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/nansencenter/django-geo-spaas/issues/40#issuecomment-932048253, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA2UBJS6OD4WBTETYPXMZLUEVZWBANCNFSM4GCNRDJA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Morten Wergeland Hansen, PhD Meteorologisk Institutt / Norwegian Meteorological Institute T.: (+47) 915 47 844