search

GeoscienceAustralia / agdc

Repository for Australian Geoscience Data Cube (AGDC) code
BSD 3-Clause "New" or "Revised" License
29 stars 24 forks source link

NBAR Landsat ingest fails - PQ works ok #73

Closed simonaoliver closed 9 years ago

simonaoliver commented 9 years ago

working on develop on Ubuntu 64:

execution example:

python -m agdc.ingest.landsat -C agdc_default.conf --source /mnt/scenes/nbar/

2015-07-15 16:59:01,534 agdc.ingest.landsat INFO Searching for datasets in /mnt/scenes/nbar/
2015-07-15 16:59:01,619 agdc.ingest.landsat.landsat_dataset INFO Opening Dataset /mnt/scenes/nbar/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306
2015-07-15 16:59:02,203 eotools.drivers._scene_dataset INFO 78 values read from metadata
2015-07-15 16:59:02,526 agdc.ingest._core INFO Ingestion failed for dataset '/mnt/scenes/nbar/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306' in 0:00:00.907321:
2015-07-15 16:59:02,527 agdc.ingest._core INFO Unable to find unique match for file pattern .*_B10\..*
2015-07-15 16:59:02,530 agdc.ingest.landsat.landsat_dataset INFO Opening Dataset /mnt/scenes/nbar/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330
2015-07-15 16:59:03,118 eotools.drivers._scene_dataset INFO 78 values read from metadata
2015-07-15 16:59:03,466 agdc.ingest._core INFO Ingestion failed for dataset '/mnt/scenes/nbar/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330' in 0:00:00.936024:
2015-07-15 16:59:03,467 agdc.ingest._core INFO Unable to find unique match for file pattern .*_B1\..*
2015-07-15 16:59:03,471 agdc.ingest._core INFO Ingestion process complete for source directory '/mnt/scenes/nbar' in 0:00:01.937171.

This is what the inputs look like - note that PQ products ingest ok.

simonaoliver@simonaoliver-VirtualBox:~/datacube/agdc-develop/agdc$ find /mnt/scenes/nbar/
/mnt/scenes/nbar/
/mnt/scenes/nbar/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306
/mnt/scenes/nbar/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306.jpg
/mnt/scenes/nbar/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306_FR.jpg
/mnt/scenes/nbar/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306/md5sum.txt
/mnt/scenes/nbar/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306/metadata.xml
/mnt/scenes/nbar/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306/scene01
/mnt/scenes/nbar/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306/scene01/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306_B10.tif
/mnt/scenes/nbar/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306/scene01/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306_B10.tif.aux.xml
/mnt/scenes/nbar/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306/scene01/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306_B20.tif
/mnt/scenes/nbar/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306/scene01/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306_B30.tif
/mnt/scenes/nbar/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306/scene01/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306_B40.tif
/mnt/scenes/nbar/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306/scene01/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306_B50.tif
/mnt/scenes/nbar/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306/scene01/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306_B70.tif
/mnt/scenes/nbar/LS7_ETM_NBAR_P54_GANBAR01-002_091_085_20150306/scene01/report.txt
/mnt/scenes/nbar/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330
/mnt/scenes/nbar/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330.jpg
/mnt/scenes/nbar/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330_FR.jpg
/mnt/scenes/nbar/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330/md5sum.txt
/mnt/scenes/nbar/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330/metadata.xml
/mnt/scenes/nbar/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330/scene01
/mnt/scenes/nbar/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330/scene01/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330_B1.tif
/mnt/scenes/nbar/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330/scene01/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330_B1.tif.aux.xml
/mnt/scenes/nbar/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330/scene01/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330_B2.tif
/mnt/scenes/nbar/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330/scene01/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330_B3.tif
/mnt/scenes/nbar/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330/scene01/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330_B4.tif
/mnt/scenes/nbar/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330/scene01/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330_B5.tif
/mnt/scenes/nbar/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330/scene01/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330_B6.tif
/mnt/scenes/nbar/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330/scene01/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330_B7.tif
/mnt/scenes/nbar/LS8_OLI_TIRS_NBAR_P54_GANBAR01-032_091_085_20150330/scene01/report.txt
jeremyh commented 9 years ago

It was caused by the *.aux.xml file that newer gdal releases create. A fix is in develop.