landsat C1 vsitar problem?

bhbraswell commented 6 years ago

This is probably another easy one. Is there a switch I'm not seeing to read from tar.gz?

gips@gips:~$ gips_process landsat -p ndvi-toa -d 2018-05-03 -s /archive/vector/temp_buffer.shp -v4        
GIPS Data Processing (v0.11.0-dev)
Retrieving inventory for site temp_buffer-0
Attempting to load LE07_L1TP_025030_20180503_20180503_01_RT.tar.gz
C1 asset
Attempting to load LE07_L1GT_025031_20180503_20180503_01_RT.tar.gz
C1 asset
Processing [ndvi-toa] on 1 dates (2 files)
Fatal: 1 error(s) occurred:
Problem accessing asset(s) in /archive/landsat/tiles/025031/2018123/LE07_L1GT_025031_20180503_20180503_01_RT.tar.gz:
Traceback (most recent call last):
  File "/gips/gips/utils.py", line 578, in cli_error_handler
    yield
  File "/gips/gips/data/core.py", line 356, in datafiles
    sds = fh.GetSubDatasets()
AttributeError: 'NoneType' object has no attribute 'GetSubDatasets'

ra-tolson commented 6 years ago

If you're trying to read it as a sub-dataset, I think your asset isn't passing the check if tarfile.is_tarfile(...), which means your file may be messed up. Maybe run file /archive/landsat/tiles/025031/2018123/LE07_L1GT_025031_20180503_20180503_01_RT.tar.gz and see what you get?

bhbraswell commented 6 years ago

Yep oops

gips@gips:~$ file /archive/landsat/tiles/025031/2018123/LE07_L1GT_025031_20180503_20180503_01_RT.tar.gz
/archive/landsat/tiles/025031/2018123/LE07_L1GT_025031_20180503_20180503_01_RT.tar.gz: HTML document, ASCII text
gips@gips:~$ cat /archive/landsat/tiles/025031/2018123/LE07_L1GT_025031_20180503_20180503_01_RT.tar.gz 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<HTML><HEAD>
<TITLE>404 Not Found</TITLE>
</HEAD><BODY>
<H1>Not Found</H1>
The requested URL /hsm/lsat1/collection01/etm/RT/2018/25/31/LE07_L1GT_025031_20180503_20180503_01_RT.tar.gz was not found on this server.<P>
</BODY></HTML>

ircwaves commented 6 years ago

Nice. Weird that it tried to get that which isn't there...maybe the RT got replaced by the T2 or T1 before the fetch happened?

Applied-GeoSolutions / gips

landsat C1 vsitar problem? #489