search

landsat-pds / landsat_ingestor

Scripts and other artifacts for landsat data ingestion into Amazon public hosting.
Apache License 2.0
102 stars 18 forks source link

corrupt tar files reprocessed indefinately #9

Closed warmerdam closed 9 years ago

warmerdam commented 9 years ago

Every two hours as we try to reprocess the tarq contents this corrupt scene is tried and fails. We need some better logic to migrate such corrupt files to the tarq_corrupt area like we do for the quick test in puller_s3queue.py. It is a little tricker for this case since it happens significantly later. 

l8_process_scene.py --verbose -s s3queue --clean --overwrite --list-file job_33049805.csv LC81790442014335LGN00
Scene LC81790442014335LGN00 already exists on destination bucket.
Processing scene: LC81790442014335LGN00
Fetching: http://s3-us-west-2.amazonaws.com/landsat-pds/tarq/LC81790442014335LGN00.tar.gz
...
.....
....
.....
.....
....
.....
.....
.....
.....
.....
.....
....
....
......
.....
....
.....
.....
.LC81790442014335LGN00_B1.TIF
LC81790442014335LGN00_B2.TIF
LC81790442014335LGN00_B3.TIF
LC81790442014335LGN00_B4.TIF
LC81790442014335LGN00_B5.TIF
LC81790442014335LGN00_B6.TIF
LC81790442014335LGN00_B7.TIF
LC81790442014335LGN00_B8.TIF
LC81790442014335LGN00_B9.TIF
LC81790442014335LGN00_B10.TIF
LC81790442014335LGN00_B11.TIF
LC81790442014335LGN00_BQA.TIF

gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
LC81790442014335LGN00.tar.gz successfully downloaded (942628864 bytes)
tar xvf LC81790442014335LGN00.tar.gz --directory=LC81790442014335LGN00 
Traceback (most recent call last):
  File "/opt/planet/programs/landsat_ingestor/ingestor/l8_process_scene.py", line 110, in <module>
    status = main(sys.argv[1:])
  File "/opt/planet/programs/landsat_ingestor/ingestor/l8_process_scene.py", line 101, in main
    overwrite = args.overwrite)
  File "/opt/planet/programs/landsat_ingestor/ingestor/l8_process_scene.py", line 58, in process
    local_dir = splitter.split(scene_root, local_tarfile, verbose=verbose)
  File "/opt/planet/programs/landsat_ingestor/ingestor/splitter.py", line 63, in split
    verbose=verbose)
  File "/opt/planet/programs/landsat_ingestor/ingestor/splitter.py", line 13, in run_command
    raise Exception('command "%s" failed with code %d.' % (cmd, result))
Exception: command "tar xvf LC81790442014335LGN00.tar.gz --directory=LC81790442014335LGN00 " failed with code 512.
Task ended with status 1
+ STATUS=1
kapadia commented 9 years ago

https://github.com/landsat-pds/landsat_ingestor/commit/b86d2b9f50623fc55c74610bd88b066b6ebcfb31 address the corrupt tarballs.