search

gina-alaska / sandy-utils

processing bits attached to the sandy (and other) data processing streams.
Apache License 2.0
0 stars 0 forks source link

bad geotifs #56

Closed spruceboy closed 7 years ago

spruceboy commented 7 years ago

Some of the geotifs we making on the nrt stack are either short or have some other problem, and generate errors in gdal. A problem.

teknofire commented 7 years ago

Hrmm, can you link an example here?

spruceboy commented 7 years ago

It doesn't happen always - I will dig around and try to find an example. I noticed it a week ago, but figured it was an issue on my local gdal instalation. Apparnetly it is not though.

spruceboy commented 7 years ago

[jcable@spam geo-tiff_test]$ gdal_translate 14_46_32_32_npp.20170613.2151_true_color.tif foo.tif ERROR 1: TIFFFetchDirectory:14_46_32_32_npp.20170613.2151_true_color.tif: Can not read TIFF directory count wget http://feeder.gina.alaska.edu/dragonfly/2017/06/13/14_46_32_32_npp.20170613.2151_true_color.tif ERROR 1: TIFFReadDirectory:14_46_32_32_npp.20170613.2151_true_color.tif: Failed to read directory at offset 0 Input file size is 12000, 12000 0...10...20...30...40...50...60...70...80...90...100 - done.

spruceboy commented 7 years ago

Hmm, not sure what is going on. That geotif in the nrt stack looks fine: http://nrt-dds-prod.gina.alaska.edu/uafgina/snpp/geotiff_l2/viirs/2017/06/npp.17164.2151/npp.20170613.2151_true_color.tif

spruceboy commented 7 years ago

A feeder ecosystem issue maybe @teknofire ?

teknofire commented 7 years ago

feeder shouldn't be doing anything to the geotiffs

spruceboy commented 7 years ago

463d5dde4aedc757d601b34bb2b3c86f 14_46_32_32_npp.20170613.2151_true_color.tif 6fbad8dfb45d63b3b66d4c7d84ac94fb npp.20170613.2151_true_color.tif

definately different.
I can't find other examples ,will use the rss feed to pull data and see if I can find other examples. It seems to me somewhere along the line the tifs out of the nrt stack are getting currupted.

teknofire commented 7 years ago

looks like there is something that is happening after the the mirror_products script from feeder-flinger0 and the copy to the gluster on feeder-worker0. I've added some additional checks on the two conveyor scripts that happen there to try and detect if there is a slow wget or scp happening that might be causing it to process a partial file