stray light branch DNB error

[wstraka3]

This may be a CSPP issue with this directory, but, I'm getting the following errors:

ERROR : There were no files or more than one fitting the pattern /data2/users/pops/VIIRS/madison/d20130722_t0941.processing/GDNBO_npp_d20130722_t0941594_e0943235_b08982_cspp_dev.h5 DEBUG : Couldn't identify terrain corrected geonav file for file /data2/users/pops/VIIRS/madison/d20130722_t0941.processing/SVDNB_npp_d20130722_t0941594_e0943235_b08982_c20130722095355385957_cspp_dev.h5 ERROR : There were no files or more than one fitting the pattern /data2/users/pops/VIIRS/madison/d20130722_t0941.processing/GDNBO_npp_d20130722_t0941594_e0943235b08982cspp_dev.h5 ERROR : Could not find TC or non-TC navigation files for /data2/users/pops/VIIRS/madison/d20130722_t0941.processing/SVDNB_npp_d20130722_t0941594_e0943235_b08982_c20130722095355385957_cspp_dev.h5 ERROR : There was an error getting information from filename '/data2/users/pops/VIIRS/madison/d20130722_t0941.processing/SVDNB_npp_d20130722_t0941594_e0943235_b08982_c20130722095355385957_cspp_dev.h5' Traceback (most recent call last): File "/data2/users/pops/polar2grid-swbundle-1.1_straylight/polar2grid/py/polar2grid_viirs/polar2grid/viirs/viirs_imager_to_swath.py", line 122, in get_meta_data finfo = file_info(fn) File "/data2/users/pops/polar2grid-swbundle-1.1_straylight/polar2grid/py/polar2grid_viirs/polar2grid/viirs/viirs_guidebook.py", line 614, in file_info raise ValueError("Could not find TC or non-TC navigation files for %s" % (fn,)) ValueError: Could not find TC or non-TC navigation files for /data2/users/pops/VIIRS/madison/d20130722_t0941.processing/SVDNB_npp_d20130722_t0941594_e0943235_b08982_c20130722095355385957_cspp_dev.h5

This results in 80% of the image not being displayed . Data located /data1/test_data//d20130722_t0941.processing/. Tried several different commands

$viirs2awips_exe_dir/viirs2awips.sh -vvv --num-procs=4 ${grid_command} --remove-stray-light -f $data_fl/SV*.h5 --new-dnb

$viirs2awips_exe_dir/viirs2awips.sh -vvv --num-procs=4 ${grid_command} --remove-stray-light -f $data_fl/SVDNB*.h5 --new-dnb

$viirs2awips_exe_dir/viirs2awips.sh -vvv --num-procs=4 ${grid_command} -f $data_fl/SVDNB*.h5 --new-dnb

$viirs2awips_exe_dir/viirs2awips.sh -vvv --num-procs=4 ${grid_command} --remove-stray-light -f $data_fl/SV*.h5

Seems to just be the DNB that is affected, which means it might be an input file problem.

[wstraka3]

This could be because there are two sets of files in the same folder. Worth some testing.

[djhoese]

William, you're correct that the two sets of files are causing this problem. What's happening is the frontend is looking for the "TC" GDNBO file and is finding two matches. Since it's not sure how to handle that it looks for "non-TC" GDNBO files and finds the same problem. I say TC vs non-TC because this strategy is used for the geolocation of the I and M bands (prefer the TC, but if not try the non-TC), for the DNB it just duplicates the functionality.

So first, we'll need to figure out why it has 2 sets of files. Second, I could make the logic more complex if the creation date ("c") in the geolocation filename can be determined from the data filename. Doing this *might give it the brains to make a proper decision.

[wstraka3]

Dave

On 7/22/2013 1:37 PM, David wrote:

William, you're correct that the two sets of files are causing this problem. What's happening is the frontend is looking for the "TC" GDNBO file and is finding two matches. Since it's not sure how to handle that it looks for "non-TC" GDNBO files and finds the same problem. I say TC vs non-TC because this strategy is used for the geolocation of the I and M bands (prefer the TC, but if not try the non-TC), for the DNB it just duplicates the functionality.

I'm a little confused. How is it looking for the terrain corrected files when they don't exist for the DNB?

Also, why was it not having the problem with the I-Bands??

So first, we'll need to figure out why it has 2 sets of files.

That is simple enough. There is the data from CIMSS and the data from Oregon State being put into one folder on the CIMSS FTP server. But I have seen this sort of thing happen with data from CLASS. Either way, it is a situation that could occur elsewhere.

Second, I could make the logic more complex if the creation date ("c*") in the geolocation filename can be determined from the data filename. Doing this /might/ give it the brains to make a proper decision.

Possibly. But given that they are so close to each other in creation time, might that cause an issue?

Why not pull the Nav from the Nav_GEO_REF attribute (which is a required attribute per the VIIRS SDS documentation).

William Straka III Associate Researcher University of Wisconsin - Madison Space Science and Engineering Center (SSEC) Cooperative Institute for Meteorological Satellite Studies (CIMSS) 1225 West Dayton Street Madison, WI 53706 Phone: (608) 262-5128 Fax: (608) 262-5974 Email: william.straka@ssec.wisc.edu

[djhoese]

What I meant by "duplicates the functionality" is that since the DNB doesn't have TC files it searches for the non-TC files twice...but it "thinks" that it is looking for TC then non-TC. It's preparation for when TC DNB does exist. You can see this happening in the log messages since it looks for the same file pattern twice:

ERROR : There were no files or more than one fitting the pattern /data2/users/pops/VIIRS/madison/d20130722_t0941.processing/GDNBO_npp_d20130722_t0941594_e0943235_b08982_*_cspp_dev.h5

I'll have to look at the log files sometime this week and figure out why the I bands didn't fail. Were there duplicates of both? Were they processed at the same time? We can talk about this in person tomorrow.

Having two sets of data go to the same directory isn't preferred in my opinion and should be handled outside of polar2grid (again opinion). That being said, the creation time search could be more restrictive. Right now it searches for any creation time "c", it could use maybe the date and hour of the day "c2013072309". It might work, but it also restricts what the user is able to do (i.e. they need to have the navigation and data files created at the same time).

Iirc we didn't use the NAV_GEO_REF when we first started making the VIIRS frontend because they weren't correct. Either the creation time was wrong, or it only had the non-TC filename, or some other thing. It might have also been wrong for only one of the types of bands, DNB vs I vs M.

[djhoese]

Looking back on this issue it looks like the problem was 2 datasets being in the same directory. Closing now unless there is more to discuss.