search

ssec / polar2grid

Tools for reading, remapping, and writing satellite instrument data.
http://www.ssec.wisc.edu/software/polar2grid/
GNU General Public License v3.0
72 stars 34 forks source link

P2G v2.0 rc3 failed to produce VIIRS AWIPS NetCDF files on test pass #106

Closed kathys closed 9 years ago

kathys commented 9 years ago

Testing the new p2g release, I ran a command on this pass of our VIIRS SDR data:

ftp://ftp.ssec.wisc.edu/pub/eosdb/npp/viirs/2015_05_06_126_1851/sdr/

This is a short pass consisting of 4 granules.

The command I used was: viirs2awips.sh -g 211e 211w --debug -f ${viirs_sdrdir}/SV{I01,I02,I03,I04,I05,DNB}.h5 ${viirs_sdrdir}/G{ITC,DNB}O.h5

The processing did not produce any NetCDF files. Lots of info in log file, but it is still not clear to me why it failed. It also looks like it was trying to process bands and grids that I did not tell it to. Perhaps there is some syntax that has changed?

I placed the output logfile here:
ftp://ftp.ssec.wisc.edu/pub/ssec/kathys/viirs2awips_20150506_185157.log

djhoese commented 9 years ago

This looks like an error in a configuration file. I'm not sure exactly how/why this is happening, but I'll try to get you a fix as soon as possible.

What bands/grids did you not want that you think it is processing for?

kathys commented 9 years ago

When I look at the log file, I see entries like this:

logical defaults products [2015-05-07 16:53:51,401] : PID 27364 : WARNING : polar2grid.core.roles : loadable_products : Missing proper data files to create product: m06 [2015-05-07 16:53:51,402] : PID 27364 : WARNING : polar2grid.core.roles : loadable_products : Missing proper data files to create product: m07 [2015-05-07 16:53:51,402] : PID 27364 : WARNING : polar2grid.core.roles : loadable_products : Missing proper data files to create product: m01 [2015-05-07 16:53:51,402] : PID 27364 : WARNING : polar2grid.core.roles : loadable_products : Missing proper data files to create product: m04 [2015-05-07 16:53:51,402] : PID 27364 : WARNING : polar2grid.core.roles : loadable_products : Missing proper data files to create product: m11 [2015-05-07 16:53:51,402] : PID 27364 : WARNING : polar2grid.core.roles : loadable_products : Missing proper data files to create product: m10 [2015-05-07 16:53:51,402] : PID 27364 : WARNING : polar2grid.core.roles : loadable_products : Missing proper data files to create product: m13 [2015-05-07 16:53:51,402] : PID 27364 : WARNING : polar2grid.core.roles : loadable_products : Missing proper data files to create product: m12 [2015-05-07 16:53:51,402] : PID 27364 : WARNING : polar2grid.core.roles : loadable_products : Missing proper data files to create product: m15 [2015-05-07 16:53:51,402] : PID 27364 : WARNING : polar2grid.core.roles : loadable_products : Missing proper data files to create product: m14 [2015-05-07 16:53:51,402] : PID 27364 : WARNING : polar2grid.core.roles : loadable_products : Missing proper data files to create product: m16 [2015-05-07 16:53:51,402] : PID 27364 : WARNING : polar2grid.core.roles : loadable_products : Missing proper data files to create product: m02 .......

[2015-05-07 16:53:51,375] : PID 27364 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '203' origin from (lon: 115.601000, lat: 44.646000) to (x: -4952960.856044, y: 381022.610378) [2015-05-07 16:53:51,375] : PID 27364 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '203_10km' origin from (lon: 115.601000, lat: 44.646000) to (x: -4952960.856044, y: 381022.610378) [2015-05-07 16:53:51,375] : PID 27364 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '204' origin from (lon: 110.000000, lat: 60.644000) to (x: -7314463.252793, y: 8020503.279488) [2015-05-07 16:53:51,375] : PID 27364 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '204_10km' origin from (lon: 110.000000, lat: 60.644000) to (x: -7314463.252793, y: 8020503.279488) [2015-05-07 16:53:51,375] : PID 27364 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'dwd_germany' origin from (lon: -2.000000, lat: 56.000000) to (x: -755570.370463, y: -3554679.114710) [2015-05-07 16:53:51,375] : PID 27364 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'australia' origin from (lon: 105.000000, lat: 5.000000) to (x: -3896182.177765, y: 557305.257275) [2015-05-07 16:53:51,376] : PID 27364 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'australia2' origin from (lon: 105.000000, lat: 5.000000) to (x: -3896182.177765, y: 557305.257275)

[2015-05-07 16:53:51,402] : PID 27364 : WARNING : polar2grid.core.roles : loadable_products : Missing proper data files to create product: m05 [2015-05-07 16:53:51,402] : PID 27364 : WARNING : polar2grid.core.roles : loadable_products : Missing proper data files to create product: m09 [2015-05-07 16:53:51,402] : PID 27364 : WARNING : polar2grid.core.roles : loadable_products : Missing proper data files to create product: m08 [2015-05-07 16:53:51,403] : PID 27364 : WARNING : polar2grid.core.roles : loadable_products : Missing proper data files to create product: m03

djhoese commented 9 years ago

That is due to the default being to try to create all products. I can turn off the WARNING level for those and make them DEBUG information instead, but that causes even more confusion if you ask for a product and don't provide the right data files.

The part about converting the grids is when it is loading the grid configuration file it has to convert any "degree" parameters in to "meters" in the projection space. That I made debug information because most users don't need to know it. The log file does show that it only remapped to the 2 grids you specified.

kathys commented 9 years ago

Okay. Just so I understand, if I wanted to specify only those grids and only those products, I would use the -p flag? For example, -p i01 i02 i03 i04 i05 histogram_dnb adaptive_dnb dynamic_dnb

djhoese commented 9 years ago

Yes that would work...once I fix the config bug of course.

kathys commented 9 years ago

How about fog product? What would I use for that?

djhoese commented 9 years ago

ifog for that. You can also specify the --list-products flag to get a list of what's possible for the frontend you specified.

kathys commented 9 years ago

Very useful. Thank you.

kathys commented 9 years ago

viirs2awips.sh --list-products
just prints out same info as viir2awips.sh without arguments. Am I implementing it correctly?

djhoese commented 9 years ago

That is the right flag, but I had to try it out myself to realize that it requires files/dirs specified with the "-f" flag so it can tell you what products it can make from those files. So it tells you "available products" instead of "all products". If you only had DNB and GDNBO files in a directory it would only list the DNB based products.

kathys commented 9 years ago

Tried it and it worked. Thanks.

djhoese commented 9 years ago

This bug and all of the others we've discussed have been fixed and made available for testing in the rc4 tarball on meelo. It took me a while to reproduce this because I was ignoring the --debug flag (which just keeps intermediate files around after execution) that you were providing. It turns out that that was the problem. Specifying this option was causing a keyword argument to be passed to the rescaler in the code as a default configuration value (I can go in more detail if you want). It's fixed now and if you could test it out that would be great. It looks good on my end.

One thing to note: rc4 includes the change to dynamic DNB that Curtis Seaman emailed us about. It doesn't seem to have an affect on the daytime and transition test case I have. It did have an effect on an old nighttime case. It went from all white to an image where you could see some city lights. You can test it yourself and can tell when the update makes a difference if there are more than one Dynamic DNB saturation percentage: debug messages.

kathys commented 9 years ago

Thanks very much Dave for working so hard on this. This is a great software package.

I forgot to tell you that I am on vacation this week. I will try out all of my processing threads using rc4 next week.

Kathy

P.S. No meeting on Wednesday:)

On Sun, 10 May 2015 15:59:25 -0700 David Hoese notifications@github.com wrote:

This bug and all of the others we've discussed have been fixed and made available for testing in the rc4 tarball on meelo. It took me a while to reproduce this because I was ignoring the --debug flag (which just keeps intermediate files around after execution) that you were providing. It turns out that that was the problem. Specifying this option was causing a keyword argument to be passed to the rescaler in the code as a default configuration value (I can go in more detail if you want). It's fixed now and if you could test it out that would be great. It looks good on my end.

One thing to note: rc4 includes the change to dynamic DNB that Curtis Seaman emailed us about. It doesn't seem to have an affect on the daytime and transition test case I have. It did have an effect on an old nighttime case. It went from all white to an image where you could see some city lights. You can test it yourself and can tell when the update makes a difference if there are more than one Dynamic DNB saturation percentage: debug messages.

Reply to this email directly or view it on GitHub: https://github.com/davidh-ssec/polar2grid/issues/106#issuecomment-100713542