search

ssec / polar2grid

Tools for reading, remapping, and writing satellite instrument data.
http://www.ssec.wisc.edu/software/polar2grid/
GNU General Public License v3.0
72 stars 34 forks source link

The dtype in avhrr2gtiff.sh #115

Closed jiaozhh closed 7 years ago

jiaozhh commented 8 years ago

Which string can be used for the dtype in avhrr2gtiff.sh commond? float, real4, etc. are all wrong. Thanks.

djhoese commented 8 years ago

real4 should work. What kind of error or unexpected result are you getting?

jiaozhh commented 8 years ago

Sorry, I made a mistake. I use -dtype real4, but it should be --dtype real4. The input command line is avhrr2gtiff.sh -g lcc_asia -p band4_bt band5_bt --dtype real4 -f /mnt/hgfs/wf/hrpt_noaa18_20160326_2004_17191.l1b But I got another error as follows: [2016-03-29 10:54:35,792] : PID 6185 : DEBUG : avhrr2gtiff : main : Starting script with arguments: /home/jzh/soft/polar2grid/ShellB3/lib/python2.7/site-packages/polar2grid-2.0.0-py2.7-linux-x86_64.egg/polar2grid/glue.py avhrr gtiff -g lcc_asia -p band4_bt band5_bt --dtype real4 -f /mnt/hgfs/wf/hrpt_noaa18_20160326_2004_17191.l1b -vv [2016-03-29 10:54:35,792] : PID 6185 : INFO : avhrr2gtiff : main : Initializing swath extractor... [2016-03-29 10:54:35,802] : PID 6185 : DEBUG : polar2grid.avhrr.avhrr2swath : load_files : Recognize file /mnt/hgfs/wf/hrpt_noaa18_20160326_2004_17191.l1b as file type FT_AAPP [2016-03-29 10:54:35,810] : PID 6185 : DEBUG : polar2grid.core.script_utils : rename_log_file : Log renamed from '/home/jzh/avhrr/avhrr2gtiff_fail.log' to 'avhrr2gtiff_20160326_091937.log' [2016-03-29 10:54:35,811] : PID 6185 : INFO : avhrr2gtiff : main : Initializing remapping... [2016-03-29 10:54:35,811] : PID 6185 : DEBUG : polar2grid.grids.manager : init : Using default grid configuration: 'grids.conf' [2016-03-29 10:54:35,811] : PID 6185 : DEBUG : polar2grid.grids.manager : init : Loading grid configuration 'grids.conf' [2016-03-29 10:54:35,813] : PID 6185 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'dwd_germany' origin from (lon: -2.000000, lat: 56.000000) to (x: -755570.370463, y: -3554679.114710) [2016-03-29 10:54:35,813] : PID 6185 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'australia' origin from (lon: 105.000000, lat: 5.000000) to (x: -3896182.177765, y: 557305.257275) [2016-03-29 10:54:35,813] : PID 6185 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'australia2' origin from (lon: 105.000000, lat: 5.000000) to (x: -3896182.177765, y: 557305.257275) [2016-03-29 10:54:35,813] : PID 6185 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '211e' origin from (lon: -123.044000, lat: 59.844000) to (x: -1950510.636800, y: 4368587.226913) [2016-03-29 10:54:35,814] : PID 6185 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '211e_hi' origin from (lon: -123.044000, lat: 59.844000) to (x: -1950510.636800, y: 4368587.226913) [2016-03-29 10:54:35,814] : PID 6185 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '211w' origin from (lon: -152.855000, lat: 54.536000) to (x: -4226045.730436, y: 4368634.765713) [2016-03-29 10:54:35,814] : PID 6185 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '203' origin from (lon: 115.601000, lat: 44.646000) to (x: -4952960.856044, y: 381022.610378) [2016-03-29 10:54:35,814] : PID 6185 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '203_10km' origin from (lon: 115.601000, lat: 44.646000) to (x: -4952960.856044, y: 381022.610378) [2016-03-29 10:54:35,814] : PID 6185 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '204' origin from (lon: 110.000000, lat: 60.644000) to (x: -7314463.252793, y: 8020503.279488) [2016-03-29 10:54:35,814] : PID 6185 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '204_10km' origin from (lon: 110.000000, lat: 60.644000) to (x: -7314463.252793, y: 8020503.279488) [2016-03-29 10:54:35,815] : PID 6185 : INFO : avhrr2gtiff : main : Initializing backend... [2016-03-29 10:54:35,815] : PID 6185 : DEBUG : polar2grid.core.rescale : init : Loading rescale configuration files: polar2grid.core:rescale_configs/rescale.ini [2016-03-29 10:54:35,815] : PID 6185 : DEBUG : polar2grid.core.roles : open_config_file : Loading package provided configuration file: 'polar2grid.core:rescale_configs/rescale.ini' [2016-03-29 10:54:35,828] : PID 6185 : INFO : avhrr2gtiff : main : Initializing compositor objects... [2016-03-29 10:54:35,828] : PID 6185 : INFO : avhrr2gtiff : main : Extracting swaths from data files available... [2016-03-29 10:54:35,828] : PID 6185 : DEBUG : polar2grid.avhrr.avhrr2swath : create_scene : Loading scene data... [2016-03-29 10:54:35,829] : PID 6185 : DEBUG : polar2grid.core.frontend_utils : dependency_ordered_products : Searching for dependencies for 'band4_bt' [2016-03-29 10:54:35,829] : PID 6185 : DEBUG : polar2grid.core.frontend_utils : dependency_ordered_products : Searching for dependencies for 'band5_bt' [2016-03-29 10:54:35,829] : PID 6185 : INFO : polar2grid.avhrr.avhrr2swath : create_scene : Creating navigation product 'longitude1km' [2016-03-29 10:54:35,829] : PID 6185 : DEBUG : polar2grid.avhrr.avhrr2swath : create_raw_swath_object : Using file type 'FT_AAPP' and getting file key 'LongitudeVar' for product 'longitude1km' [2016-03-29 10:54:35,829] : PID 6185 : DEBUG : polar2grid.avhrr.avhrr2swath : create_raw_swath_object : Writing product 'longitude1km' data to binary file [2016-03-29 10:54:35,829] : PID 6185 : DEBUG : polar2grid.core.frontend_utils : write_var_to_flat_binary : Writing binary data for 'LongitudeVar' to file 'longitude1km.dat' [2016-03-29 10:54:36,347] : PID 6185 : DEBUG : polar2grid.avhrr.readers : geolocation_calibration : Interpolating 40km navigation to 1km... [2016-03-29 10:54:41,783] : PID 6185 : DEBUG : polar2grid.core.fbf : append : 4313 rows in output file [2016-03-29 10:54:41,784] : PID 6185 : DEBUG : polar2grid.core.frontend_utils : write_var_to_flat_binary : File longitude1km.dat has shape (4313, 2048) [2016-03-29 10:54:41,784] : PID 6185 : INFO : polar2grid.avhrr.avhrr2swath : create_scene : Creating navigation product 'latitude1km' [2016-03-29 10:54:41,785] : PID 6185 : DEBUG : polar2grid.avhrr.avhrr2swath : create_raw_swath_object : Using file type 'FT_AAPP' and getting file key 'LatitudeVar' for product 'latitude1km' [2016-03-29 10:54:41,785] : PID 6185 : DEBUG : polar2grid.avhrr.avhrr2swath : create_raw_swath_object : Writing product 'latitude1km' data to binary file [2016-03-29 10:54:41,785] : PID 6185 : DEBUG : polar2grid.core.frontend_utils : write_var_to_flat_binary : Writing binary data for 'LatitudeVar' to file 'latitude1km.dat' [2016-03-29 10:54:41,864] : PID 6185 : DEBUG : polar2grid.core.fbf : append : 4313 rows in output file [2016-03-29 10:54:41,865] : PID 6185 : DEBUG : polar2grid.core.frontend_utils : write_var_to_flat_binary : File latitude1km.dat has shape (4313, 2048) [2016-03-29 10:54:41,865] : PID 6185 : INFO : polar2grid.avhrr.avhrr2swath : create_scene : Creating data product 'band4_bt' [2016-03-29 10:54:41,865] : PID 6185 : DEBUG : polar2grid.avhrr.avhrr2swath : create_raw_swath_object : Using file type 'FT_AAPP' and getting file key 'Band4Var' for product 'band4_bt' [2016-03-29 10:54:41,866] : PID 6185 : DEBUG : polar2grid.avhrr.avhrr2swath : create_raw_swath_object : Writing product 'band4_bt' data to binary file [2016-03-29 10:54:41,866] : PID 6185 : DEBUG : polar2grid.core.frontend_utils : write_var_to_flat_binary : Writing binary data for 'Band4Var' to file 'band4_bt.dat' [2016-03-29 10:54:42,955] : PID 6185 : DEBUG : polar2grid.core.fbf : append : 4313 rows in output file [2016-03-29 10:54:42,956] : PID 6185 : DEBUG : polar2grid.core.frontend_utils : write_var_to_flat_binary : File band4_bt.dat has shape (4313, 2048) [2016-03-29 10:54:42,956] : PID 6185 : DEBUG : polar2grid.core.containers : set_persist : Setting persist to False for child 'swath_definition' [2016-03-29 10:54:42,957] : PID 6185 : INFO : polar2grid.avhrr.avhrr2swath : create_scene : Creating data product 'band5_bt' [2016-03-29 10:54:42,957] : PID 6185 : DEBUG : polar2grid.avhrr.avhrr2swath : create_raw_swath_object : Using file type 'FT_AAPP' and getting file key 'Band5Var' for product 'band5_bt' [2016-03-29 10:54:42,957] : PID 6185 : DEBUG : polar2grid.avhrr.avhrr2swath : create_raw_swath_object : Writing product 'band5_bt' data to binary file [2016-03-29 10:54:42,957] : PID 6185 : DEBUG : polar2grid.core.frontend_utils : write_var_to_flat_binary : Writing binary data for 'Band5Var' to file 'band5_bt.dat' [2016-03-29 10:54:44,089] : PID 6185 : DEBUG : polar2grid.core.fbf : append : 4313 rows in output file [2016-03-29 10:54:44,090] : PID 6185 : DEBUG : polar2grid.core.frontend_utils : write_var_to_flat_binary : File band5_bt.dat has shape (4313, 2048) [2016-03-29 10:54:44,090] : PID 6185 : DEBUG : polar2grid.core.containers : set_persist : Setting persist to False for child 'swath_definition' [2016-03-29 10:54:44,091] : PID 6185 : DEBUG : avhrr2gtiff : main : Backend known grids: None [2016-03-29 10:54:44,091] : PID 6185 : DEBUG : avhrr2gtiff : main : Forced Grids: ['lcc_asia'] [2016-03-29 10:54:44,091] : PID 6185 : DEBUG : avhrr2gtiff : main : Grids that will be mapped to: ['lcc_asia'] [2016-03-29 10:54:44,091] : PID 6185 : INFO : avhrr2gtiff : main : Remapping to grid lcc_asia [2016-03-29 10:54:44,091] : PID 6185 : DEBUG : polar2grid.remap.remap : remap_scene : Remap scene being run with method 'ewa' [2016-03-29 10:54:44,091] : PID 6185 : DEBUG : polar2grid.remap.remap : remap_scene : Running ll2cr on the highest resolution swath to determine if it fits [2016-03-29 10:54:44,091] : PID 6185 : DEBUG : polar2grid.remap.remap : run_ll2cr : Swath '1km_nav' -> Grid 'lcc_asia' [2016-03-29 10:54:44,289] : PID 6185 : DEBUG : polar2grid.remap.ll2cr : ll2cr : Copying longitude and latitude arrays because inplace processing could not be done [2016-03-29 10:54:44,337] : PID 6185 : DEBUG : polar2grid.remap.ll2cr : ll2cr : Running dynamic version of ll2cr... [2016-03-29 10:54:47,875] : PID 6185 : DEBUG : polar2grid.remap.ll2cr : ll2cr : Copying result arrays back to provided inplace array [2016-03-29 10:54:47,928] : PID 6185 : DEBUG : polar2grid.remap.remap : run_ll2cr : Grid information: grid_name: lcc_asia proj4_definition: +proj=lcc +datum=WGS84 +ellps=WGS84 +lat_0=25 +lat_1=25 +lon_0=105 +units=m +no_defs cell_width: 1000.0 cell_height: -1000.0 width: 4944 height: 5652 origin_x: -1518191.32929 origin_y: 4703463.20494 [2016-03-29 10:54:47,928] : PID 6185 : DEBUG : polar2grid.remap.remap : run_ll2cr : Data fits in grid lcc_asia and uses 100.000000% of the swath [2016-03-29 10:54:48,921] : PID 6185 : INFO : polar2grid.remap.remap : remap_scene : Grid information: grid_name: lcc_asia proj4_definition: +proj=lcc +datum=WGS84 +ellps=WGS84 +lat_0=25 +lat_1=25 +lon_0=105 +units=m +no_defs cell_width: 1000.0 cell_height: -1000.0 width: 4944 height: 5652 origin_x: -1518191.32929 origin_y: 4703463.20494 [2016-03-29 10:54:48,921] : PID 6185 : DEBUG : polar2grid.remap.remap : _remap_scene_ewa : Running ll2cr on the geolocation data for the following products: band4_bt band5_bt [2016-03-29 10:54:48,922] : PID 6185 : DEBUG : polar2grid.remap.remap : _remap_scene_ewa : Running fornav for the following products: band4_bt band5_bt [2016-03-29 10:54:48,923] : PID 6185 : DEBUG : polar2grid.remap.fornav : group_iter : Yielding group of size 2 because group size is 2 [2016-03-29 10:54:48,923] : PID 6185 : DEBUG : polar2grid.remap.fornav : fornav : Processing 2 of 2 input arrays [2016-03-29 10:54:54,035] : PID 6185 : DEBUG : polar2grid.remap.remap : _remap_scene_ewa : EWA resampling found 58.952361% of the grid covered for band4_bt [2016-03-29 10:54:54,035] : PID 6185 : DEBUG : polar2grid.remap.remap : _remap_scene_ewa : EWA resampling found 58.952361% of the grid covered for band5_bt [2016-03-29 10:54:54,035] : PID 6185 : DEBUG : polar2grid.remap.remap : _safe_remove : Removing intermediate file 'll2cr_rows_lcc_asia_1km_nav.dat'... [2016-03-29 10:54:54,036] : PID 6185 : DEBUG : polar2grid.remap.remap : _safe_remove : Removing intermediate file 'll2cr_cols_lcc_asia_1km_nav.dat'... [2016-03-29 10:54:54,045] : PID 6185 : INFO : avhrr2gtiff : main : Creating output from data mapped to grid lcc_asia [2016-03-29 10:54:54,045] : PID 6185 : ERROR : polar2grid.core.roles : create_output_from_scene : Could not create output for 'band4_bt' [2016-03-29 10:54:54,045] : PID 6185 : DEBUG : polar2grid.core.roles : create_output_from_scene : Backend exception: Traceback (most recent call last): File "/home/jzh/soft/polar2grid/ShellB3/lib/python2.7/site-packages/polar2grid.core-2.0.0-py2.7.egg/polar2grid/core/roles.py", line 737, in create_output_from_scene output_fn = self.create_output_from_product(gridded_product, _kwargs) File "/home/jzh/soft/polar2grid/ShellB3/lib/python2.7/site-packages/polar2grid-2.0.0-py2.7-linux-x86_64.egg/polar2grid/gtiff_backend.py", line 181, in create_output_from_product etype = np2etype[data_type] KeyError: <type 'numpy.float32'> [2016-03-29 10:54:54,139] : PID 6185 : ERROR : polar2grid.core.roles : create_output_from_scene : Could not create output for 'band5_bt' [2016-03-29 10:54:54,139] : PID 6185 : DEBUG : polar2grid.core.roles : create_output_from_scene : Backend exception: Traceback (most recent call last): File "/home/jzh/soft/polar2grid/ShellB3/lib/python2.7/site-packages/polar2grid.core-2.0.0-py2.7.egg/polar2grid/core/roles.py", line 737, in create_output_from_scene output_fn = self.create_output_from_product(gridded_product, _kwargs) File "/home/jzh/soft/polar2grid/ShellB3/lib/python2.7/site-packages/polar2grid-2.0.0-py2.7-linux-x86_64.egg/polar2grid/gtiff_backend.py", line 181, in create_output_from_product etype = np2etype[data_type] KeyError: <type 'numpy.float32'> [2016-03-29 10:54:54,139] : PID 6185 : INFO : avhrr2gtiff : main : Processing data for grid lcc_asia complete [2016-03-29 10:54:54,139] : PID 6185 : DEBUG : polar2grid.core.containers : cleanup : Removing associated file that is no longer needed: 'band4_bt.dat' [2016-03-29 10:54:54,183] : PID 6185 : DEBUG : polar2grid.core.containers : cleanup : Removing associated file that is no longer needed: 'band5_bt.dat' [2016-03-29 10:54:54,187] : PID 6185 : DEBUG : polar2grid.core.containers : cleanup : Removing associated file that is no longer needed: 'longitude1km.dat' [2016-03-29 10:54:54,240] : PID 6185 : DEBUG : polar2grid.core.containers : cleanup : Removing associated file that is no longer needed: 'latitude1km.dat' [2016-03-29 10:54:54,245] : PID 6185 : DEBUG : polar2grid.core.containers : cleanup : Removing associated file that is no longer needed: 'grid_lcc_asia_band4_bt.dat' [2016-03-29 10:54:54,260] : PID 6185 : DEBUG : polar2grid.core.containers : cleanup : Removing associated file that is no longer needed: 'grid_lcc_asia_band5_bt.dat'

I change the data type of geotiff only because I only want to get the brightness temperature in float type, not the scaled values. And is it possible to get the band radiance not brightness temperature? Thank you !!

djhoese commented 8 years ago

For AVHRR it is not possible to get radiances instead of brightness temperatures right now. After looking at the code it looks like I did not configure the geotiff backend to support float values. You could use the "HDF5" backend via avhrr2hdf5.sh or there is a binary backend but no wrapper script for it. You could call it with this command:

polar2grid/ShellB3/bin/python -m polar2grid.glue avhrr binary ...

This would create a flat binary file for each band.

jiaozhh commented 8 years ago

Ok. Thanks. avhrr2hdf5.sh works. Is there a tool in Polar2Grid that can convert this HDF5 file to a geotiff file? And I hope that the geotiff backend can support float values in the next release.

djhoese commented 8 years ago

Sorry, no there is not a tool like that. What are you doing that requires it to be a geotiff? Geotiffs are usually meant to be an image format and not a data container format (like HDF5). If you are reading these geotiffs in to a program that requires float geotiffs then I can tell you what part of the code to "hack" to add the float functionality, although I haven't tested it yet. For data that isn't meant for an image an HDF5 or binary file are the more traditional choices.

jiaozhh commented 8 years ago

I want first to convert AVHRR data to the geolocated brightness temperature image and than to retrieve the land surface temperature. For the Int8 geotiff files, I have to scale the pixel values to the real brightness temperature, for now I have no idea about that. For the HDF5 brightness temperature image the remote sensing software can not read it as a geolocated image. So These two options make me unconfortable. Please tell me which part I can hack to add float functionality because I can use Python. Thank you very much.

djhoese commented 8 years ago

Sorry I didn't get back to you sooner, I thought I had already answered this. Let me test out some things and I'll let you know what you have to do.

djhoese commented 8 years ago

You should be able to modify the file /home/jzh/soft/polar2grid/ShellB3/lib/python2.7/site-packages/polar2grid-2.0.0-py2.7-linux-x86_64.egg/polar2grid/gtiff_backend.py and modify it the same way I did in this commit: https://github.com/davidh-ssec/polar2grid/commit/f125bb8e376ccb702c4d6a84ddb0ffa9af35976c

What I'm doing in the first change in this commit is adding a mapping from the numpy 32-bit float data type to the GDAL 32-bit float geotiff data type. This allows you to create a float geotiff, however P2G will attempt to scale the data which you don't want so the second set of changes only scales the data if the data type is not floating. Hope that makes sense, let me know if you have questions.

jiaozhh commented 8 years ago

Hi, David. Thank you very much. I had changed the mapping, but was confused by the rescale class. Now I changed the code as your hint, and got the correct results.