search

pytroll / satpy

Python package for earth-observing satellite data processing
http://satpy.readthedocs.org/en/latest/
GNU General Public License v3.0
1.05k stars 290 forks source link

GK-2A AMI reader does not support regional data #1193

Open zxdawn opened 4 years ago

zxdawn commented 4 years ago

Describe the bug

dim names

I downloaded the AMI Data from here. But, the dim names of image_pixel_values are dim_y and dim_x:

ushort image_pixel_values(dim_y=2600, dim_x=3000);
  :number_of_total_pixels = 7800000; // int
  :min_pixel_value = 0.0; // double
  :max_pixel_value = 7828.0; // double
  :average_pixel_value = 4499.87; // double
  :channel_name = "IR105";
  :number_of_total_bits_per_pixel = 16; // int
  :number_of_valid_bits_per_pixel = 13UB; // ubyte
  :_ChunkSizes = 1300U, 1500U; // uint

This is different from the reader: https://github.com/pytroll/satpy/blob/cf9aab464517b3af66f7de1699974bb310d68b0e/satpy/readers/ami_l1b.py#L52

attrs

Here's the output of self.nc.attrs:

{'projection_type': 'lambert_conformal_conic', 'standard_parallel1': 30.0, 'standard_parallel2': 60.0, 'origin_latitude': 38.0, 'central_meridian': 126.0, 'false_easting': 0.0, 'false_northing': 0.0, 'image_width': 3000, 'image_height': 2600, 'pixel_size': 2000.0, 'upper_left_easting': -2999000.0, 'upper_left_northing': 2599000.0, 'upper_right_easting': 2999000.0, 'upper_right_northing': 2599000.0, 'lower_left_easting': -2999000.0, 'lower_left_northing': -2599000.0, 'lower_right_easting': 2999000.0, 'lower_right_northing': -2599000.0, '_CoordinateTransformType': 0.0, '_CoordinateAxisTypes': 0.0, 'file_name': 'gk2a_ami_le1b_ir105_ea020lc_201907250710.nc', 'origianl_sourece_file': 'gk2a_ami_le1b_ir105_fd020ge_201907250710.nc', 'number_of_columns': 3000, 'number_of_lines': 2600, 'total_pixel_data_size': 7800000, 'channel_center_wavelength': '10.5', 'channel_spatial_resolution': '2.0', 'data_processing_center': 'NMSC', 'data_processing_mode': 'operation', 'file_format_version': '1.0.0_20181120', 'instrument_name': 'AMI', 'satellite_name': 'GK-2A'}

Can't find observation_start_time and observation_end_time.

zxdawn commented 4 years ago

Oh, I see. I downloaded the East Asia regional data. And, the earth_equatorial_radius and something else are all missing .... Should we hard code these things?

zxdawn commented 4 years ago

Here's the info of regional data:

:file_name = "gk2a_ami_le1b_ir105_ea020lc_201907250530.nc";
:origianl_sourece_file = "gk2a_ami_le1b_ir105_fd020ge_201907250530.nc";

Is this the subset of the full disk, instead of a specific scan mode?

djhoese commented 4 years ago

Here's the files I have:

gk2a_ami_le1b_ir087_fd020ge_201901260310.nc
gk2a_ami_le1b_ir096_fd020ge_201901260310.nc
gk2a_ami_le1b_ir105_fd020ge_201901260310.nc
gk2a_ami_le1b_ir112_fd020ge_201901260310.nc
gk2a_ami_le1b_ir123_fd020ge_201901260310.nc
gk2a_ami_le1b_ir133_fd020ge_201901260310.nc
gk2a_ami_le1b_nr013_fd020ge_201901260310.nc
gk2a_ami_le1b_nr016_fd020ge_201901260310.nc
gk2a_ami_le1b_sw038_fd020ge_201901260310.nc
gk2a_ami_le1b_vi004_fd010ge_201901260310.nc
gk2a_ami_le1b_vi005_fd010ge_201901260310.nc
gk2a_ami_le1b_vi006_fd005ge_201901260310.nc
gk2a_ami_le1b_vi008_fd010ge_201901260310.nc
gk2a_ami_le1b_wv063_fd020ge_201901260310.nc
gk2a_ami_le1b_wv069_fd020ge_201901260310.nc
gk2a_ami_le1b_wv073_fd020ge_201901260310.nc

These are full disk and have this:

                :file_name = "gk2a_ami_le1b_ir105_fd020ge_201901260310.nc" ;

I'm a little concerned by the number of typos in that "original_source_file" attribute.

zxdawn commented 4 years ago

Yes. The original files are full disk data. Others are sunsetted regional data. So, shouldn't we care about the Subsetted data provided on their website?

djhoese commented 4 years ago

Yes, supporting subsets would be great. I didn't realize it was an official KMA website that was producing files with typos like that. When I wrote the original AMI reader I don't think these files existed. Any idea if the full disk files from this website still work with the current reader? I wonder if they've revised the file scheme and things have changed and need to change in the reader.

zxdawn commented 4 years ago

The current reader works well for the full disk data provided by the website.

djhoese commented 4 years ago

I'm changing this to an enhancement rather than a bug since this is adding functionality that didn't exist before, not necessarily that these files were meant to be supported and aren't.

@zxdawn Any chance that you'll have the time to update the reader to support these files?

zxdawn commented 4 years ago

@djhoese I can try when I have time. But, these important attributes are missing in the regional data. Should I hard code these in the reader?

djhoese commented 4 years ago
  1. Email their people and tell them about the missing attributes and the typo for "original_source_file".
  2. start/end time you can probably make up from the filename
  3. Radius parameters... :fearful: hard code them if not available?
zxdawn commented 4 years ago

@djhoese I can get the start time from the filename, but there's no end time info in the regional data. BTW, I think the start time from the file isn't the exact start time of regional data.

Anyway, the time resolution is 10 minutes. How about setting end time = 10 minutes + start time?

djhoese commented 4 years ago

I think we usually make end time equal to start time if end time doesn't exist. From what we know from ABI there could be different modes for AMI in the future where 10 minutes can't be assumed, right? If you make a PR for either one I'll be ok with it. People on the pytroll slack may have other ideas/opinions?

zxdawn commented 4 years ago

I sent one email to KMA about the missing attrs one month ago:

I found the regional data miss some important attributes, like DN_to_Radiance_Gain and DN_to_Radiance_Offset. Is it possible to complete the attributes of the regional product, just like the full disk data?

They don't reply .... So, do you have any suggestion for this problem?

djhoese commented 4 years ago

If they don't intend to update these files then I suppose we have to hard code things but that may be best in a separate file handler (even if they still go in the same reader configuration). I'm also not sure we can guarantee that hard coded values are accurate. If you're saying that the gain/offset attributes are missing and if they aren't in some document as being unchanging then I'm not sure how usable these files are.

zxdawn commented 4 years ago

Agree with you. I will leave this open in case someone meets this issue in the future.

Hard codes are not quite useful at this stage, especially for international users. Also hope the KMA will update attrs of these regional data.

mraspaud commented 4 years ago

So what should we do here? We could try sending another email? Would hardcoded parameters make this at least a little usable, or should we just drop this if we don't have an answer?

zxdawn commented 4 years ago

@mraspaud If you know someone working at KMA, you can try sending an email ;) Otherwise, I prefer dropping this data ...

mraspaud commented 4 years ago

I don't have any contact there unfortunately. Maybe someone @djhoese knows has a contact?

simonrp84 commented 4 years ago

I have some contacts at KMA. I'll drop them an email, but could someone summarise the problem? Just so I know exactly what needs to be asked.

zxdawn commented 4 years ago

@simonrp84 Thanks! Here's the summary:

Currently, the AMI reader works well for the full disk data. But, the data provided on the website have other two kinds of data: East Asia and Korea.

For the regional data, they don't save important attributes like DN_to_Radiance_Gain and DN_to_Radiance_Offset. It would be better to include these attributes.

djhoese commented 4 years ago

@simonrp84 Thanks. Note that in general these files don't follow the same scheme that their "parent" files do. For example, dimension names are different. Additionally there are typos in the attribute names. But what @zxdawn said is the most important part, we're missing necessary fields to be able to even use the data. I think there were also some navigation/spacecraft information that was missing like Earth radii and height of the satellite.

simonrp84 commented 4 years ago

Ok, thanks both. If I could have a complete list of the important parameters (in addition to the two above) that are missing then that'd be great. Just want to get all the info together so I can send it as one email.

songhan89 commented 4 years ago

Hi there,

Allow me to help summarise this issue. I'm from Met Service Singapore. The details are as followed.

I'm guessing we can show the difference between the two (as shown below) to KMA and see if they can standardise it?

I have correspondences with KMA NMSC, if @simonrp84 hasn't written in, I could offer to do so.

Thanks.

  1. First, i checked the FD and LA .nc from operational ftp. They have full attributes in the .nc.

The extracted attributes are as followed:

netcdf gk2a_ami_le1b_vi004_la010ge_202007160700 {
dimensions:
    dim_image_y = 1000 ;
    dim_image_x = 1000 ;
    dim_boa_swaths = UNLIMITED ; // (0 currently)
    dim_1 = 1 ;
    dim_inr_perform = 1 ;
    dim_vis_stars = UNLIMITED ; // (0 currently)
    dim_ir_stars = UNLIMITED ; // (0 currently)
    dim_matched_lmks = 1 ;
variables:
    ushort image_pixel_values(dim_image_y, dim_image_x) ;
        image_pixel_values:channel_name = "VI004" ;
        image_pixel_values:detector_side = 2UB ;
        image_pixel_values:number_of_total_pixels = 1000000 ;
        image_pixel_values:number_of_error_pixels = 0U ;
        image_pixel_values:max_pixel_value = 1470US ;
        image_pixel_values:min_pixel_value = 113US ;
        image_pixel_values:average_pixel_value = 296.991962 ;
        image_pixel_values:stddev_pixel_value = 188.788270831084 ;
        image_pixel_values:number_of_total_bits_per_pixel = 16 ;
        image_pixel_values:number_of_data_quality_flag_bits_per_pixel = 2 ;
        image_pixel_values:number_of_valid_bits_per_pixel = 11UB ;
        image_pixel_values:data_quality_flag_meaning = "0:good_pixel, 1:conditionally_usable_pixel, 2:out_of_scan_area_pixel, 3:error_pixel" ;
        image_pixel_values:ground_sample_distance_ew = 2.8e-05 ;
        image_pixel_values:ground_sample_distance_ns = 2.8e-05 ;
    ubyte brightness_object_avoidance(dim_boa_swaths) ;
    ubyte sc_position(dim_1) ;
        sc_position:sc_position_first_pixel = -26112322.5703353, 33102923.9694336, 3735.97626382434 ;
        sc_position:sc_position_center_pixel = -26112334.3264007, 33102928.2384297, 3736.22117043814 ;
        sc_position:sc_position_last_pixel = -26112346.9119147, 33102932.8050031, 3736.47332088062 ;
        sc_position:sc_zenith_angle = 1.00954420896943 ;
        sc_position:sc_azimuth_angle = 3.14000220109703 ;
    ubyte sun_position(dim_1) ;
        sun_position:sun_position_ecef = 33603273368.8888, 137656705777.167, 55156854172.0323 ;
        sun_position:sun_zenith_angle = 0.845932095746772 ;
        sun_position:sun_azimuth_angle = 4.51160488907156 ;
    ubyte moon_position(dim_1) ;
        moon_position:moon_position_ecef = 356158389.043127, 125647154.963388, 122056945.64671 ;
    ubyte nav_measurement_type(dim_1) ;
    uint number_of_inr_performance(dim_1) ;
    double nav_average_residual_ew(dim_1) ;
    double nav_average_residual_ns(dim_1) ;
    double nav_inr_performance_time(dim_inr_perform) ;
    double nav_residual_ew(dim_inr_perform) ;
    double nav_residual_ns(dim_inr_perform) ;
    double nav_max_att_roll(dim_inr_perform) ;
    double nav_max_att_pitch(dim_inr_perform) ;
    double nav_max_att_yaw(dim_inr_perform) ;
    double nav_min_att_roll(dim_inr_perform) ;
    double nav_min_att_pitch(dim_inr_perform) ;
    double nav_min_att_yaw(dim_inr_perform) ;
    double nav_average_att_roll(dim_inr_perform) ;
    double nav_average_att_pitch(dim_inr_perform) ;
    double nav_average_att_yaw(dim_inr_perform) ;
    double nav_stddev_att_roll(dim_inr_perform) ;
    double nav_stddev_att_pitch(dim_inr_perform) ;
    double nav_stddev_att_yaw(dim_inr_perform) ;
    uint number_of_vis_stars(dim_1) ;
    double vis_star_measure_time(dim_vis_stars) ;
    double vis_star_spectral_radiance(dim_vis_stars) ;
    double vis_star_right_ascension(dim_vis_stars) ;
    double vis_star_declination(dim_vis_stars) ;
    double vis_star_residual_ew(dim_vis_stars) ;
    double vis_star_residual_ns(dim_vis_stars) ;
    uint number_of_ir_stars(dim_1) ;
    double ir_star_measure_time(dim_ir_stars) ;
    double ir_star_spectral_radiance(dim_ir_stars) ;
    double ir_star_right_ascension(dim_ir_stars) ;
    double ir_star_declination(dim_ir_stars) ;
    double ir_star_residual_ew(dim_ir_stars) ;
    double ir_star_residual_ns(dim_ir_stars) ;
    uint number_of_matched_lmk(dim_1) ;
    uint matched_lmk_id(dim_matched_lmks) ;
    ubyte matched_lmk_channel(dim_matched_lmks) ;
    ubyte matched_lmk_chip_type(dim_matched_lmks) ;
    double matched_lmk_longitude(dim_matched_lmks) ;
    double matched_lmk_latitude(dim_matched_lmks) ;
    double matched_lmk_l1a_ew(dim_matched_lmks) ;
    double matched_lmk_l1a_ns(dim_matched_lmks) ;
    double matched_lmk_residual_ew(dim_matched_lmks) ;
    double matched_lmk_residual_ns(dim_matched_lmks) ;
    double matched_lmk_accuracy(dim_matched_lmks) ;
    double matched_lmk_corr_coeff(dim_matched_lmks) ;
    double matched_lmk_sharpness(dim_matched_lmks) ;
    double matched_lmk_second_peak(dim_matched_lmks) ;
    double matched_lmk_cloud_coverage(dim_matched_lmks) ;
    double matched_lmk_chip_cloud(dim_matched_lmks) ;
    double matched_lmk_measure_time(dim_matched_lmks) ;
    double gsics_coeff_intercept(dim_1) ;
    double gsics_coeff_intercept_standard_error(dim_1) ;
    double gsics_coeff_slope(dim_1) ;
    double gsics_coeff_slope_standard_error(dim_1) ;
    double gsics_coeff_quadratic(dim_1) ;
    double gsics_coeff_quadratic_standard_error(dim_1) ;
    double gsics_coeff_valid_range_upper_limit(dim_1) ;
    double gsics_coeff_valid_range_lower_limit(dim_1) ;
    double gsics_coeff_start_time_of_validity_period(dim_1) ;
    double gsics_coeff_end_time_of_validity_period(dim_1) ;

// global attributes:
        :satellite_name = "GK-2A" ;
        :instrument_name = "AMI" ;
        :data_processing_center = "NMSC" ;
        :data_processing_mode = "operation" ;
        :channel_spatial_resolution = "1.0" ;
        :channel_center_wavelength = "0.47" ;
        :scene_acquisition_time = "20200716_070040" ;
        :mission_reference_time = "20200716_070000" ;
        :file_generation_time = "20200716_070141" ;
        :file_name = "gk2a_ami_le1b_vi004_la010ge_202007160700.nc" ;
        :file_format_version = "1.0.0_20181120" ;
        :geometric_correction_sw_version = "GK2_INRSM_V1.3" ;
        :star_catalog_version = "1.0.0_20170303" ;
        :landmark_catalog_version = "1.0.0_20190502" ;
        :calibration_table_version = "v.3.0_20190415" ;
        :DN_to_Radiance_Gain = 0.363545805215835 ;
        :DN_to_Radiance_Offset = -7.27090454101562 ;
        :Radiance_to_Albedo_c = 0.001558245 ;
        :total_pixel_data_size = 2000000 ;
        :number_of_total_swaths = 3 ;
        :number_of_columns = 1000 ;
        :number_of_lines = 1000 ;
        :observation_mode = "LA" ;
        :observation_start_time = 648154840.13077 ;
        :observation_end_time = 648154882.512259 ;
        :time_synchro_obt = 648154840.13077 ;
        :time_synchro_utc = 648154840.134765 ;
        :projection_type = "GEOS" ;
        :sub_longitude = 2.23751210105673 ;
        :cfac = 40850677.8066787 ;
        :lfac = -40850677.8066787 ;
        :coff = 627.749996044806 ;
        :loff = 4193.00002716269 ;
        :nominal_satellite_height = 42164000. ;
        :earth_equatorial_radius = 6378137. ;
        :earth_polar_radius = 6356752.3 ;
        :earth_ellipsoid_center = 0., 0., 0. ;
        :image_upperleft_latitude = 0.77494039244198 ;
        :image_upperleft_longitude = 2.09224167381294 ;
        :image_lowerright_latitude = 0.545849228072878 ;
        :image_lowerright_longitude = 2.30772054521243 ;
        :image_center_latitude = 0.651892426924154 ;
        :image_center_longitude = 2.21147741380357 ;
        :image_upperleft_x = -0.0175489998892546 ;
        :image_upperleft_y = 0.117376000760555 ;
        :image_lowerright_x = 0.0104230001107454 ;
        :image_lowerright_y = 0.0894040007605553 ;
        :resampling_kernel_type = "SINC" ;
  1. As @zxdawn rightly pointed out, the problem lies with the EA and KO .nc which are hosted in the KMA archive webserver. I've extracted the attributes as followed. As we can see, there's a long list of key attributes missing. 
netcdf gk2a_ami_le1b_ir087_ko020lc_202007160000 {
dimensions:
    dim_y = 900 ;
    dim_x = 900 ;
variables:
    ushort image_pixel_values(dim_y, dim_x) ;
        image_pixel_values:number_of_total_pixels = 810000 ;
        image_pixel_values:average_pixel_value = 5233.72261798246 ;
        image_pixel_values:channel_name = "IR087" ;
        image_pixel_values:max_pixel_value = 7964US ;
        image_pixel_values:min_pixel_value = 3627US ;
        image_pixel_values:number_of_total_bits_per_pixel = 16 ;
        image_pixel_values:number_of_valid_bits_per_pixel = 13UB ;

// global attributes:
        :projection_type = "lambert_conformal_conic" ;
        :standard_parallel1 = 30. ;
        :standard_parallel2 = 60. ;
        :origin_latitude = 38. ;
        :central_meridian = 126. ;
        :false_easting = 0. ;
        :false_northing = 0. ;
        :image_width = 900U ;
        :image_height = 900U ;
        :pixel_size = 2000. ;
        :upper_left_easting = -899000. ;
        :upper_left_northing = 899000. ;
        :upper_right_easting = 899000. ;
        :upper_right_northing = 899000. ;
        :lower_left_easting = -899000. ;
        :lower_left_northing = -899000. ;
        :lower_right_easting = 899000. ;
        :lower_right_northing = -899000. ;
        :_CoordinateTransformType = 0. ;
        :_CoordinateAxisTypes = 0. ;
        :file_name = "gk2a_ami_le1b_ir087_ko020lc_202007160000.nc" ;
        :origianl_sourece_file = "gk2a_ami_le1b_ir087_ela020ge_202007160000.nc" ;
        :number_of_columns = 900 ;
        :number_of_lines = 900 ;
        :total_pixel_data_size = 810000 ;
        :channel_center_wavelength = "8.7" ;
        :channel_spatial_resolution = "2.0" ;
        :data_processing_center = "NMSC" ;
        :data_processing_mode = "operation" ;
        :file_format_version = "1.0.0_20181120" ;
        :instrument_name = "AMI" ;
        :satellite_name = "GK-2A" ;

gk2a_ami_le1b_vi004_fd010ge_202007160700.txt gk2a_ami_le1b_ir087_ko020lc_202007160000.txt gk2a_ami_le1b_ir087_ea020lc_202007160000.txt gk2a_ami_le1b_vi004_la010ge_202007160700.txt

simonrp84 commented 4 years ago

Hi @songhan89. I have already written to KMA but have not heard back from them. It would be great if you could also get in touch with your contacts there, perhaps two voices will work better than one!