Open zxdawn opened 4 years ago
Oh, I see. I downloaded the East Asia regional data. And, the earth_equatorial_radius
and something else are all missing .... Should we hard code these things?
Here's the info of regional data:
:file_name = "gk2a_ami_le1b_ir105_ea020lc_201907250530.nc";
:origianl_sourece_file = "gk2a_ami_le1b_ir105_fd020ge_201907250530.nc";
Is this the subset of the full disk, instead of a specific scan mode?
Here's the files I have:
gk2a_ami_le1b_ir087_fd020ge_201901260310.nc
gk2a_ami_le1b_ir096_fd020ge_201901260310.nc
gk2a_ami_le1b_ir105_fd020ge_201901260310.nc
gk2a_ami_le1b_ir112_fd020ge_201901260310.nc
gk2a_ami_le1b_ir123_fd020ge_201901260310.nc
gk2a_ami_le1b_ir133_fd020ge_201901260310.nc
gk2a_ami_le1b_nr013_fd020ge_201901260310.nc
gk2a_ami_le1b_nr016_fd020ge_201901260310.nc
gk2a_ami_le1b_sw038_fd020ge_201901260310.nc
gk2a_ami_le1b_vi004_fd010ge_201901260310.nc
gk2a_ami_le1b_vi005_fd010ge_201901260310.nc
gk2a_ami_le1b_vi006_fd005ge_201901260310.nc
gk2a_ami_le1b_vi008_fd010ge_201901260310.nc
gk2a_ami_le1b_wv063_fd020ge_201901260310.nc
gk2a_ami_le1b_wv069_fd020ge_201901260310.nc
gk2a_ami_le1b_wv073_fd020ge_201901260310.nc
These are full disk and have this:
:file_name = "gk2a_ami_le1b_ir105_fd020ge_201901260310.nc" ;
I'm a little concerned by the number of typos in that "original_source_file" attribute.
Yes. The original files are full disk data. Others are sunsetted regional data. So, shouldn't we care about the Subsetted data provided on their website?
Yes, supporting subsets would be great. I didn't realize it was an official KMA website that was producing files with typos like that. When I wrote the original AMI reader I don't think these files existed. Any idea if the full disk files from this website still work with the current reader? I wonder if they've revised the file scheme and things have changed and need to change in the reader.
The current reader works well for the full disk data provided by the website.
I'm changing this to an enhancement rather than a bug since this is adding functionality that didn't exist before, not necessarily that these files were meant to be supported and aren't.
@zxdawn Any chance that you'll have the time to update the reader to support these files?
@djhoese I can try when I have time. But, these important attributes are missing in the regional data. Should I hard code these in the reader?
@djhoese I can get the start time from the filename, but there's no end time info in the regional data. BTW, I think the start time from the file isn't the exact start time of regional data.
Anyway, the time resolution is 10 minutes. How about setting end time = 10 minutes + start time?
I think we usually make end time equal to start time if end time doesn't exist. From what we know from ABI there could be different modes for AMI in the future where 10 minutes can't be assumed, right? If you make a PR for either one I'll be ok with it. People on the pytroll slack may have other ideas/opinions?
I sent one email to KMA about the missing attrs one month ago:
I found the regional data miss some important attributes, like DN_to_Radiance_Gain and DN_to_Radiance_Offset. Is it possible to complete the attributes of the regional product, just like the full disk data?
They don't reply .... So, do you have any suggestion for this problem?
If they don't intend to update these files then I suppose we have to hard code things but that may be best in a separate file handler (even if they still go in the same reader configuration). I'm also not sure we can guarantee that hard coded values are accurate. If you're saying that the gain/offset attributes are missing and if they aren't in some document as being unchanging then I'm not sure how usable these files are.
Agree with you. I will leave this open in case someone meets this issue in the future.
Hard codes are not quite useful at this stage, especially for international users. Also hope the KMA will update attrs of these regional data.
So what should we do here? We could try sending another email? Would hardcoded parameters make this at least a little usable, or should we just drop this if we don't have an answer?
@mraspaud If you know someone working at KMA, you can try sending an email ;) Otherwise, I prefer dropping this data ...
I don't have any contact there unfortunately. Maybe someone @djhoese knows has a contact?
I have some contacts at KMA. I'll drop them an email, but could someone summarise the problem? Just so I know exactly what needs to be asked.
@simonrp84 Thanks! Here's the summary:
Currently, the AMI reader works well for the full disk data. But, the data provided on the website have other two kinds of data: East Asia and Korea.
For the regional data, they don't save important attributes like DN_to_Radiance_Gain and DN_to_Radiance_Offset. It would be better to include these attributes.
@simonrp84 Thanks. Note that in general these files don't follow the same scheme that their "parent" files do. For example, dimension names are different. Additionally there are typos in the attribute names. But what @zxdawn said is the most important part, we're missing necessary fields to be able to even use the data. I think there were also some navigation/spacecraft information that was missing like Earth radii and height of the satellite.
Ok, thanks both. If I could have a complete list of the important parameters (in addition to the two above) that are missing then that'd be great. Just want to get all the info together so I can send it as one email.
Hi there,
Allow me to help summarise this issue. I'm from Met Service Singapore. The details are as followed.
I'm guessing we can show the difference between the two (as shown below) to KMA and see if they can standardise it?
I have correspondences with KMA NMSC, if @simonrp84 hasn't written in, I could offer to do so.
Thanks.
.nc
from operational ftp. They have full attributes in the .nc
. The extracted attributes are as followed:
netcdf gk2a_ami_le1b_vi004_la010ge_202007160700 {
dimensions:
dim_image_y = 1000 ;
dim_image_x = 1000 ;
dim_boa_swaths = UNLIMITED ; // (0 currently)
dim_1 = 1 ;
dim_inr_perform = 1 ;
dim_vis_stars = UNLIMITED ; // (0 currently)
dim_ir_stars = UNLIMITED ; // (0 currently)
dim_matched_lmks = 1 ;
variables:
ushort image_pixel_values(dim_image_y, dim_image_x) ;
image_pixel_values:channel_name = "VI004" ;
image_pixel_values:detector_side = 2UB ;
image_pixel_values:number_of_total_pixels = 1000000 ;
image_pixel_values:number_of_error_pixels = 0U ;
image_pixel_values:max_pixel_value = 1470US ;
image_pixel_values:min_pixel_value = 113US ;
image_pixel_values:average_pixel_value = 296.991962 ;
image_pixel_values:stddev_pixel_value = 188.788270831084 ;
image_pixel_values:number_of_total_bits_per_pixel = 16 ;
image_pixel_values:number_of_data_quality_flag_bits_per_pixel = 2 ;
image_pixel_values:number_of_valid_bits_per_pixel = 11UB ;
image_pixel_values:data_quality_flag_meaning = "0:good_pixel, 1:conditionally_usable_pixel, 2:out_of_scan_area_pixel, 3:error_pixel" ;
image_pixel_values:ground_sample_distance_ew = 2.8e-05 ;
image_pixel_values:ground_sample_distance_ns = 2.8e-05 ;
ubyte brightness_object_avoidance(dim_boa_swaths) ;
ubyte sc_position(dim_1) ;
sc_position:sc_position_first_pixel = -26112322.5703353, 33102923.9694336, 3735.97626382434 ;
sc_position:sc_position_center_pixel = -26112334.3264007, 33102928.2384297, 3736.22117043814 ;
sc_position:sc_position_last_pixel = -26112346.9119147, 33102932.8050031, 3736.47332088062 ;
sc_position:sc_zenith_angle = 1.00954420896943 ;
sc_position:sc_azimuth_angle = 3.14000220109703 ;
ubyte sun_position(dim_1) ;
sun_position:sun_position_ecef = 33603273368.8888, 137656705777.167, 55156854172.0323 ;
sun_position:sun_zenith_angle = 0.845932095746772 ;
sun_position:sun_azimuth_angle = 4.51160488907156 ;
ubyte moon_position(dim_1) ;
moon_position:moon_position_ecef = 356158389.043127, 125647154.963388, 122056945.64671 ;
ubyte nav_measurement_type(dim_1) ;
uint number_of_inr_performance(dim_1) ;
double nav_average_residual_ew(dim_1) ;
double nav_average_residual_ns(dim_1) ;
double nav_inr_performance_time(dim_inr_perform) ;
double nav_residual_ew(dim_inr_perform) ;
double nav_residual_ns(dim_inr_perform) ;
double nav_max_att_roll(dim_inr_perform) ;
double nav_max_att_pitch(dim_inr_perform) ;
double nav_max_att_yaw(dim_inr_perform) ;
double nav_min_att_roll(dim_inr_perform) ;
double nav_min_att_pitch(dim_inr_perform) ;
double nav_min_att_yaw(dim_inr_perform) ;
double nav_average_att_roll(dim_inr_perform) ;
double nav_average_att_pitch(dim_inr_perform) ;
double nav_average_att_yaw(dim_inr_perform) ;
double nav_stddev_att_roll(dim_inr_perform) ;
double nav_stddev_att_pitch(dim_inr_perform) ;
double nav_stddev_att_yaw(dim_inr_perform) ;
uint number_of_vis_stars(dim_1) ;
double vis_star_measure_time(dim_vis_stars) ;
double vis_star_spectral_radiance(dim_vis_stars) ;
double vis_star_right_ascension(dim_vis_stars) ;
double vis_star_declination(dim_vis_stars) ;
double vis_star_residual_ew(dim_vis_stars) ;
double vis_star_residual_ns(dim_vis_stars) ;
uint number_of_ir_stars(dim_1) ;
double ir_star_measure_time(dim_ir_stars) ;
double ir_star_spectral_radiance(dim_ir_stars) ;
double ir_star_right_ascension(dim_ir_stars) ;
double ir_star_declination(dim_ir_stars) ;
double ir_star_residual_ew(dim_ir_stars) ;
double ir_star_residual_ns(dim_ir_stars) ;
uint number_of_matched_lmk(dim_1) ;
uint matched_lmk_id(dim_matched_lmks) ;
ubyte matched_lmk_channel(dim_matched_lmks) ;
ubyte matched_lmk_chip_type(dim_matched_lmks) ;
double matched_lmk_longitude(dim_matched_lmks) ;
double matched_lmk_latitude(dim_matched_lmks) ;
double matched_lmk_l1a_ew(dim_matched_lmks) ;
double matched_lmk_l1a_ns(dim_matched_lmks) ;
double matched_lmk_residual_ew(dim_matched_lmks) ;
double matched_lmk_residual_ns(dim_matched_lmks) ;
double matched_lmk_accuracy(dim_matched_lmks) ;
double matched_lmk_corr_coeff(dim_matched_lmks) ;
double matched_lmk_sharpness(dim_matched_lmks) ;
double matched_lmk_second_peak(dim_matched_lmks) ;
double matched_lmk_cloud_coverage(dim_matched_lmks) ;
double matched_lmk_chip_cloud(dim_matched_lmks) ;
double matched_lmk_measure_time(dim_matched_lmks) ;
double gsics_coeff_intercept(dim_1) ;
double gsics_coeff_intercept_standard_error(dim_1) ;
double gsics_coeff_slope(dim_1) ;
double gsics_coeff_slope_standard_error(dim_1) ;
double gsics_coeff_quadratic(dim_1) ;
double gsics_coeff_quadratic_standard_error(dim_1) ;
double gsics_coeff_valid_range_upper_limit(dim_1) ;
double gsics_coeff_valid_range_lower_limit(dim_1) ;
double gsics_coeff_start_time_of_validity_period(dim_1) ;
double gsics_coeff_end_time_of_validity_period(dim_1) ;
// global attributes:
:satellite_name = "GK-2A" ;
:instrument_name = "AMI" ;
:data_processing_center = "NMSC" ;
:data_processing_mode = "operation" ;
:channel_spatial_resolution = "1.0" ;
:channel_center_wavelength = "0.47" ;
:scene_acquisition_time = "20200716_070040" ;
:mission_reference_time = "20200716_070000" ;
:file_generation_time = "20200716_070141" ;
:file_name = "gk2a_ami_le1b_vi004_la010ge_202007160700.nc" ;
:file_format_version = "1.0.0_20181120" ;
:geometric_correction_sw_version = "GK2_INRSM_V1.3" ;
:star_catalog_version = "1.0.0_20170303" ;
:landmark_catalog_version = "1.0.0_20190502" ;
:calibration_table_version = "v.3.0_20190415" ;
:DN_to_Radiance_Gain = 0.363545805215835 ;
:DN_to_Radiance_Offset = -7.27090454101562 ;
:Radiance_to_Albedo_c = 0.001558245 ;
:total_pixel_data_size = 2000000 ;
:number_of_total_swaths = 3 ;
:number_of_columns = 1000 ;
:number_of_lines = 1000 ;
:observation_mode = "LA" ;
:observation_start_time = 648154840.13077 ;
:observation_end_time = 648154882.512259 ;
:time_synchro_obt = 648154840.13077 ;
:time_synchro_utc = 648154840.134765 ;
:projection_type = "GEOS" ;
:sub_longitude = 2.23751210105673 ;
:cfac = 40850677.8066787 ;
:lfac = -40850677.8066787 ;
:coff = 627.749996044806 ;
:loff = 4193.00002716269 ;
:nominal_satellite_height = 42164000. ;
:earth_equatorial_radius = 6378137. ;
:earth_polar_radius = 6356752.3 ;
:earth_ellipsoid_center = 0., 0., 0. ;
:image_upperleft_latitude = 0.77494039244198 ;
:image_upperleft_longitude = 2.09224167381294 ;
:image_lowerright_latitude = 0.545849228072878 ;
:image_lowerright_longitude = 2.30772054521243 ;
:image_center_latitude = 0.651892426924154 ;
:image_center_longitude = 2.21147741380357 ;
:image_upperleft_x = -0.0175489998892546 ;
:image_upperleft_y = 0.117376000760555 ;
:image_lowerright_x = 0.0104230001107454 ;
:image_lowerright_y = 0.0894040007605553 ;
:resampling_kernel_type = "SINC" ;
.nc
which are hosted in the KMA archive webserver. I've extracted the attributes as followed. As we can see, there's a long list of key attributes missing. netcdf gk2a_ami_le1b_ir087_ko020lc_202007160000 {
dimensions:
dim_y = 900 ;
dim_x = 900 ;
variables:
ushort image_pixel_values(dim_y, dim_x) ;
image_pixel_values:number_of_total_pixels = 810000 ;
image_pixel_values:average_pixel_value = 5233.72261798246 ;
image_pixel_values:channel_name = "IR087" ;
image_pixel_values:max_pixel_value = 7964US ;
image_pixel_values:min_pixel_value = 3627US ;
image_pixel_values:number_of_total_bits_per_pixel = 16 ;
image_pixel_values:number_of_valid_bits_per_pixel = 13UB ;
// global attributes:
:projection_type = "lambert_conformal_conic" ;
:standard_parallel1 = 30. ;
:standard_parallel2 = 60. ;
:origin_latitude = 38. ;
:central_meridian = 126. ;
:false_easting = 0. ;
:false_northing = 0. ;
:image_width = 900U ;
:image_height = 900U ;
:pixel_size = 2000. ;
:upper_left_easting = -899000. ;
:upper_left_northing = 899000. ;
:upper_right_easting = 899000. ;
:upper_right_northing = 899000. ;
:lower_left_easting = -899000. ;
:lower_left_northing = -899000. ;
:lower_right_easting = 899000. ;
:lower_right_northing = -899000. ;
:_CoordinateTransformType = 0. ;
:_CoordinateAxisTypes = 0. ;
:file_name = "gk2a_ami_le1b_ir087_ko020lc_202007160000.nc" ;
:origianl_sourece_file = "gk2a_ami_le1b_ir087_ela020ge_202007160000.nc" ;
:number_of_columns = 900 ;
:number_of_lines = 900 ;
:total_pixel_data_size = 810000 ;
:channel_center_wavelength = "8.7" ;
:channel_spatial_resolution = "2.0" ;
:data_processing_center = "NMSC" ;
:data_processing_mode = "operation" ;
:file_format_version = "1.0.0_20181120" ;
:instrument_name = "AMI" ;
:satellite_name = "GK-2A" ;
gk2a_ami_le1b_vi004_fd010ge_202007160700.txt gk2a_ami_le1b_ir087_ko020lc_202007160000.txt gk2a_ami_le1b_ir087_ea020lc_202007160000.txt gk2a_ami_le1b_vi004_la010ge_202007160700.txt
Hi @songhan89. I have already written to KMA but have not heard back from them. It would be great if you could also get in touch with your contacts there, perhaps two voices will work better than one!
Describe the bug
dim names
I downloaded the AMI Data from here. But, the dim names of
image_pixel_values
aredim_y
anddim_x
:This is different from the reader: https://github.com/pytroll/satpy/blob/cf9aab464517b3af66f7de1699974bb310d68b0e/satpy/readers/ami_l1b.py#L52
attrs
Here's the output of
self.nc.attrs
:Can't find
observation_start_time
andobservation_end_time
.