Using H5Coro to read a NetCDF file

ashiklom commented 11 months ago

I'm trying to use H5Coro to read some NetCDF-4 (underlying storage is HDF5) data in an S3 bucket. But, I'm running into some errors.

Here's the code:

import srpybin
import os

url = "s3://noaa-goes17/ABI-L2-FDCC/2018/239/00/OR_ABI-L2-FDCC-M3_G17_s20182390052191_e20182390054564_c20182390055159.nc"

asset = "awsodr"      # Dummy variable
resource = os.path.basename(url)
identity = "mystore"  # Dummy variable
driver = "s3"
path = os.path.dirname(url.lstrip("s3://"))
region = "us-east-1"
endpoint = "https://noaa-goes17.s3.amazonaws.com"

h5file = srpybin.h5coro(asset, resource, identity, driver, path, region, endpoint)
# Get metadata for specific variables.
# This works...
h5file.meta("Temp")
h5file.meta("Power")

# Try to read a particular variable.
# This fails with "invalid read location", perhaps because the data are not 
# strictly 2-dimensional and/or there are extra rows.
# Arguments: (Variable name, column, start row, number of rows)
h5file.read("Temp", 0, 0, 1)
h5file.read("Power", 0, 0, 1)

This works:

In [11]: h5file.meta("Power")
Out[11]: 
{'elements': 3750000,
 'typesize': 4,
 'datasize': 15000000,
 'datatype': 'FLOAT',
 'numcols': 2500,
 'numrows': 1500}

But this doesn't:

In [12]: h5file.read("Power", 0, 0, 1)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[12], line 1
----> 1 h5file.read("Power", 0, 0, 1)

RuntimeError: invalid location to read data: 250000, 0 (Power)

I get similar errors trying to read other variables from this file.

The full NetCDF header for this file is included in the Details below. Note that Power is a stricly 2-dimensional variable, so it should work with just a row-col retrieval.

Any thoughts on what's going on?

``` netcdf OR_ABI-L2-FDCC-M3_G17_s20182390002191_e20182390004564_c20182390005163 { dimensions: y = 1500 ; x = 2500 ; number_of_time_bounds = 2 ; band = 1 ; number_of_image_bounds = 2 ; number_of_sunglint_angle_bounds = 2 ; number_of_LZA_bounds = 2 ; number_of_SZA_bounds = 2 ; variables: short Area(y, x) ; Area:_FillValue = -1s ; Area:long_name = "ABI L2+ Fire-Hot Spot Characterization: Fire Area" ; Area:standard_name = "fire_area" ; Area:_Unsigned = "true" ; Area:valid_range = 0s, -6s ; Area:scale_factor = 6.098e-05f ; Area:add_offset = 0.004f ; Area:units = "km2" ; Area:coordinates = "sunglint_angle local_zenith_angle solar_zenith_angle t y x" ; Area:grid_mapping = "goes_imager_projection" ; Area:cell_methods = "sunglint_angle: point (no pixel produced) local_zenith_angle: point (good quality pixel produced) solar_zenith_angle: point (good quality pixel produced) t: point" ; short Temp(y, x) ; Temp:_FillValue = -1s ; Temp:long_name = "ABI L2+ Fire-Hot Spot Characterization: Fire Temperature" ; Temp:standard_name = "fire_temperature" ; Temp:_Unsigned = "true" ; Temp:valid_range = 0s, -6s ; Temp:scale_factor = 0.05493667f ; Temp:add_offset = 400.f ; Temp:units = "K" ; Temp:resolution = "y: 0.000056 rad x: 0.000056 rad" ; Temp:coordinates = "sunglint_angle local_zenith_angle solar_zenith_angle t y x" ; Temp:grid_mapping = "goes_imager_projection" ; Temp:cell_measures = "area: Area" ; Temp:cell_methods = "sunglint_angle: point (no pixel produced) local_zenith_angle: point (good quality pixel produced) solar_zenith_angle: point (good quality pixel produced) t: point" ; Temp:ancillary_variables = "DQF" ; short Mask(y, x) ; Mask:_FillValue = -99s ; Mask:long_name = "ABI L2+ Fire - Hot Spot Characterization: Fire Mask" ; Mask:valid_range = 0s, 245s ; Mask:units = "1" ; Mask:resolution = "y: 0.000056 rad x: 0.000056 rad" ; Mask:coordinates = "sunglint_angle local_zenith_angle solar_zenith_angle t y x" ; Mask:grid_mapping = "goes_imager_projection" ; Mask:cell_methods = "sunglint_angle: point (no pixel produced) local_zenith_angle: point (good quality pixel produced) solar_zenith_angle: point (good quality pixel produced) t: point area: point" ; Mask:flag_values = 0s, 10s, 11s, 12s, 13s, 14s, 15s, 30s, 31s, 32s, 33s, 34s, 35s, 40s, 50s, 60s, 100s, 120s, 121s, 123s, 124s, 125s, 126s , 127s, 150s, 151s, 152s, 153s, 170s, 180s, 182s, 185s, 186s, 187s, 188s, 200s, 205s, 210s, 215s, 220s, 225s, 230s, 240s, 245s ; Mask:flag_meanings = "unprocessed_pixel good_fire_pixel saturated_fire_pixel cloud_contaminated_fire_pixel high_probability_fire_pixel med ium_probability_fire_pixel low_probability_fire_pixel temporally_filtered_good_fire_pixel temporally_filtered_saturated_fire_pixel temporally_filtered_clo ud_contaminated_fire_pixel temporally_filtered_high_probability_fire_pixel temporally_filtered_medium_probability_fire_pixel temporally_filtered_low_proba bility_fire_pixel off_earth_pixel LZA_block_out_zone SZA_or_glint_angle_block_out_zone processed_no_fire_pixel missing_input_3.9um_pixel missing_input_11. 2um_pixel saturated_input_3.9um_pixel saturated_input_11.2um_pixel invalid_input_radiance_value below_threshold_input_3.9um_pixel below_threshold_input_11 .2um_pixel invalid_ecosystem_UMD_land_cover_type_sea_water_or_MODIS_land_mask_types_or_framework_desert_mask_type_bright_desert invalid_ecosystem_USGS_typ e_sea_water invalid_ecosystem_USGS_types_coastline_fringe_or_compound_coastlines invalid_ecosystem_USGS_types_inland_water_or_water_and_island_fringe_or_l and_and_water_shore_or_land_and_water_rivers no_background_value_could_be_computed conversion_error_between_BT_and_radiance conversion_error_radiance_to_a djusted_BT modified_Dozier_technique_bisection_method_invalid_computed_BT modifed_Dozier_technique_Newton_method_invalid_computed_radiance modifed_Dozier_ technique_Newton_method_invalid_computed_fire_brighness_temp modifed_Dozier_technique_Newton_method_invalid_computed_fire_area cloud_pixel_detected_by_11. 2um_threshold_test cloud_pixel_detected_by_negative_difference_3.9um_minus_11.2um_threshold_test cloud_pixel_detected_by_positive_difference_3.9um_minus_1 1.2um_threshold_test cloud_pixel_detected_by_albedo_threshold_test cloud_pixel_detected_by_12.3um_threshold_test cloud_pixel_detected_by_negative_differen ce_11.2um_minus_12.3um_threshold_test cloud_pixel_detected_by_positive_difference_11.2um_minus_12.3um_threshold_test cloud_edge_pixel_detected_by_along_sc an_reflectivity_and_3.9um_threshold_test cloud_edge_pixel_detected_by_along_scan_reflectivity_and_albedo_threshold_test" ; Mask:ancillary_variables = "DQF Temp Power" ; Mask:number_of_fire_categories = 6b ; Mask:percent_good_fire_pixel_or_temporally_filtered_good_fire_pixel = 3.7e-06f ; Mask:percent_saturated_fire_pixel_or_temporally_filtered_saturated_fire_pixel = 0.f ; Mask:percent_cloud_contaminated_fire_pixel_or_temporally_filtered_cloud_contaminated_fire_pixel = 8.e-06f ; Mask:percent_high_probability_fire_pixel_or_temporally_filtered_high_probability_fire_pixel = 5.6e-06f ; Mask:percent_medium_probability_fire_pixel_or_temporally_filtered_medium_probability_fire_pixel = 2.4e-06f ; Mask:percent_low_probability_fire_pixel_or_temporally_filtered_low_probability_fire_pixel = 1.71e-05f ; Mask:invalid_fire_MODIS_land_mask_types_definition = "shallow ocean, ocean coastlines and lake shorelines, deep or shallow inland water, o r moderate, continental, or deep ocean" ; float Power(y, x) ; Power:_FillValue = -9.f ; Power:long_name = "ABI L2+ Fire-Hot Spot Characterization: Fire Radiative Power" ; Power:standard_name = "fire_radiative_power" ; Power:valid_range = 0.f, 200000.f ; Power:units = "MW" ; Power:resolution = "y: 0.000056 rad x: 0.000056 rad" ; Power:coordinates = "sunglint_angle local_zenith_angle solar_zenith_angle t y x" ; Power:grid_mapping = "goes_imager_projection" ; Power:cell_measures = "area: Area" ; Power:cell_methods = "sunglint_angle: point (no pixel produced) local_zenith_angle: point (good quality pixel produced) solar_zenith_angle : point (good quality pixel produced) t: point" ; Power:ancillary_variables = "DQF" ; byte DQF(y, x) ; DQF:_FillValue = -1b ; DQF:long_name = "ABI L2+ Fire - Hot Spot Characterization data quality flags" ; DQF:standard_name = "status_flag" ; DQF:_Unsigned = "true" ; DQF:valid_range = 0b, 5b ; DQF:units = "1" ; DQF:coordinates = "sunglint_angle local_zenith_angle solar_zenith_angle t y x" ; DQF:grid_mapping = "goes_imager_projection" ; DQF:cell_methods = "sunglint_angle: point (no retrieval) local_zenith_angle: point solar_zenith_angle: point t: point area: point" ; DQF:flag_values = 0b, 1b, 2b, 3b, 4b, 5b ; DQF:flag_meanings = "good_quality_fire_pixel_qf good_quality_fire_free_land_pixel_qf invalid_due_to_opaque_cloud_pixel_qf invalid_due_to_s urface_type_or_sunglint_or_LZA_threshold_exceeded_or_off_earth_or_missing_input_data_qf invalid_due_to_bad_input_data_qf invalid_due_to_algorithm_failure_ qf" ; DQF:number_of_qf_values = 6b ; DQF:percent_good_quality_fire_pixel_qf = 3.68e-05f ; DQF:percent_good_fire_free_land_pixel_qf = 0.2822923f ; DQF:percent_invalid_due_to_opaque_cloud_pixel_qf = 0.1895283f ; DQF:percent_invalid_due_to_surface_type_or_sunglint_or_LZA_threshold_exceeded_or_off_earth_or_missing_input_data_qf = 0.5247891f ; DQF:percent_invalid_due_to_bad_input_data_qf = 4.64e-05f ; DQF:percent_invalid_due_to_algorithm_failure_qf = 0.0033072f ; double t ; t:long_name = "J2000 epoch mid-point between the start and end image scan in seconds" ; t:standard_name = "time" ; t:units = "seconds since 2000-01-01 12:00:00" ; t:axis = "T" ; t:bounds = "time_bounds" ; short y(y) ; y:scale_factor = -5.6e-05f ; y:add_offset = 0.126532f ; y:units = "rad" ; y:axis = "Y" ; y:long_name = "GOES fixed grid projection y-coordinate" ; y:standard_name = "projection_y_coordinate" ; short x(x) ; x:scale_factor = 5.6e-05f ; x:add_offset = -0.075012f ; x:units = "rad" ; x:axis = "X" ; x:long_name = "GOES fixed grid projection x-coordinate" ; x:standard_name = "projection_x_coordinate" ; double time_bounds(number_of_time_bounds) ; time_bounds:long_name = "Scan start and end times in seconds since epoch (2000-01-01 12:00:00)" ; int goes_imager_projection ; goes_imager_projection:long_name = "GOES-R ABI fixed grid projection" ; goes_imager_projection:grid_mapping_name = "geostationary" ; goes_imager_projection:perspective_point_height = 35786023. ; goes_imager_projection:semi_major_axis = 6378137. ; goes_imager_projection:semi_minor_axis = 6356752.31414 ; goes_imager_projection:inverse_flattening = 298.2572221 ; goes_imager_projection:latitude_of_projection_origin = 0. ; goes_imager_projection:longitude_of_projection_origin = -89.5 ; goes_imager_projection:sweep_angle_axis = "x" ; float y_image ; y_image:long_name = "GOES-R fixed grid projection y-coordinate center of image" ; y_image:standard_name = "projection_y_coordinate" ; y_image:units = "rad" ; y_image:axis = "Y" ; float y_image_bounds(number_of_image_bounds) ; y_image_bounds:long_name = "GOES-R fixed grid projection y-coordinate north/south extent of image" ; float x_image ; x_image:long_name = "GOES-R fixed grid projection x-coordinate center of image" ; x_image:standard_name = "projection_x_coordinate" ; x_image:units = "rad" ; x_image:axis = "X" ; float x_image_bounds(number_of_image_bounds) ; x_image_bounds:long_name = "GOES-R fixed grid projection x-coordinate west/east extent of image" ; float nominal_satellite_subpoint_lat ; nominal_satellite_subpoint_lat:long_name = "nominal satellite subpoint latitude (platform latitude)" ; nominal_satellite_subpoint_lat:standard_name = "latitude" ; nominal_satellite_subpoint_lat:_FillValue = -999.f ; nominal_satellite_subpoint_lat:units = "degrees_north" ; float nominal_satellite_subpoint_lon ; nominal_satellite_subpoint_lon:long_name = "nominal satellite subpoint longitude (platform longitude)" ; nominal_satellite_subpoint_lon:standard_name = "longitude" ; nominal_satellite_subpoint_lon:_FillValue = -999.f ; nominal_satellite_subpoint_lon:units = "degrees_east" ; float nominal_satellite_height ; nominal_satellite_height:long_name = "nominal satellite height above GRS 80 ellipsoid (platform altitude)" ; nominal_satellite_height:standard_name = "height_above_reference_ellipsoid" ; nominal_satellite_height:_FillValue = -999.f ; nominal_satellite_height:units = "km" ; float geospatial_lat_lon_extent ; geospatial_lat_lon_extent:long_name = "geospatial latitude and longitude references" ; geospatial_lat_lon_extent:geospatial_westbound_longitude = -140.6163f ; geospatial_lat_lon_extent:geospatial_northbound_latitude = 52.76771f ; geospatial_lat_lon_extent:geospatial_eastbound_longitude = -49.17929f ; geospatial_lat_lon_extent:geospatial_southbound_latitude = 14.00016f ; geospatial_lat_lon_extent:geospatial_lat_center = 29.294f ; geospatial_lat_lon_extent:geospatial_lon_center = -91.406f ; geospatial_lat_lon_extent:geospatial_lat_nadir = 0.f ; geospatial_lat_lon_extent:geospatial_lon_nadir = -89.5f ; geospatial_lat_lon_extent:geospatial_lat_units = "degrees_north" ; geospatial_lat_lon_extent:geospatial_lon_units = "degrees_east" ; float sunglint_angle ; sunglint_angle:long_name = "threshold angle between the line of sight to the satellite and the direction of the beam of incident solar rad iation for good quality fire-hot spot characterization data production" ; sunglint_angle:standard_name = "sunglint_angle" ; sunglint_angle:units = "degree" ; sunglint_angle:bounds = "sunglint_angle_bounds" ; float sunglint_angle_bounds(number_of_sunglint_angle_bounds) ; sunglint_angle_bounds:long_name = "sunglint angle degree range where fire-hot spot characterization data is not produced " ; float local_zenith_angle ; local_zenith_angle:long_name = "threshold angle between the line of sight to the satellite and the local zenith at the observation target for good quality fire-hot spot characterization data production" ; local_zenith_angle:standard_name = "platform_zenith_angle" ; local_zenith_angle:units = "degree" ; local_zenith_angle:bounds = "local_zenith_angle_bounds" ; float local_zenith_angle_bounds(number_of_LZA_bounds) ; local_zenith_angle_bounds:long_name = "local zenith angle degree range where good quality fire-hot spot characterization data is produced" ; float solar_zenith_angle ; solar_zenith_angle:long_name = "threshold angle between the line of sight to the sun and the local zenith at the observation target for go od quality fire-hot spot characterization data production" ; solar_zenith_angle:standard_name = "solar_zenith_angle" ; solar_zenith_angle:units = "degree" ; solar_zenith_angle:bounds = "solar_zenith_angle_bounds" ; float solar_zenith_angle_bounds(number_of_SZA_bounds) ; solar_zenith_angle_bounds:long_name = "solar zenith angle degree range where good quality fire-hot spot characterization data is produced" ; int total_number_of_pixels_with_fires_detected ; total_number_of_pixels_with_fires_detected:long_name = "total number of pixels with fires detected" ; total_number_of_pixels_with_fires_detected:_FillValue = -1 ; total_number_of_pixels_with_fires_detected:units = "count" ; total_number_of_pixels_with_fires_detected:coordinates = "sunglint_angle local_zenith_angle solar_zenith_angle t y_image x_image" ; total_number_of_pixels_with_fires_detected:grid_mapping = "goes_imager_projection" ; total_number_of_pixels_with_fires_detected:cell_methods = "sunglint_angle: sum (no pixel produced) local_zenith_angle: sum solar_zenith_an gle: sum t: sum area: sum (interval: 0.000056 rad comment: good quality fire pixels only) where fire over land" ; int total_number_of_pixels_with_fire_temperature ; total_number_of_pixels_with_fire_temperature:long_name = "total number of pixels with fire temperature reported" ; total_number_of_pixels_with_fire_temperature:_FillValue = -1 ; total_number_of_pixels_with_fire_temperature:units = "count" ; total_number_of_pixels_with_fire_temperature:coordinates = "sunglint_angle local_zenith_angle solar_zenith_angle t y_image x_image" ; total_number_of_pixels_with_fire_temperature:grid_mapping = "goes_imager_projection" ; total_number_of_pixels_with_fire_temperature:cell_methods = "sunglint_angle: sum (no pixel produced) local_zenith_angle: sum solar_zenith_ angle: sum t: sum area: sum (interval: 0.000056 rad comment: good quality fire pixels with fire mask values of good_fire_pixel or temporally_filtered_good _fire_pixel only) where fire over land" ; int total_number_of_pixels_with_fire_area ; total_number_of_pixels_with_fire_area:long_name = "total number of pixels with fire area reported" ; total_number_of_pixels_with_fire_area:_FillValue = -1 ; total_number_of_pixels_with_fire_area:units = "count" ; total_number_of_pixels_with_fire_area:coordinates = "sunglint_angle local_zenith_angle solar_zenith_angle t y_image x_image" ; total_number_of_pixels_with_fire_area:grid_mapping = "goes_imager_projection" ; total_number_of_pixels_with_fire_area:cell_methods = "sunglint_angle: sum (no pixel produced) local_zenith_angle: sum solar_zenith_angle: sum t: sum area: sum (interval: 0.000056 rad comment: good quality fire pixels with fire mask values of good_fire_pixel or temporally_filtered_good_fire_p ixel only) where fire over land" ; int total_number_of_pixels_with_fire_radiative_power ; total_number_of_pixels_with_fire_radiative_power:long_name = "total number of pixels with fire radiative power reported" ; total_number_of_pixels_with_fire_radiative_power:_FillValue = -1 ; total_number_of_pixels_with_fire_radiative_power:units = "count" ; total_number_of_pixels_with_fire_radiative_power:coordinates = "sunglint_angle local_zenith_angle solar_zenith_angle t y_image x_image" ; total_number_of_pixels_with_fire_radiative_power:grid_mapping = "goes_imager_projection" ; total_number_of_pixels_with_fire_radiative_power:cell_methods = "sunglint_angle: sum (no pixel produced) local_zenith_angle: sum solar_zen ith_angle: sum t: sum area: sum (interval: 0.000056 rad comment: good quality fire pixels with fire mask values of good_fire_pixel, high_probability_fire_ pixel, medium_probability_fire_pixel, temporally_filtered_good_fire_pixel, temporally_filtered_high_probability_fire_pixel, or temporally_filtered_medium_ probability_fire_pixel where adjacent region has sufficient non-fire, clear pixels only) where fire over land" ; int fire_temperature_outlier_pixel_count ; fire_temperature_outlier_pixel_count:long_name = "number of pixels with fire temperature reported whose value is outside valid measurement range" ; fire_temperature_outlier_pixel_count:_FillValue = -1 ; fire_temperature_outlier_pixel_count:units = "count" ; fire_temperature_outlier_pixel_count:coordinates = "sunglint_angle local_zenith_angle solar_zenith_angle t y_image x_image" ; fire_temperature_outlier_pixel_count:grid_mapping = "goes_imager_projection" ; fire_temperature_outlier_pixel_count:cell_methods = "sunglint_angle: sum (no pixel produced) local_zenith_angle: sum solar_zenith_angle: s um t: sum area: sum (interval: 0.000056 rad comment: good quality fire pixels with fire mask values of good_fire_pixel or temporally_filtered_good_fire_pi xel whose values are outside valid measurement range only) where fire over land" ; int fire_area_outlier_pixel_count ; fire_area_outlier_pixel_count:long_name = "number of pixels with fire area reported whose value is outside valid measurement range" ; fire_area_outlier_pixel_count:_FillValue = -1 ; fire_area_outlier_pixel_count:units = "count" ; fire_area_outlier_pixel_count:coordinates = "sunglint_angle local_zenith_angle solar_zenith_angle t y_image x_image" ; fire_area_outlier_pixel_count:grid_mapping = "goes_imager_projection" ; fire_area_outlier_pixel_count:cell_methods = "sunglint_angle: sum (no pixel produced) local_zenith_angle: sum solar_zenith_angle: sum t: s um area: sum (interval: 0.000056 rad comment: good quality fire pixels with fire mask values of good_fire_pixel or temporally_filtered_good_fire_pixel who se values are outside valid measurement range only) where fire over land" ; int fire_radiative_power_outlier_pixel_count ; fire_radiative_power_outlier_pixel_count:long_name = "number of pixels with fire radiative power reported whose value is outside valid mea surement range" ; fire_radiative_power_outlier_pixel_count:_FillValue = -1 ; fire_radiative_power_outlier_pixel_count:units = "count" ; fire_radiative_power_outlier_pixel_count:coordinates = "sunglint_angle local_zenith_angle solar_zenith_angle t y_image x_image" ; fire_radiative_power_outlier_pixel_count:grid_mapping = "goes_imager_projection" ; fire_radiative_power_outlier_pixel_count:cell_methods = "sunglint_angle: sum (no pixel produced) local_zenith_angle: sum solar_zenith_angl e: sum t: sum area: sum (interval: 0.000056 rad comment: good quality fire pixels with fire mask values of good_fire_pixel, high_probability_fire_pixel, m edium_probability_fire_pixel, temporally_filtered_good_fire_pixel, temporally_filtered_high_probability_fire_pixel, or temporally_filtered_medium_probabil ity_fire_pixel where adjacent region has sufficient non-fire, clear pixels whose values are outside valid measurement range only) where fire over land" ; float minimum_fire_temperature ; minimum_fire_temperature:long_name = "minimum fire temperature" ; minimum_fire_temperature:standard_name = "fire_temperature" ; minimum_fire_temperature:_FillValue = -999.f ; minimum_fire_temperature:valid_range = 600.f, 1200.f ; minimum_fire_temperature:units = "K" ; minimum_fire_temperature:coordinates = "sunglint_angle local_zenith_angle solar_zenith_angle t y_image x_image" ; minimum_fire_temperature:grid_mapping = "goes_imager_projection" ; minimum_fire_temperature:cell_methods = "sunglint_angle: sum (no pixel produced) local_zenith_angle: sum solar_zenith_angle: sum t: sum ar ea: minimum (interval: 0.000056 rad comment: good quality fire pixels with fire mask values of good_fire_pixel or temporally_filtered_good_fire_pixel only ) where fire over land" ; float maximum_fire_temperature ; maximum_fire_temperature:long_name = "maximum fire temperature" ; maximum_fire_temperature:standard_name = "fire_temperature" ; maximum_fire_temperature:_FillValue = -999.f ; maximum_fire_temperature:valid_range = 600.f, 1200.f ; maximum_fire_temperature:units = "K" ; maximum_fire_temperature:coordinates = "sunglint_angle local_zenith_angle solar_zenith_angle t y_image x_image" ; maximum_fire_temperature:grid_mapping = "goes_imager_projection" ; maximum_fire_temperature:cell_methods = "sunglint_angle: sum (no pixel produced) local_zenith_angle: sum solar_zenith_angle: sum t: sum ar ea: maximum (interval: 0.000056 rad comment: good quality fire pixels with fire mask values of good_fire_pixel or temporally_filtered_good_fire_pixel only ) where fire over land" ; float mean_fire_temperature ; mean_fire_temperature:long_name = "mean fire temperature" ; mean_fire_temperature:standard_name = "fire_temperature" ; mean_fire_temperature:_FillValue = -999.f ; mean_fire_temperature:valid_range = 600.f, 1200.f ; mean_fire_temperature:units = "K" ; mean_fire_temperature:coordinates = "sunglint_angle local_zenith_angle solar_zenith_angle t y_image x_image" ; mean_fire_temperature:grid_mapping = "goes_imager_projection" ; mean_fire_temperature:cell_methods = "sunglint_angle: sum (no pixel produced) local_zenith_angle: sum solar_zenith_angle: sum t: sum area: mean (interval: 0.000056 rad comment: good quality fire pixels with fire mask values of good_fire_pixel or temporally_filtered_good_fire_pixel only) wher e fire over land" ; float standard_deviation_fire_temperature ; standard_deviation_fire_temperature:long_name = "standard deviation of fire temperature values" ; standard_deviation_fire_temperature:standard_name = "fire_temperature" ; standard_deviation_fire_temperature:_FillValue = -999.f ; standard_deviation_fire_temperature:units = "K" ; standard_deviation_fire_temperature:coordinates = "sunglint_angle local_zenith_angle solar_zenith_angle t y_image x_image" ; standard_deviation_fire_temperature:grid_mapping = "goes_imager_projection" ; standard_deviation_fire_temperature:cell_methods = "sunglint_angle: sum (no pixel produced) local_zenith_angle: sum solar_zenith_angle: su m t: sum area: standard_deviation (interval: 0.000056 rad comment: good quality fire pixels with fire mask values of good_fire_pixel or temporally_filtere d_good_fire_pixel only) where fire over land" ; float minimum_fire_area ; minimum_fire_area:long_name = "minimum fire area" ; minimum_fire_area:standard_name = "fire_area" ; minimum_fire_area:_FillValue = -999.f ; minimum_fire_area:valid_range = 0.004f, 4.f ; minimum_fire_area:units = "km2" ; minimum_fire_area:coordinates = "sunglint_angle local_zenith_angle solar_zenith_angle t y_image x_image" ; minimum_fire_area:grid_mapping = "goes_imager_projection" ; minimum_fire_area:cell_methods = "sunglint_angle: sum (no pixel produced) local_zenith_angle: sum solar_zenith_angle: sum t: sum area: min imum (interval: 0.000056 rad comment: good quality fire pixels with fire mask values of good_fire_pixel or temporally_filtered_good_fire_pixel only) where fire over land" ; float maximum_fire_area ; maximum_fire_area:long_name = "maximum fire area" ; maximum_fire_area:standard_name = "fire_area" ; maximum_fire_area:_FillValue = -999.f ; maximum_fire_area:valid_range = 0.004f, 4.f ; maximum_fire_area:units = "km2" ; maximum_fire_area:coordinates = "sunglint_angle local_zenith_angle solar_zenith_angle t y_image x_image" ; maximum_fire_area:grid_mapping = "goes_imager_projection" ; maximum_fire_area:cell_methods = "sunglint_angle: sum (no pixel produced) local_zenith_angle: sum solar_zenith_angle: sum t: sum area: max imum (interval: 0.000056 rad comment: good quality fire pixels with fire mask values of good_fire_pixel or temporally_filtered_good_fire_pixel only) where fire over land" ; float mean_fire_area ; mean_fire_area:long_name = "mean fire area" ; mean_fire_area:standard_name = "fire_area" ; mean_fire_area:_FillValue = -999.f ; mean_fire_area:valid_range = 0.004f, 4.f ; mean_fire_area:units = "km2" ; mean_fire_area:coordinates = "sunglint_angle local_zenith_angle solar_zenith_angle t y_image x_image" ; mean_fire_area:grid_mapping = "goes_imager_projection" ; mean_fire_area:cell_methods = "sunglint_angle: sum (no pixel produced) local_zenith_angle: sum solar_zenith_angle: sum t: sum area: mean ( interval: 0.000056 rad comment: good quality fire pixels with fire mask values of good_fire_pixel or temporally_filtered_good_fire_pixel only) where fire over land" ; float standard_deviation_fire_area ; standard_deviation_fire_area:long_name = "standard deviation of fire area values" ; standard_deviation_fire_area:standard_name = "fire_area" ; standard_deviation_fire_area:_FillValue = -999.f ; standard_deviation_fire_area:units = "km2" ; standard_deviation_fire_area:coordinates = "sunglint_angle local_zenith_angle solar_zenith_angle t y_image x_image" ; standard_deviation_fire_area:grid_mapping = "goes_imager_projection" ; standard_deviation_fire_area:cell_methods = "sunglint_angle: sum (no pixel produced) local_zenith_angle: sum solar_zenith_angle: sum t: su m area: standard_deviation (interval: 0.000056 rad comment: good quality fire pixels with fire mask values of good_fire_pixel or temporally_filtered_good_ fire_pixel only) where fire over land" ; float minimum_fire_radiative_power ; minimum_fire_radiative_power:long_name = "minimum fire radiative power" ; minimum_fire_radiative_power:standard_name = "fire_radiative_power" ; minimum_fire_radiative_power:_FillValue = -999.f ; minimum_fire_radiative_power:valid_range = 75.f, 50000.f ; minimum_fire_radiative_power:units = "MW" ; minimum_fire_radiative_power:coordinates = "sunglint_angle local_zenith_angle solar_zenith_angle t y_image x_image" ; minimum_fire_radiative_power:grid_mapping = "goes_imager_projection" ; minimum_fire_radiative_power:cell_methods = "sunglint_angle: sum (no pixel produced) local_zenith_angle: sum solar_zenith_angle: sum t: su m area: minimum (interval: 0.000056 rad comment: good quality fire pixels with fire mask values of good_fire_pixel, high_probability_fire_pixel, medium_pr obability_fire_pixel, temporally_filtered_good_fire_pixel, temporally_filtered_high_probability_fire_pixel, or temporally_filtered_medium_probability_fire _pixel where adjacent region has sufficient non-fire, clear pixels only) where fire over land" ; float maximum_fire_radiative_power ; maximum_fire_radiative_power:long_name = "maximum fire radiative power" ; maximum_fire_radiative_power:standard_name = "fire_radiative_power" ; maximum_fire_radiative_power:_FillValue = -999.f ; maximum_fire_radiative_power:valid_range = 75.f, 50000.f ; maximum_fire_radiative_power:units = "MW" ; maximum_fire_radiative_power:coordinates = "sunglint_angle local_zenith_angle solar_zenith_angle t y_image x_image" ; maximum_fire_radiative_power:grid_mapping = "goes_imager_projection" ; maximum_fire_radiative_power:cell_methods = "sunglint_angle: sum (no pixel produced) local_zenith_angle: sum solar_zenith_angle: sum t: su m area: maximum (interval: 0.000056 rad comment: good quality fire pixels with fire mask values of good_fire_pixel, high_probability_fire_pixel, medium_pr obability_fire_pixel, temporally_filtered_good_fire_pixel, temporally_filtered_high_probability_fire_pixel, or fire temporally_filtered_medium_probability _fire_pixel where adjacent region has sufficient non-fire, clear pixels only) where fire over land" ; float mean_fire_radiative_power ; mean_fire_radiative_power:long_name = "mean fire radiative power" ; mean_fire_radiative_power:standard_name = "fire_radiative_power" ; mean_fire_radiative_power:_FillValue = -999.f ; mean_fire_radiative_power:valid_range = 75.f, 50000.f ; mean_fire_radiative_power:units = "MW" ; mean_fire_radiative_power:coordinates = "sunglint_angle local_zenith_angle solar_zenith_angle t y_image x_image" ; mean_fire_radiative_power:grid_mapping = "goes_imager_projection" ; mean_fire_radiative_power:cell_methods = "sunglint_angle: sum (no pixel produced) local_zenith_angle: sum solar_zenith_angle: sum t: sum a rea: mean (interval: 0.000056 rad comment: good quality fire pixels with fire mask values of good_fire_pixel, high_probability_fire_pixel, medium_probabil ity_fire_pixel, temporally_filtered_good_fire_pixel, temporally_filtered_high_probability_fire_pixel, or temporally_filtered_medium_probability_fire_pixel where adjacent region has sufficient non-fire, clear pixels only) where fire over land" ; float standard_deviation_fire_radiative_power ; standard_deviation_fire_radiative_power:long_name = "standard deviation of fire radiative power values" ; standard_deviation_fire_radiative_power:standard_name = "fire_radiative_power" ; standard_deviation_fire_radiative_power:_FillValue = -999.f ; standard_deviation_fire_radiative_power:units = "MW" ; standard_deviation_fire_radiative_power:coordinates = "sunglint_angle local_zenith_angle solar_zenith_angle t y_image x_image" ; standard_deviation_fire_radiative_power:grid_mapping = "goes_imager_projection" ; standard_deviation_fire_radiative_power:cell_methods = "sunglint_angle: sum (no pixel produced) local_zenith_angle: sum solar_zenith_angle : sum t: sum area: standard_deviation (interval: 0.000056 rad comment: good quality fire pixels with fire mask values of good_fire_pixel, high_probability _fire_pixel, medium_probability_fire_pixel, temporally_filtered_good_fire_pixel, temporally_filtered_high_probability_fire_pixel, or temporally_filtered_m edium_probability_fire_pixel where adjacent region has sufficient non-fire, clear pixels only) where fire over land" ; int algorithm_dynamic_input_data_container ; algorithm_dynamic_input_data_container:long_name = "container for filenames of dynamic algorithm input data" ; algorithm_dynamic_input_data_container:input_ABI_L2_auxiliary_solar_zenith_angle_data = "null" ; algorithm_dynamic_input_data_container:input_ABI_L2_auxiliary_sunglint_angle_data = "null" ; algorithm_dynamic_input_data_container:input_ABI_radiance_band_7 = "OR_ABI-L1b-RADC-M3C07_G17_s20182390002191_e20182390004564_c*.nc" ; algorithm_dynamic_input_data_container:input_ABI_L2_brightness_temperature_band_7_2km_data = "OR_ABI-L2-CMIPC-M3C07_G17_s20182390002191_e2 0182390004564_c*.nc" ; algorithm_dynamic_input_data_container:input_ABI_L2_brightness_temperature_band_14_2km_data = "OR_ABI-L2-CMIPC-M3C14_G17_s20182390002191_e 20182390004564_c*.nc" ; algorithm_dynamic_input_data_container:input_ABI_L2_brightness_temperature_band_15_2km_data = "OR_ABI-L2-CMIPC-M3C15_G17_s20182390002191_e 20182390004564_c*.nc" ; algorithm_dynamic_input_data_container:input_ABI_L2_intermediate_product_reflectance_band_2_2km_data = "null" ; algorithm_dynamic_input_data_container:input_ABI_L2_intermediate_product_time_of_last_fire_data = "null" ; algorithm_dynamic_input_data_container:input_dynamic_ancillary_NWP_total_precipitable_water_data = "null" ; int processing_parm_version_container ; processing_parm_version_container:long_name = "container for processing parameter filenames" ; processing_parm_version_container:L2_processing_parm_version = "OR-PARM-FHS_G17_v01r00.zip OR-PARM-SEMISTATIC_G17_v01r00.zip" ; int algorithm_product_version_container ; algorithm_product_version_container:long_name = "container for algorithm package filename and product version" ; algorithm_product_version_container:algorithm_version = "OR-ALG-COMMON_v01r00.zip, OR-ALG-FHS_v01r00.zip" ; algorithm_product_version_container:product_version = "v01r00" ; float percent_uncorrectable_GRB_errors ; percent_uncorrectable_GRB_errors:long_name = "percent data lost due to uncorrectable GRB errors" ; percent_uncorrectable_GRB_errors:_FillValue = -999.f ; percent_uncorrectable_GRB_errors:valid_range = 0.f, 1.f ; percent_uncorrectable_GRB_errors:units = "percent" ; percent_uncorrectable_GRB_errors:coordinates = "t y_image x_image" ; percent_uncorrectable_GRB_errors:grid_mapping = "goes_imager_projection" ; percent_uncorrectable_GRB_errors:cell_methods = "t: sum area: sum (uncorrectable GRB errors only)" ; float percent_uncorrectable_L0_errors ; percent_uncorrectable_L0_errors:long_name = "percent data lost due to uncorrectable L0 errors" ; percent_uncorrectable_L0_errors:_FillValue = -999.f ; percent_uncorrectable_L0_errors:valid_range = 0.f, 1.f ; percent_uncorrectable_L0_errors:units = "percent" ; percent_uncorrectable_L0_errors:coordinates = "t y_image x_image" ; percent_uncorrectable_L0_errors:grid_mapping = "goes_imager_projection" ; percent_uncorrectable_L0_errors:cell_methods = "t: sum area: sum (uncorrectable L0 errors only)" ; // global attributes: :naming_authority = "gov.nesdis.noaa" ; :Conventions = "CF-1.7" ; :Metadata_Conventions = "Unidata Dataset Discovery v1.0" ; :standard_name_vocabulary = "CF Standard Name Table (v25, 05 July 2013)" ; :institution = "DOC/NOAA/NESDIS > U.S. Department of Commerce, National Oceanic and Atmospheric Administration, National Environmental Sat ellite, Data, and Information Services" ; :project = "GOES" ; :production_site = "NSOF" ; :production_environment = "OE" ; :spatial_resolution = "2km at nadir" ; :orbital_slot = "GOES-Test" ; :platform_ID = "G17" ; :instrument_type = "GOES R Series Advanced Baseline Imager" ; :scene_id = "CONUS" ; :instrument_ID = "FM2" ; :dataset_name = "OR_ABI-L2-FDCC-M3_G17_s20182390002191_e20182390004564_c20182390005163.nc" ; :iso_series_metadata_id = "b015d6f0-b002-11e1-afa6-0800200c9a66" ; :title = "ABI L2 Fire - Hot Spot Characterization" ; :summary = "The Fire - Hot Spot Characterization product consists of a fire mask identifying pixels as one of many fire, non-fire, and obs tructed view categories. In addition, the product consists of fire temperature,radiative power, and area for valid fire pixels that satisfy specific crite ria. This product is generated by utilizing differences in emissive bands with wavelengths 3.9 and 11.2 um to high temperature sub pixel anomalies. Produc t data is generated both day and night." ; :keywords = "HUMAN DIMENSIONS > NATURAL HAZARDS > FIRES " ; :keywords_vocabulary = "NASA Global Change Master Directory (GCMD) Earth Science Keywords, Version 7.0.0.0.0" ; :license = "Unclassified data. Access is restricted to approved users only." ; :processing_level = "National Aeronautics and Space Administration (NASA) L2" ; :date_created = "2018-08-27T00:05:16.3Z" ; :cdm_data_type = "Image" ; :time_coverage_start = "2018-08-27T00:02:19.1Z" ; :time_coverage_end = "2018-08-27T00:04:56.4Z" ; :timeline_id = "ABI Mode 3" ; :production_data_source = "Realtime" ; :id = "0910009f-7111-49f5-855e-a3b96b73afc0" ; } ```

jpswinski commented 11 months ago

Thanks @ashiklom for posting this issue. I am not sure why there is an issue reading the variable - especially if it is able to read the metadata. It looks like I am able to read the bucket on our machines in us-west-2, so I'll see if I can reproduce the problem.

jpswinski commented 4 months ago

@ashiklom, I've finally had some time over the past few months to invest in h5coro and address the issues you've raised here.

The problem was that the h5coro code did not implement all of the necessary specification features, nor did it implement higher dimensionality reads. This has been addressed now in the pure Python implementation of h5coro that can be found at: https://github.com/ICESat2-SlideRule/h5coro. The code now correctly reads and subsets the 2d variables, as well as certain attributes found in NetCDF files.

There is work still to be done which is being led by @ksharonin. She is implementing the updates needed to the C++ version of h5coro and has made progress on implementing features of the HDF5 specification used by certain variable attributes. Her work will need to be pulled into the pure Python implementation in the months to come.

SlideRuleEarth / sliderule

Using H5Coro to read a NetCDF file #292