search

BrandonSmithJ / MDN

Mixture Density Network for water constituent estimation
GNU General Public License v3.0
40 stars 34 forks source link

Cannot find model weight #20

Open stanleesocca opened 3 months ago

stanleesocca commented 3 months ago

Hi @BrandonSmithJ , Thanks for the package. I was trying some of the examples here to see if it fit my use case. I notice that the program fails to run despite having successfully cloned your repo. My assumption is that the files in the Weights folder contains the model's pre-trained weights. Am I right?

I got this example when trying the random numpy example. 

from MDN import image_estimates, get_tile_data, get_sensor_bands
import numpy as np
random_data = np.random.rand(3, 3, len(get_sensor_bands(sensor)))
sensor = 'MSI'
random_data = np.random.rand(3, 3, len(get_sensor_bands(sensor)))
chla, idxs  = image_estimates(random_data, sensor=sensor)

and a snippet of the error:

  File "<stdin>", line 1, in <module>
  File "/export/lv9/user/smor/lter_life/MDN/product_estimation.py", line 174, in image_estimates
    estimate = function(im_data, sensor=sensor, **kwargs) if im_data.size else np.zeros((0, 1))
  File "/export/lv9/user/smor/lter_life/MDN/product_estimation.py", line 138, in apply_model
    preds, idxs = get_estimates(args, x_test=x_test)
  File "/export/lv9/user/smor/lter_life/MDN/product_estimation.py", line 45, in get_estimates
    model_path = generate_config(args, create=x_train is not None)
  File "/export/lv9/user/smor/lter_life/MDN/utils.py", line 349, in generate_config
    except: raise Exception('Weights have not yet been downloaded - please run "git lfs install" and then "git lfs pull" inside the repository folder')
Exception: Weights have not yet been downloaded - please run "git lfs install" and then "git lfs pull" inside the repository folder

I see that other issue like #19 already posed some similar issue. Can you help solve this issue.

Thanks

BrandonSmithJ commented 3 months ago

Yes, the weights directory contain the pre-trained model weights. The problem is that github LFS only works for about the first week of every month due to bots downloading the repo and exhausting the limited free LFS capacity. The issue you referenced is actually a separate problem though - what you want to look at is #14, in which I provide direct download links to the weight zip files.

Once you download the weight zip files for the models you want, just unzip them in the Weights directory and everything should work as expected.

BrandonSmithJ commented 3 months ago

It's been on my todo list for awhile to stop using LFS, so I went ahead and updated the repository to host the zip files directly. You should now be able to clone it and run the example without worrying about downloading the weights separately. Let me know if you run into any issues after re-cloning the repo.

stanleesocca commented 3 months ago

Ok. The test case with numpy example work perfectly. Thanks for that.

I notice also that get_tile_data function does not work well with ncdf data from acolite. I tried it with an output a sentinel-2 acolite L2R processed file, and it fails to find the Rrs. 

>>> atmp = "./app_acolite/output/sentinels/S2A_MSIL1C_20220602/S2A_MSI_2022_06_02_11_06_23_T31UFU_L2R.nc"
>>> band, Rrs = get_tile_data(atmp, sensor, allow_neg=False)

Here is the error log. 

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
 assert(len(data) == len(features)), f'Missing features: Found {list(data.keys())}, Expecting {features}'
AssertionError: Missing features: Found [], Expecting ['Rrs']

Have you run into this type of error before? I see that in the paper, acolite was part of the atmospheric pre-processor tool used.

Thanks very much for your time.

BrandonSmithJ commented 3 months ago

I've only tested with L2W, which it does work for. L2W files have the following variables (e.g. for OLI): 

variables(dimensions): float32 lon(y, x), float32 lat(y, x), int32 l2_flags(
y, x), float32 Rrs_443(y, x), float32 Rrs_483(y, x), float32 Rrs_561(y, x), floa
t32 Rrs_655(y, x), float32 Rrs_865(y, x), float32 Rrs_1609(y, x), float32 Rrs_22
01(y, x)

Can you tell me what the netCDF variables look like?

from netCDF4 import Dataset
print( Dataset(atmp) )
stanleesocca commented 3 months ago

I changed it to the L2W file, and same error:

With print(Dataset(atmp)), I get this: 

>>> print(Dataset(atmp))
<class 'netCDF4._netCDF4.Dataset'>
root group (NETCDF4 data model, file format HDF5):
    generated_by: ACOLITE
    generated_on: 2024-03-22 11:43:41 CET
    contact: Quinten Vanhellemont
    product_type: NetCDF
    metadata_profile: beam
    metadata_version: 0.5
    auto_grouping: rhot:rhorc:rhos:rhow:Rrs:Lt
    Conventions: CF-1.7
    sensor: S2A_MSI
    isodate: 2022-06-02T11:06:23.464274+00:00
    global_dims: [3386 2099]
    sza: 31.0679062389307
    vza: 11.263176190476191
    raa: 129.46760309375418
    vaa: 295.7634761904762
    saa: 166.295873096722
    se_distance: 1.0141500310042952
    mus: 0.856556282163115
    granule: L1C_T31UFU_A036270_20220602T105627
    mgrs_tile: T31UFU
    acolite_file_type: L2R
    tile_code: T31UFU
    oname: S2A_MSI_2022_06_02_11_06_23_T31UFU
    ofile: ../app_acolite/output/sentinels/S2A_MSIL1C_20220602/S2A_MSI_2022_06_02_11_06_23_T31UFU_L2W.nc
    1_wave: 442.68570115396517
    1_name: 443
    1_f0: 19054.057897300103
    2_wave: 492.43738054332624
    2_name: 492
    2_f0: 19816.582596730874
    3_wave: 559.8501892469018
    3_name: 560
    3_f0: 18712.647798388396
    4_wave: 664.619908601124
    4_name: 665
    4_f0: 15324.95308599693
    5_wave: 704.1159402425621
    5_name: 704
    5_f0: 14122.137496285726
    6_wave: 740.478761786033
    6_name: 740
    6_f0: 12885.020009014514
    7_wave: 782.7500096627707
    7_name: 783
    7_f0: 11782.31866822806
    8_wave: 832.7883239563182
    8_name: 833
    8_f0: 10446.745777714672
    8A_wave: 864.7104683919092
    8A_name: 865
    8A_f0: 9505.78414767759
    9_wave: 945.0507000978324
    9_name: 945
    9_f0: 8123.418928961382
    10_wave: 1373.4620165399258
    10_name: 1373
    10_f0: 3585.7816638508825
    11_wave: 1613.659319714372
    11_name: 1614
    11_f0: 2368.9992957468658
    12_wave: 2202.3666356190934
    12_name: 2202
    12_f0: 812.0749465544842
    scene_xrange: [600000. 709800.]
    scene_yrange: [5900040. 5790240.]
    scene_proj4_string: +proj=utm +zone=31 +datum=WGS84 +units=m +no_defs
    scene_pixel_size: [ 10. -10.]
    scene_dims: [10980 10980]
    sub: [ 712 1496 2099 3386]
    limit: [52.8  4.6 53.1  4.9]
    xrange: [607120. 628110.]
    yrange: [5885080. 5851220.]
    proj4_string: +proj=utm +zone=31 +datum=WGS84 +units=m +no_defs
    pixel_size: [ 10. -10.]
    projection_key: transverse_mercator
    data_dimensions: [3386 2099]
    data_elements: 7107214
    uoz: 0.3
    uwv: 1.5
    wind: 2.0
    pressure: 1013.25
    acolite_version: Generic GitHub Clone c2024-03-07T17:08:12
    copy_datasets: ['lon', 'lat', 'sza', 'vza', 'raa', 'rhot_*']
    s2_write_vaa: False
    s2_write_saa: False
    dsf_aot_estimate: tiled
    dsf_spectrum_option: intercept
    dsf_wave_range: [400 900]
    s2_target_res: 10
    resolved_geometry: True
    gains_toa: [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
    offsets_toa: [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
    dsf_intercept_pixels: 200
    dsf_tile_dimensions: [600 600]
    geometry_per_band: False
    geometry_fixed_footprint: False
    luts_reduce_dimensions: True
    region_name:
    polygon_limit: True
    polylakes: False
    polylakes_database: worldlakes
    merge_tiles: False
    merge_zones: True
    extend_region: False
    l1r_crop: False
    output: ../app_acolite/output/sentinels/S2A_MSIL1C_20220602
    output_geolocation: True
    output_xy: False
    output_geometry: True
    output_rhorc: False
    add_band_name: False
    compute_contrabands: False
    netcdf_projection: True
    netcdf_compression: False
    netcdf_compression_level: 4
    netcdf_discretisation: False
    landsat_qa_bands: ['PIXEL', 'RADSAT', 'SATURATION']
    landsat_qa_output: False
    s2_auxiliary_include: False
    s2_auxiliary_project: False
    s2_auxiliary_default: False
    s2_auxiliary_interpolate: True
    s2_write_dfoo: False
    s2_dilate_blackfill: False
    s2_dilate_blackfill_iterations: 2
    geometry_type: grids_footprint
    geometry_res: 60
    geometry_override: False
    smile_correction: True
    smile_correction_tgas: True
    use_tpg: True
    use_supplied_ancillary: True
    worldview_reproject: False
    worldview_reproject_resolution: 2.0
    worldview_reproject_method: nearest
    planet_store_sr: False
    pleiades_skip_pan: False
    prisma_rhot_per_pixel_sza: True
    prisma_store_l2c: False
    prisma_store_l2c_separate_file: True
    prisma_output_pan: False
    chris_interband_calibration: False
    chris_noise_reduction: True
    desis_mask_ql: True
    clear_scratch: True
    gf_reproject_to_utm: False
    viirs_option: img+mod
    viirs_scanline_projection: True
    viirs_scanline_width: 32
    viirs_quality_flags: [   4    8  512 1024 2048 4096]
    viirs_output_tir: True
    viirs_output_tir_lt: False
    viirs_mask_mband: True
    viirs_mask_immixed: True
    viirs_mask_immixed_rat: False
    viirs_mask_immixed_dif: True
    viirs_mask_immixed_bands: I03/M10
    viirs_mask_immixed_maxrat: 0.2
    viirs_mask_immixed_maxdif: 0.002
    tact_run: False
    tact_profile_source: era5
    tact_reptran: medium
    tact_emissivity: water
    tact_output_atmosphere: False
    tact_output_intermediate: False
    tact_map: True
    ged_fill: True
    tact_range: [ 3.5 14. ]
    eminet_water_fill: True
    eminet_water_threshold: 0.0215
    eminet_model_version: 20220809
    eminet_netname: Net2
    eminet_fill: True
    eminet_fill_dilate: False
    verbosity: 5
    output_lt: False
    solar_irradiance_reference: Coddington2021_1_0nm
    blackfill_skip: True
    blackfill_max: 1.0
    blackfill_wave: 1600
    output_bt: False
    l2w_parameters: ['rhow_*', 'rhos_*', 'chl_oc3', 'chl_re_gons', 'chl_re_moses3b', 'chl_re_moses3b740']
    l2w_mask: True
    l2w_mask_wave: 1600
    l2w_mask_threshold: 0.0215
    l2w_mask_water_parameters: True
    l2w_mask_negative_rhow: True
    l2w_mask_negative_wave_range: [400 900]
    l2w_mask_cirrus: True
    l2w_mask_cirrus_threshold: 0.005
    l2w_mask_cirrus_wave: 1373
    l2w_mask_high_toa: True
    l2w_mask_high_toa_threshold: 0.3
    l2w_mask_high_toa_wave_range: [ 400 2500]
    l2w_mask_mixed: True
    l2w_data_in_memory: False
    nechad_range: [600 900]
    nechad_max_rhow_C_factor: 0.5
    l2w_mask_smooth: True
    l2w_mask_smooth_sigma: 3
    flag_exponent_swir: 0
    flag_exponent_cirrus: 1
    flag_exponent_toa: 2
    flag_exponent_negative: 3
    flag_exponent_outofscene: 4
    flag_exponent_mixed: 5
    flag_exponent_dem_shadow: 6
    atmospheric_correction: True
    aerosol_correction: dark_spectrum
    gas_transmittance: True
    min_tgas_aot: 0.85
    min_tgas_rho: 0.7
    sza_limit_replace: False
    sza_limit: 79.999
    vza_limit_replace: False
    vza_limit: 71.999
    cirrus_correction: False
    cirrus_range: [1350. 1390.]
    cirrus_g_vnir: 1.0
    cirrus_g_swir: 0.5
    ancillary_data: True
    ancillary_type: GMAO_MERRA2_MET
    uoz_default: 0.3
    uwv_default: 1.5
    pressure_default: 1013.25
    dem_pressure: False
    dem_pressure_resolved: True
    dem_pressure_percentile: 25.0
    dem_pressure_write: False
    dem_source: copernicus30
    dem_shadow_mask: False
    dem_shadow_mask_extend: True
    dem_shadow_mask_extend_metres: 3000.0
    dem_shadow_mask_filter: True
    dem_shadow_mask_filter_kernel: [3 3]
    dem_shadow_mask_erode_its: 1
    dem_shadow_mask_dilate_its: 5
    dsf_interface_reflectance: False
    dsf_interface_option: default
    dsf_interface_lut: ACOLITE-RSKY-202102-82W
    wind_default: 2.0
    dsf_residual_glint_correction: False
    dsf_residual_glint_correction_method: default
    dsf_residual_glint_wave_range: [1500 2400]
    glint_mask_rhos_wave: 1600
    glint_mask_rhos_threshold: 0.05
    glint_write_rhog_ref: False
    glint_write_rhog_all: False
    adjacency_correction: False
    adjacency_method: acstar3
    acstar3_method: iter
    acstar3_psf_raster: False
    acstar3_max_wavelength: 720.0
    acstar3_fit_all_bands: True
    acstar3_write_rhosu: True
    acstar3_write_rhoa: True
    acstar3_write_rhoe: True
    acstar3_ex: 3
    acstar3_mask_edges: True
    radcor_initial_aot: 0.3
    radcor_psf_radius: 3.5
    radcor_psf_rescale: False
    radcor_psf_complete_method: neighborhood
    radcor_select_aot: minimum
    radcor_expand_edge: True
    radcor_expand_method: mirror
    radcor_mask_edges: True
    radcor_fft_stack: False
    radcor_write_rhot: True
    radcor_write_rhoe: True
    radcor_bratio_option: percentile
    radcor_bratio_percentile: 1.0
    dsf_nbands: 2
    dsf_nbands_fit: 2
    dsf_aot_compute: min
    dsf_percentile: 1.0
    dsf_minimum_segment_size: 1
    dsf_allow_lut_boundaries: False
    dsf_filter_rhot: False
    dsf_filter_percentile: 50.0
    dsf_filter_box: [10 10]
    dsf_filter_aot: False
    dsf_smooth_aot: False
    dsf_smooth_box: [10 10]
    dsf_aot_fillnan: True
    dsf_aot_most_common_model: True
    dsf_model_selection: min_drmsd
    dsf_min_tile_cover: 0.1
    dsf_min_tile_aot: 0.01
    dsf_max_tile_aot: 1.2
    dsf_write_tiled_parameters: False
    dsf_exclude_bands: []
    dsf_write_aot_550: False
    dsf_fixed_lut: ACOLITE-LUT-202110-MOD2
    dsf_tile_smoothing: True
    dsf_tile_smoothing_kernel_size: 3
    dsf_tile_interp_method: linear
    exp_swir_threshold: 0.0215
    exp_fixed_epsilon: True
    exp_fixed_epsilon_percentile: 50.0
    exp_fixed_aerosol_reflectance: True
    exp_fixed_aerosol_reflectance_percentile: 5.0
    exp_wave1: 1600
    exp_wave2: 2200
    exp_alpha_weighted: True
    exp_output_intermediate: False
    gains: False
    gains_parameter: radiance
    offsets: False
    rgb_rhot: True
    rgb_rhos: True
    rgb_rhorc: False
    rgb_rhow: False
    map_l2w: False
    map_title: True
    map_fontname: sans-serif
    map_fontsize: 12
    map_usetex: False
    map_dpi: 300
    map_ext: png
    map_scalebar: False
    map_scalebar_position: UL
    map_scalebar_color: Black
    map_scalebar_max_fraction: 0.33
    map_colorbar: True
    map_colorbar_orientation: vertical
    map_auto_range: False
    map_auto_range_percentiles: [ 1. 99.]
    map_fill_outrange: False
    map_fill_color: LightGrey
    map_default_colormap: viridis
    rgb_red_wl: 650
    rgb_green_wl: 560
    rgb_blue_wl: 480
    rgb_min: [0. 0. 0.]
    rgb_max: [0.15 0.15 0.15]
    rgb_gamma: [1. 1. 1.]
    rgb_autoscale: False
    rgb_autoscale_percentiles: [ 5. 95.]
    rgb_stretch: linear
    pans: False
    pans_method: panr
    pans_output: rgb
    pans_bgr: ['480', '560', '655']
    pans_rgb_rhot: True
    pans_rgb_rhos: True
    pans_export_geotiff_rgb: False
    pans_sensors: ['L7_ETM', 'L8_OLI', 'L9_OLI']
    map_projected: False
    map_raster: False
    map_pcolormesh: False
    map_cartopy: False
    map_mask: True
    map_xtick_rotation: 0.0
    map_ytick_rotation: 0.0
    map_gridline_color: white
    l1r_export_geotiff: False
    l2t_export_geotiff: False
    l2r_export_geotiff: False
    l2w_export_geotiff: False
    export_geotiff_coordinates: False
    export_cloud_optimized_geotiff: False
    export_cloud_optimized_geotiff_options: ['COMPRESS=LZW', 'PREDICTOR=YES', 'LEVEL=9', 'OVERVIEWS=AUTO', 'BLOCKSIZE=1024']
    l1r_export_geotiff_rgb: False
    l2r_export_geotiff_rgb: False
    use_gdal_merge_import: False
    l1r_delete_netcdf: False
    l2t_delete_netcdf: False
    l2r_delete_netcdf: False
    l2r_pans_delete_netcdf: False
    l2w_delete_netcdf: False
    delete_acolite_run_text_files: False
    delete_acolite_output_directory: False
    delete_extracted_input: False
    reproject_outputs: ['L1R', 'L2R', 'L2W']
    reproject_before_ac: False
    output_projection: False
    output_projection_metres: False
    output_projection_resolution_align: True
    output_projection_resampling_method: bilinear
    output_projection_fillnans: False
    output_projection_filldistance: 1
    output_projection_radius: 3.0
    output_projection_epsilon: 0.0
    output_projection_neighbours: 32
    reproject_inputfile: False
    reproject_inputfile_force: False
    reproject_inputfile_dem: False
    luts: ['ACOLITE-LUT-202110-MOD1', 'ACOLITE-LUT-202110-MOD2']
    luts_pressures: [ 500  750 1013 1100]
    slicing: False
    scene_download: False
    inputfile: ../app_acolite/input/S2A_MSIL1C_20220602T105631_N0400_R094_T31UFU_20220602T144004.SAFE
    runid: 20240322_114335
    dimensions(sizes): x(2099), y(3386)
    variables(dimensions): float64 transverse_mercator(), float64 x(x), float64 y(y), float32 lon(y, x), float32 lat(y, x), float32 rhow_443(y, x), float32 rhos_443(y, x), float32 rhow_492(y, x), float32 rhos_492(y, x), float32 rhow_560(y, x), float32 rhos_560(y, x), float32 rhow_665(y, x), float32 rhos_665(y, x), float32 rhow_704(y, x), float32 rhos_704(y, x), float32 rhow_740(y, x), float32 rhos_740(y, x), float32 rhow_783(y, x), float32 rhos_783(y, x), float32 rhow_833(y, x), float32 rhos_833(y, x), float32 rhow_865(y, x), float32 rhos_865(y, x), float32 rhow_1614(y, x), float32 rhos_1614(y, x), float32 rhow_2202(y, x), float32 rhos_2202(y, x), int32 l2_flags(y, x), float32 chl_oc3(y, x), float32 chl_re_gons(y, x), float32 chl_re_moses3b(y, x), float32 chl_re_moses3b740(y, x)
    groups:
BrandonSmithJ commented 3 months ago

It doesn't look like you have acolite set to output Rrs - the file only contains the rho bands. When processing with acolite, you'll have to make sure you use the configuration parameter 'l2wparameters=Rrs' in order to have acolite include Rrs in the output file.

From your nefCDF attributes, it looks like you're currently using l2w_parameters: ['rhow_*', 'rhos_*', 'chl_oc3', 'chl_re_gons', 'chl_re_moses3b', 'chl_re_moses3b740'], so you'll just need to add Rrs_* to that list.

stanleesocca commented 3 months ago

Ok. Make sense now. I will rerun the pre-processing and see what happen. I will keep you in touch.

Thanks.