Open stanleesocca opened 3 weeks ago
Yes, the weights directory contain the pre-trained model weights. The problem is that github LFS only works for about the first week of every month due to bots downloading the repo and exhausting the limited free LFS capacity. The issue you referenced is actually a separate problem though - what you want to look at is #14, in which I provide direct download links to the weight zip files.
Once you download the weight zip files for the models you want, just unzip them in the Weights directory and everything should work as expected.
It's been on my todo list for awhile to stop using LFS, so I went ahead and updated the repository to host the zip files directly. You should now be able to clone it and run the example without worrying about downloading the weights separately. Let me know if you run into any issues after re-cloning the repo.
Ok. The test case with numpy example work perfectly. Thanks for that.
I notice also that get_tile_data
function does not work well with ncdf data from acolite. I tried it with an output a sentinel-2 acolite L2R processed file, and it fails to find the Rrs.
>>> atmp = "./app_acolite/output/sentinels/S2A_MSIL1C_20220602/S2A_MSI_2022_06_02_11_06_23_T31UFU_L2R.nc"
>>> band, Rrs = get_tile_data(atmp, sensor, allow_neg=False)
Here is the error log.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
assert(len(data) == len(features)), f'Missing features: Found {list(data.keys())}, Expecting {features}'
AssertionError: Missing features: Found [], Expecting ['Rrs']
Have you run into this type of error before? I see that in the paper, acolite was part of the atmospheric pre-processor tool used.
Thanks very much for your time.
I've only tested with L2W, which it does work for. L2W files have the following variables (e.g. for OLI):
variables(dimensions): float32 lon(y, x), float32 lat(y, x), int32 l2_flags(
y, x), float32 Rrs_443(y, x), float32 Rrs_483(y, x), float32 Rrs_561(y, x), floa
t32 Rrs_655(y, x), float32 Rrs_865(y, x), float32 Rrs_1609(y, x), float32 Rrs_22
01(y, x)
Can you tell me what the netCDF variables look like?
from netCDF4 import Dataset
print( Dataset(atmp) )
I changed it to the L2W file, and same error:
With print(Dataset(atmp))
, I get this:
>>> print(Dataset(atmp))
<class 'netCDF4._netCDF4.Dataset'>
root group (NETCDF4 data model, file format HDF5):
generated_by: ACOLITE
generated_on: 2024-03-22 11:43:41 CET
contact: Quinten Vanhellemont
product_type: NetCDF
metadata_profile: beam
metadata_version: 0.5
auto_grouping: rhot:rhorc:rhos:rhow:Rrs:Lt
Conventions: CF-1.7
sensor: S2A_MSI
isodate: 2022-06-02T11:06:23.464274+00:00
global_dims: [3386 2099]
sza: 31.0679062389307
vza: 11.263176190476191
raa: 129.46760309375418
vaa: 295.7634761904762
saa: 166.295873096722
se_distance: 1.0141500310042952
mus: 0.856556282163115
granule: L1C_T31UFU_A036270_20220602T105627
mgrs_tile: T31UFU
acolite_file_type: L2R
tile_code: T31UFU
oname: S2A_MSI_2022_06_02_11_06_23_T31UFU
ofile: ../app_acolite/output/sentinels/S2A_MSIL1C_20220602/S2A_MSI_2022_06_02_11_06_23_T31UFU_L2W.nc
1_wave: 442.68570115396517
1_name: 443
1_f0: 19054.057897300103
2_wave: 492.43738054332624
2_name: 492
2_f0: 19816.582596730874
3_wave: 559.8501892469018
3_name: 560
3_f0: 18712.647798388396
4_wave: 664.619908601124
4_name: 665
4_f0: 15324.95308599693
5_wave: 704.1159402425621
5_name: 704
5_f0: 14122.137496285726
6_wave: 740.478761786033
6_name: 740
6_f0: 12885.020009014514
7_wave: 782.7500096627707
7_name: 783
7_f0: 11782.31866822806
8_wave: 832.7883239563182
8_name: 833
8_f0: 10446.745777714672
8A_wave: 864.7104683919092
8A_name: 865
8A_f0: 9505.78414767759
9_wave: 945.0507000978324
9_name: 945
9_f0: 8123.418928961382
10_wave: 1373.4620165399258
10_name: 1373
10_f0: 3585.7816638508825
11_wave: 1613.659319714372
11_name: 1614
11_f0: 2368.9992957468658
12_wave: 2202.3666356190934
12_name: 2202
12_f0: 812.0749465544842
scene_xrange: [600000. 709800.]
scene_yrange: [5900040. 5790240.]
scene_proj4_string: +proj=utm +zone=31 +datum=WGS84 +units=m +no_defs
scene_pixel_size: [ 10. -10.]
scene_dims: [10980 10980]
sub: [ 712 1496 2099 3386]
limit: [52.8 4.6 53.1 4.9]
xrange: [607120. 628110.]
yrange: [5885080. 5851220.]
proj4_string: +proj=utm +zone=31 +datum=WGS84 +units=m +no_defs
pixel_size: [ 10. -10.]
projection_key: transverse_mercator
data_dimensions: [3386 2099]
data_elements: 7107214
uoz: 0.3
uwv: 1.5
wind: 2.0
pressure: 1013.25
acolite_version: Generic GitHub Clone c2024-03-07T17:08:12
copy_datasets: ['lon', 'lat', 'sza', 'vza', 'raa', 'rhot_*']
s2_write_vaa: False
s2_write_saa: False
dsf_aot_estimate: tiled
dsf_spectrum_option: intercept
dsf_wave_range: [400 900]
s2_target_res: 10
resolved_geometry: True
gains_toa: [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
offsets_toa: [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
dsf_intercept_pixels: 200
dsf_tile_dimensions: [600 600]
geometry_per_band: False
geometry_fixed_footprint: False
luts_reduce_dimensions: True
region_name:
polygon_limit: True
polylakes: False
polylakes_database: worldlakes
merge_tiles: False
merge_zones: True
extend_region: False
l1r_crop: False
output: ../app_acolite/output/sentinels/S2A_MSIL1C_20220602
output_geolocation: True
output_xy: False
output_geometry: True
output_rhorc: False
add_band_name: False
compute_contrabands: False
netcdf_projection: True
netcdf_compression: False
netcdf_compression_level: 4
netcdf_discretisation: False
landsat_qa_bands: ['PIXEL', 'RADSAT', 'SATURATION']
landsat_qa_output: False
s2_auxiliary_include: False
s2_auxiliary_project: False
s2_auxiliary_default: False
s2_auxiliary_interpolate: True
s2_write_dfoo: False
s2_dilate_blackfill: False
s2_dilate_blackfill_iterations: 2
geometry_type: grids_footprint
geometry_res: 60
geometry_override: False
smile_correction: True
smile_correction_tgas: True
use_tpg: True
use_supplied_ancillary: True
worldview_reproject: False
worldview_reproject_resolution: 2.0
worldview_reproject_method: nearest
planet_store_sr: False
pleiades_skip_pan: False
prisma_rhot_per_pixel_sza: True
prisma_store_l2c: False
prisma_store_l2c_separate_file: True
prisma_output_pan: False
chris_interband_calibration: False
chris_noise_reduction: True
desis_mask_ql: True
clear_scratch: True
gf_reproject_to_utm: False
viirs_option: img+mod
viirs_scanline_projection: True
viirs_scanline_width: 32
viirs_quality_flags: [ 4 8 512 1024 2048 4096]
viirs_output_tir: True
viirs_output_tir_lt: False
viirs_mask_mband: True
viirs_mask_immixed: True
viirs_mask_immixed_rat: False
viirs_mask_immixed_dif: True
viirs_mask_immixed_bands: I03/M10
viirs_mask_immixed_maxrat: 0.2
viirs_mask_immixed_maxdif: 0.002
tact_run: False
tact_profile_source: era5
tact_reptran: medium
tact_emissivity: water
tact_output_atmosphere: False
tact_output_intermediate: False
tact_map: True
ged_fill: True
tact_range: [ 3.5 14. ]
eminet_water_fill: True
eminet_water_threshold: 0.0215
eminet_model_version: 20220809
eminet_netname: Net2
eminet_fill: True
eminet_fill_dilate: False
verbosity: 5
output_lt: False
solar_irradiance_reference: Coddington2021_1_0nm
blackfill_skip: True
blackfill_max: 1.0
blackfill_wave: 1600
output_bt: False
l2w_parameters: ['rhow_*', 'rhos_*', 'chl_oc3', 'chl_re_gons', 'chl_re_moses3b', 'chl_re_moses3b740']
l2w_mask: True
l2w_mask_wave: 1600
l2w_mask_threshold: 0.0215
l2w_mask_water_parameters: True
l2w_mask_negative_rhow: True
l2w_mask_negative_wave_range: [400 900]
l2w_mask_cirrus: True
l2w_mask_cirrus_threshold: 0.005
l2w_mask_cirrus_wave: 1373
l2w_mask_high_toa: True
l2w_mask_high_toa_threshold: 0.3
l2w_mask_high_toa_wave_range: [ 400 2500]
l2w_mask_mixed: True
l2w_data_in_memory: False
nechad_range: [600 900]
nechad_max_rhow_C_factor: 0.5
l2w_mask_smooth: True
l2w_mask_smooth_sigma: 3
flag_exponent_swir: 0
flag_exponent_cirrus: 1
flag_exponent_toa: 2
flag_exponent_negative: 3
flag_exponent_outofscene: 4
flag_exponent_mixed: 5
flag_exponent_dem_shadow: 6
atmospheric_correction: True
aerosol_correction: dark_spectrum
gas_transmittance: True
min_tgas_aot: 0.85
min_tgas_rho: 0.7
sza_limit_replace: False
sza_limit: 79.999
vza_limit_replace: False
vza_limit: 71.999
cirrus_correction: False
cirrus_range: [1350. 1390.]
cirrus_g_vnir: 1.0
cirrus_g_swir: 0.5
ancillary_data: True
ancillary_type: GMAO_MERRA2_MET
uoz_default: 0.3
uwv_default: 1.5
pressure_default: 1013.25
dem_pressure: False
dem_pressure_resolved: True
dem_pressure_percentile: 25.0
dem_pressure_write: False
dem_source: copernicus30
dem_shadow_mask: False
dem_shadow_mask_extend: True
dem_shadow_mask_extend_metres: 3000.0
dem_shadow_mask_filter: True
dem_shadow_mask_filter_kernel: [3 3]
dem_shadow_mask_erode_its: 1
dem_shadow_mask_dilate_its: 5
dsf_interface_reflectance: False
dsf_interface_option: default
dsf_interface_lut: ACOLITE-RSKY-202102-82W
wind_default: 2.0
dsf_residual_glint_correction: False
dsf_residual_glint_correction_method: default
dsf_residual_glint_wave_range: [1500 2400]
glint_mask_rhos_wave: 1600
glint_mask_rhos_threshold: 0.05
glint_write_rhog_ref: False
glint_write_rhog_all: False
adjacency_correction: False
adjacency_method: acstar3
acstar3_method: iter
acstar3_psf_raster: False
acstar3_max_wavelength: 720.0
acstar3_fit_all_bands: True
acstar3_write_rhosu: True
acstar3_write_rhoa: True
acstar3_write_rhoe: True
acstar3_ex: 3
acstar3_mask_edges: True
radcor_initial_aot: 0.3
radcor_psf_radius: 3.5
radcor_psf_rescale: False
radcor_psf_complete_method: neighborhood
radcor_select_aot: minimum
radcor_expand_edge: True
radcor_expand_method: mirror
radcor_mask_edges: True
radcor_fft_stack: False
radcor_write_rhot: True
radcor_write_rhoe: True
radcor_bratio_option: percentile
radcor_bratio_percentile: 1.0
dsf_nbands: 2
dsf_nbands_fit: 2
dsf_aot_compute: min
dsf_percentile: 1.0
dsf_minimum_segment_size: 1
dsf_allow_lut_boundaries: False
dsf_filter_rhot: False
dsf_filter_percentile: 50.0
dsf_filter_box: [10 10]
dsf_filter_aot: False
dsf_smooth_aot: False
dsf_smooth_box: [10 10]
dsf_aot_fillnan: True
dsf_aot_most_common_model: True
dsf_model_selection: min_drmsd
dsf_min_tile_cover: 0.1
dsf_min_tile_aot: 0.01
dsf_max_tile_aot: 1.2
dsf_write_tiled_parameters: False
dsf_exclude_bands: []
dsf_write_aot_550: False
dsf_fixed_lut: ACOLITE-LUT-202110-MOD2
dsf_tile_smoothing: True
dsf_tile_smoothing_kernel_size: 3
dsf_tile_interp_method: linear
exp_swir_threshold: 0.0215
exp_fixed_epsilon: True
exp_fixed_epsilon_percentile: 50.0
exp_fixed_aerosol_reflectance: True
exp_fixed_aerosol_reflectance_percentile: 5.0
exp_wave1: 1600
exp_wave2: 2200
exp_alpha_weighted: True
exp_output_intermediate: False
gains: False
gains_parameter: radiance
offsets: False
rgb_rhot: True
rgb_rhos: True
rgb_rhorc: False
rgb_rhow: False
map_l2w: False
map_title: True
map_fontname: sans-serif
map_fontsize: 12
map_usetex: False
map_dpi: 300
map_ext: png
map_scalebar: False
map_scalebar_position: UL
map_scalebar_color: Black
map_scalebar_max_fraction: 0.33
map_colorbar: True
map_colorbar_orientation: vertical
map_auto_range: False
map_auto_range_percentiles: [ 1. 99.]
map_fill_outrange: False
map_fill_color: LightGrey
map_default_colormap: viridis
rgb_red_wl: 650
rgb_green_wl: 560
rgb_blue_wl: 480
rgb_min: [0. 0. 0.]
rgb_max: [0.15 0.15 0.15]
rgb_gamma: [1. 1. 1.]
rgb_autoscale: False
rgb_autoscale_percentiles: [ 5. 95.]
rgb_stretch: linear
pans: False
pans_method: panr
pans_output: rgb
pans_bgr: ['480', '560', '655']
pans_rgb_rhot: True
pans_rgb_rhos: True
pans_export_geotiff_rgb: False
pans_sensors: ['L7_ETM', 'L8_OLI', 'L9_OLI']
map_projected: False
map_raster: False
map_pcolormesh: False
map_cartopy: False
map_mask: True
map_xtick_rotation: 0.0
map_ytick_rotation: 0.0
map_gridline_color: white
l1r_export_geotiff: False
l2t_export_geotiff: False
l2r_export_geotiff: False
l2w_export_geotiff: False
export_geotiff_coordinates: False
export_cloud_optimized_geotiff: False
export_cloud_optimized_geotiff_options: ['COMPRESS=LZW', 'PREDICTOR=YES', 'LEVEL=9', 'OVERVIEWS=AUTO', 'BLOCKSIZE=1024']
l1r_export_geotiff_rgb: False
l2r_export_geotiff_rgb: False
use_gdal_merge_import: False
l1r_delete_netcdf: False
l2t_delete_netcdf: False
l2r_delete_netcdf: False
l2r_pans_delete_netcdf: False
l2w_delete_netcdf: False
delete_acolite_run_text_files: False
delete_acolite_output_directory: False
delete_extracted_input: False
reproject_outputs: ['L1R', 'L2R', 'L2W']
reproject_before_ac: False
output_projection: False
output_projection_metres: False
output_projection_resolution_align: True
output_projection_resampling_method: bilinear
output_projection_fillnans: False
output_projection_filldistance: 1
output_projection_radius: 3.0
output_projection_epsilon: 0.0
output_projection_neighbours: 32
reproject_inputfile: False
reproject_inputfile_force: False
reproject_inputfile_dem: False
luts: ['ACOLITE-LUT-202110-MOD1', 'ACOLITE-LUT-202110-MOD2']
luts_pressures: [ 500 750 1013 1100]
slicing: False
scene_download: False
inputfile: ../app_acolite/input/S2A_MSIL1C_20220602T105631_N0400_R094_T31UFU_20220602T144004.SAFE
runid: 20240322_114335
dimensions(sizes): x(2099), y(3386)
variables(dimensions): float64 transverse_mercator(), float64 x(x), float64 y(y), float32 lon(y, x), float32 lat(y, x), float32 rhow_443(y, x), float32 rhos_443(y, x), float32 rhow_492(y, x), float32 rhos_492(y, x), float32 rhow_560(y, x), float32 rhos_560(y, x), float32 rhow_665(y, x), float32 rhos_665(y, x), float32 rhow_704(y, x), float32 rhos_704(y, x), float32 rhow_740(y, x), float32 rhos_740(y, x), float32 rhow_783(y, x), float32 rhos_783(y, x), float32 rhow_833(y, x), float32 rhos_833(y, x), float32 rhow_865(y, x), float32 rhos_865(y, x), float32 rhow_1614(y, x), float32 rhos_1614(y, x), float32 rhow_2202(y, x), float32 rhos_2202(y, x), int32 l2_flags(y, x), float32 chl_oc3(y, x), float32 chl_re_gons(y, x), float32 chl_re_moses3b(y, x), float32 chl_re_moses3b740(y, x)
groups:
It doesn't look like you have acolite set to output Rrs - the file only contains the rho bands. When processing with acolite, you'll have to make sure you use the configuration parameter 'l2wparameters=Rrs' in order to have acolite include Rrs in the output file.
From your nefCDF attributes, it looks like you're currently using l2w_parameters: ['rhow_*', 'rhos_*', 'chl_oc3', 'chl_re_gons', 'chl_re_moses3b', 'chl_re_moses3b740']
, so you'll just need to add Rrs_*
to that list.
Ok. Make sense now. I will rerun the pre-processing and see what happen. I will keep you in touch.
Thanks.
Hi @BrandonSmithJ , Thanks for the package. I was trying some of the examples here to see if it fit my use case. I notice that the program fails to run despite having successfully cloned your repo. My assumption is that the files in the Weights folder contains the model's pre-trained weights. Am I right?
I got this example when trying the random numpy example.
and a snippet of the error:
I see that other issue like #19 already posed some similar issue. Can you help solve this issue.
Thanks