search

environmental-forecasting / preprocess-toolbox

A toolbox for processing downloaded datasets according to common approaches for environmental data
MIT License
0 stars 0 forks source link

Basic implementations of processors #1

Closed JimCircadian closed 1 month ago

JimCircadian commented 3 months ago

Effectively a migration of code to the CLI stubs:

The following were planned to incorporate correction calculation AND icenet_process normalisation / variable preparation w.r.t source data. However, these are now all implemented via argument calls to implementations in the icenet library to introduce / implement application specific processing atop the NormalisingChannelProcessor implementation built natively into preprocess_toolbox. This is far more flexible, as it means preprocess_toolbox is very dataset agnostic, reducing specificity of preprocessing, and resulting in very low profile implementation overhead for new applications.

JimCircadian commented 2 months ago

I've decided to retain the original behaviour of having a "true" reference cube for regridding, simply as new references are likely required for many ground truths (the available geospatial data just isn't available in a lot of downloads)

JimCircadian commented 1 month ago

The full workflow to produce loaders (in this case the necessary structure for the IceNet SIC runs) is in place. Here's a dump of the commands, with progress into #5 now being required to finish off the initial implementation: 

preprocess_missing_time -n siconca data/osisaf/dataset_config.month.hemi.north.json osi_sic
preprocess_missing_spatial -n siconca data/osi_sic/dataset_config.month.hemi.north.json
preprocess_regrid data/era5/dataset_config.month.hemi.north.json ref.osisaf.nc era5_osi
preprocess_rotate -n uas,vas data/era5_osi/dataset_config.month.hemi.north.json ref.osisaf.nc
preprocess_dataset --abs siconca --trends siconca --trend-lead 1,3 -ps train -sn train,val,test -ss '2022-1-1|2021-1-1,2022-11-1,2023-1-1' -se '2022-10-31|2021-12-31,2022-12-31,2023-2-28' -i icenet.data.processors.osisaf:SICPreProcessor data/osi_sic/dataset_config.month.hemi.north.json test_osisaf
preprocess_dataset --abs uas,vas --anom tas,ta500,tos,psl,zg500,zg250 -ps train -sn train,val,test -ss '2022-1-1|2021-1-1,2022-11-1,2023-1-1' -se '2022-10-31|2021-12-31,2022-12-31,2023-2-28' -i icenet.data.processors.cds:ERA5PreProcessor data/era5_osi/dataset_config.month.hemi.north.json test_era5
preprocess_loader_init test processed.test_osisaf.json processed.test_era5.json
preprocess_add_channel loader.test.json data/osi_sic/dataset_config.month.hemi.north.json sin icenet.data.meta:SinProcessor
preprocess_add_channel loader.test.json data/osi_sic/dataset_config.month.hemi.north.json cos icenet.data.meta:CosProcessor
preprocess_add_channel loader.test.json data/osi_sic/dataset_config.month.hemi.north.json land icenet.data.meta:LandMaskChannelProcessor
preprocess_add_mask loader.test.json data/osi_sic/dataset_config.month.hemi.north.json land icenet.data.masks.nsidc:land
preprocess_add_mask loader.test.json data/osi_sic/dataset_config.month.hemi.north.json polarhole icenet.data.masks.nsidc:polarhole
preprocess_add_mask loader.test.json data/osi_sic/dataset_config.month.hemi.north.json active_grid_cell icenet.data.masks.nsidc:active_grid_cell

There are numerous errors now logged in #6, #7 and #8, but the basic structure and generalised implementation is in place: no IceNet or SIC logic is contained herein and there is a clear pathway to applying this to other applications (which will eventually be demonstrated under #9)

$ preprocess_missing_time -n siconca data/osisaf/dataset_config.month.hemi.north.json osi_sic
INFO:root:Attempting to instantiate SICDatasetConfig with loaded configuration
INFO:root:Creating path: ./data/osi_sic/month/hemi.north
INFO:root:Copying ./data/osisaf/month/hemi.north to ./data/osi_sic/month/hemi.north
INFO:root:Applying copy_to to identifier osi_sic
INFO:root:Got 4 filenames to open dataset with!
INFO:root:Processing missing dates for siconca
  [See issue #6]
INFO:root:Processing 0 missing dates
INFO:root:Written new data to /home/jambyr/icenet4/pipeline/tmpwmgih0vv and merging with ./data/osi_sic/month/hemi.north/siconca/2020.nc
INFO:root:Saving merged data to ./data/osi_sic/month/hemi.north/siconca/2020.nc... 
INFO:root:Written new data to /home/jambyr/icenet4/pipeline/tmpv8qe_37c and merging with ./data/osi_sic/month/hemi.north/siconca/2021.nc
INFO:root:Saving merged data to ./data/osi_sic/month/hemi.north/siconca/2021.nc... 
INFO:root:Written new data to /home/jambyr/icenet4/pipeline/tmpn59u1eb4 and merging with ./data/osi_sic/month/hemi.north/siconca/2022.nc
INFO:root:Saving merged data to ./data/osi_sic/month/hemi.north/siconca/2022.nc... 
INFO:root:Written new data to /home/jambyr/icenet4/pipeline/tmpyphxscwv and merging with ./data/osi_sic/month/hemi.north/siconca/2023.nc
INFO:root:Saving merged data to ./data/osi_sic/month/hemi.north/siconca/2023.nc... 
INFO:root:Writing configuration to ./data/osi_sic/dataset_config.month.hemi.north.json
INFO:root:Saved dataset config ./data/osi_sic/dataset_config.month.hemi.north.json

$ preprocess_missing_spatial -n siconca data/osi_sic/dataset_config.month.hemi.north.json
INFO:root:Attempting to instantiate SICDatasetConfig with loaded configuration
INFO:root:Got 4 filenames to open dataset with!
INFO:root:Processing missing dates for siconca
INFO:root:Interpolating spatial data for 202201
WARNING:root:Interpolation failed for 202201, assignment will not take place
  [See issue #6 / #7]
INFO:root:Written new data to /home/jambyr/icenet4/pipeline/tmpdf_najsl and merging with ./data/osi_sic/month/hemi.north/siconca/2020.nc
INFO:root:Saving merged data to ./data/osi_sic/month/hemi.north/siconca/2020.nc... 
INFO:root:Written new data to /home/jambyr/icenet4/pipeline/tmpnptrg91e and merging with ./data/osi_sic/month/hemi.north/siconca/2021.nc
INFO:root:Saving merged data to ./data/osi_sic/month/hemi.north/siconca/2021.nc... 
INFO:root:Written new data to /home/jambyr/icenet4/pipeline/tmpjiznkjzj and merging with ./data/osi_sic/month/hemi.north/siconca/2022.nc
INFO:root:Saving merged data to ./data/osi_sic/month/hemi.north/siconca/2022.nc... 
INFO:root:Written new data to /home/jambyr/icenet4/pipeline/tmphqc15y3_ and merging with ./data/osi_sic/month/hemi.north/siconca/2023.nc
INFO:root:Saving merged data to ./data/osi_sic/month/hemi.north/siconca/2023.nc... 
INFO:root:Loading configuration ./data/osi_sic/dataset_config.month.hemi.north.json
INFO:root:Writing configuration to ./data/osi_sic/dataset_config.month.hemi.north.json
INFO:root:Saved dataset config ./data/osi_sic/dataset_config.month.hemi.north.json

$ preprocess_regrid data/era5/dataset_config.month.hemi.north.json ref.osisaf.nc era5_osi
INFO:root:Attempting to instantiate ERA5DatasetConfig with loaded configuration
INFO:root:Creating path: ./data/era5_osi/month/hemi.north
INFO:root:Copying ./data/era5/month/hemi.north to ./data/era5_osi/month/hemi.north
INFO:root:Applying copy_to to identifier era5_osi
INFO:root:Got 32 filenames to open dataset with!
INFO:root:Regridding dataset
  [See issue #8]
INFO:root:Writing configuration to ./data/era5_osi/dataset_config.month.hemi.north.json
INFO:root:Saved dataset config ./data/era5_osi/dataset_config.month.hemi.north.json

$ preprocess_rotate -n uas,vas data/era5_osi/dataset_config.month.hemi.north.json ref.osisaf.nc
INFO:root:Attempting to instantiate ERA5DatasetConfig with loaded configuration
INFO:root:Got 8 filenames to open dataset with!
  [See issue #8]
INFO:root:Rotating ./data/era5_osi/month/hemi.north/uas/2020.nc and ./data/era5_osi/month/hemi.north/vas/2020.nc
INFO:root:Rotating ./data/era5_osi/month/hemi.north/uas/2021.nc and ./data/era5_osi/month/hemi.north/vas/2021.nc
INFO:root:Rotating ./data/era5_osi/month/hemi.north/uas/2022.nc and ./data/era5_osi/month/hemi.north/vas/2022.nc
INFO:root:Rotating ./data/era5_osi/month/hemi.north/uas/2023.nc and ./data/era5_osi/month/hemi.north/vas/2023.nc
INFO:root:Loading configuration ./data/era5_osi/dataset_config.month.hemi.north.json
INFO:root:Writing configuration to ./data/era5_osi/dataset_config.month.hemi.north.json
INFO:root:Saved dataset config ./data/era5_osi/dataset_config.month.hemi.north.json

$ preprocess_dataset --abs siconca --trends siconca --trend-lead 1,3 -ps train -sn train,val,test -ss '2022-1-1|2021-1-1,2022-11-1,2023-1-1' -se '2022-10-31|2021-12-31,2022-12-31,2023-2-28' -i icenet.data.processors.osisaf:SICPreProcessor data/osi_sic/dataset_config.month.hemi.north.json test_osisaf
INFO:root:Attempting to instantiate SICDatasetConfig with loaded configuration
  [See issue #6]
INFO:root:Got 22 dates for train
INFO:root:Got 2 dates for val
INFO:root:Got 2 dates for test
INFO:root:Creating path: ./processed/test_osisaf
INFO:root:Loading configuration ./data/osi_sic/dataset_config.month.hemi.north.json
INFO:root:Processing 22 dates for train category
INFO:root:Including lead of 3 days
INFO:root:Processing 2 dates for val category
INFO:root:Including lead of 3 days
INFO:root:Processing 2 dates for test category
INFO:root:Including lead of 3 days
INFO:root:Got 3 files for train:siconca
INFO:root:Got 2 files for val:siconca
INFO:root:Got 1 files for test:siconca
INFO:root:Opening files for siconca
INFO:root:Generating trend data up to 3 steps ahead for 36 dates
INFO:root:Generating 72 trend dates
INFO:root:Writing new trend cache for siconca
INFO:root:Normalising siconca
INFO:root:Writing configuration to ./processed.test_osisaf.json
INFO:root:Saved dataset config ./processed.test_osisaf.json

$ preprocess_dataset --abs uas,vas --anom tas,ta500,tos,psl,zg500,zg250 -ps train -sn train,val,test -ss '2022-1-1|2021-1-1,2022-11-1,2023-1-1' -se '2022-10-31|2021-12-31,2022-12-31,2023-2-28' -i icenet.data.processors.cds:ERA5PreProcessor data/era5_osi/dataset_config.month.hemi.north.json test_era5
INFO:root:Attempting to instantiate ERA5DatasetConfig with loaded configuration
  [See issue #6]
INFO:root:Got 22 dates for train
INFO:root:Got 2 dates for val
INFO:root:Got 2 dates for test
INFO:root:Creating path: ./processed/test_era5
INFO:root:Loading configuration ./data/era5_osi/dataset_config.month.hemi.north.json
INFO:root:Processing 22 dates for train category
INFO:root:Including lead of 3 days
INFO:root:Processing 2 dates for val category
INFO:root:Including lead of 3 days
INFO:root:Processing 2 dates for test category
INFO:root:Including lead of 3 days
INFO:root:Got 3 files for train:tas
INFO:root:Got 3 files for train:ta500
INFO:root:Got 3 files for train:tos
INFO:root:Got 3 files for train:psl
INFO:root:Got 3 files for train:zg250
INFO:root:Got 3 files for train:zg500
INFO:root:Got 3 files for train:uas
INFO:root:Got 3 files for train:vas
INFO:root:Got 2 files for val:tas
INFO:root:Got 2 files for val:ta500
INFO:root:Got 2 files for val:tos
INFO:root:Got 2 files for val:psl
INFO:root:Got 2 files for val:zg250
INFO:root:Got 2 files for val:zg500
INFO:root:Got 2 files for val:uas
INFO:root:Got 2 files for val:vas
INFO:root:Got 1 files for test:tas
INFO:root:Got 1 files for test:ta500
INFO:root:Got 1 files for test:tos
INFO:root:Got 1 files for test:psl
INFO:root:Got 1 files for test:zg250
INFO:root:Got 1 files for test:zg500
INFO:root:Got 1 files for test:uas
INFO:root:Got 1 files for test:vas
INFO:root:Opening files for uas
INFO:root:Normalising uas
INFO:root:Opening files for vas
INFO:root:Normalising vas
INFO:root:Opening files for tas
INFO:root:Generating climatology ./processed/test_era5/params/climatology.tas
INFO:root:Normalising tas
INFO:root:Opening files for ta500
INFO:root:Generating climatology ./processed/test_era5/params/climatology.ta500
INFO:root:Normalising ta500
INFO:root:Opening files for tos
INFO:root:Generating climatology ./processed/test_era5/params/climatology.tos
INFO:root:Normalising tos
INFO:root:Opening files for psl
INFO:root:Generating climatology ./processed/test_era5/params/climatology.psl
INFO:root:Normalising psl
INFO:root:Opening files for zg500
INFO:root:Generating climatology ./processed/test_era5/params/climatology.zg500
INFO:root:Normalising zg500
INFO:root:Opening files for zg250
INFO:root:Generating climatology ./processed/test_era5/params/climatology.zg250
INFO:root:Normalising zg250
INFO:root:Writing configuration to ./processed.test_era5.json
INFO:root:Saved dataset config ./processed.test_era5.json

$ preprocess_loader_init test processed.test_osisaf.json processed.test_era5.json
INFO:root:Configuration processed.test_osisaf.json being loaded
INFO:root:Configuration processed.test_era5.json being loaded

$ preprocess_add_channel loader.test.json data/osi_sic/dataset_config.month.hemi.north.json sin icenet.data.meta:SinProcessor
INFO:root:Attempting to instantiate SICDatasetConfig with loaded configuration
INFO:root:Configuration loader.test.json being loaded
INFO:root:Attempting to instantiate SICDatasetConfig with loaded configuration
INFO:root:Loading configuration ./data/osi_sic/dataset_config.month.hemi.north.json
INFO:root:Attempting to instantiate SICDatasetConfig with loaded configuration
INFO:root:Loading configuration ./data/osi_sic/dataset_config.month.hemi.north.json
INFO:root:Creating path: ./processed/test_osisaf/sin
INFO:root:Configuration loader.test.json being loaded
INFO:root:Writing over loader.test.json

$ preprocess_add_channel loader.test.json data/osi_sic/dataset_config.month.hemi.north.json cos icenet.data.meta:CosProcessor
INFO:root:Attempting to instantiate SICDatasetConfig with loaded configuration
INFO:root:Configuration loader.test.json being loaded
INFO:root:Attempting to instantiate SICDatasetConfig with loaded configuration
INFO:root:Loading configuration ./data/osi_sic/dataset_config.month.hemi.north.json
INFO:root:Attempting to instantiate SICDatasetConfig with loaded configuration
INFO:root:Loading configuration ./data/osi_sic/dataset_config.month.hemi.north.json
INFO:root:Creating path: ./processed/test_osisaf/cos
INFO:root:Configuration loader.test.json being loaded
INFO:root:Writing over loader.test.json

$ preprocess_add_channel loader.test.json data/osi_sic/dataset_config.month.hemi.north.json land icenet.data.meta:LandMaskChannelProcessor
INFO:root:Attempting to instantiate SICDatasetConfig with loaded configuration
INFO:root:Configuration loader.test.json being loaded
INFO:root:Attempting to instantiate SICDatasetConfig with loaded configuration
INFO:root:Loading configuration ./data/osi_sic/dataset_config.month.hemi.north.json
INFO:root:Attempting to instantiate SICDatasetConfig with loaded configuration
INFO:root:Loading configuration ./data/osi_sic/dataset_config.month.hemi.north.json
INFO:root:Creating path: ./processed/test_osisaf/land
INFO:root:Creating path: ./processed/masks
...
ice_conc_nh_ease2-250_cdr-v2p0_20000 100%[===================================================================>]   9.40M  2.93MB/s    in 3.2s    
2024-07-25 10:05:49 (2.93 MB/s) - ‘./processed/masks/masks/ice_conc_nh_ease2-250_cdr-v2p0_200001021200.nc’ saved [9856120]
INFO:root:Child returned: 0
INFO:root:Saving ./processed/masks/masks/active_grid_cell_mask_01.npy
INFO:root:Saving ./processed/masks/masks/land_mask.npy
INFO:root:Saving polarhole ./processed/masks/masks/polarhole1_mask.npy
INFO:root:Saving polarhole ./processed/masks/masks/polarhole2_mask.npy
INFO:root:Saving polarhole ./processed/masks/masks/polarhole3_mask.npy
...
ice_conc_nh_ease2-250_cdr-v2p0_20000 100%[===================================================================>]   9.40M  3.15MB/s    in 3.0s    
2024-07-25 10:05:52 (3.15 MB/s) - ‘./processed/masks/masks/ice_conc_nh_ease2-250_cdr-v2p0_200002021200.nc’ saved [9856120]
INFO:root:Child returned: 0
INFO:root:Saving ./processed/masks/masks/active_grid_cell_mask_02.npy
...
ice_conc_nh_ease2-250_cdr-v2p0_20000 100%[===================================================================>]   9.40M  3.13MB/s    in 3.0s    
2024-07-25 10:05:56 (3.13 MB/s) - ‘./processed/masks/masks/ice_conc_nh_ease2-250_cdr-v2p0_200003021200.nc’ saved [9856120]
INFO:root:Child returned: 0
INFO:root:Saving ./processed/masks/masks/active_grid_cell_mask_03.npy
...
ice_conc_nh_ease2-250_cdr-v2p0_20000 100%[===================================================================>]   9.40M  3.13MB/s    in 3.0s    
2024-07-25 10:05:59 (3.13 MB/s) - ‘./processed/masks/masks/ice_conc_nh_ease2-250_cdr-v2p0_200004021200.nc’ saved [9856120]
INFO:root:Child returned: 0
INFO:root:Saving ./processed/masks/masks/active_grid_cell_mask_04.npy
...
ice_conc_nh_ease2-250_cdr-v2p0_20000 100%[===================================================================>]   9.40M  3.12MB/s    in 3.0s    
2024-07-25 10:06:03 (3.12 MB/s) - ‘./processed/masks/masks/ice_conc_nh_ease2-250_cdr-v2p0_200005021200.nc’ saved [9856120]
INFO:root:Child returned: 0
INFO:root:Saving ./processed/masks/masks/active_grid_cell_mask_05.npy
...
ice_conc_nh_ease2-250_cdr-v2p0_20000 100%[===================================================================>]   9.40M  3.12MB/s    in 3.0s    
2024-07-25 10:06:07 (3.12 MB/s) - ‘./processed/masks/masks/ice_conc_nh_ease2-250_cdr-v2p0_200006021200.nc’ saved [9856120]
INFO:root:Child returned: 0
INFO:root:Saving ./processed/masks/masks/active_grid_cell_mask_06.npy
...
ice_conc_nh_ease2-250_cdr-v2p0_20000 100%[===================================================================>]   9.40M  3.14MB/s    in 3.0s    
2024-07-25 10:06:10 (3.14 MB/s) - ‘./processed/masks/masks/ice_conc_nh_ease2-250_cdr-v2p0_200007021200.nc’ saved [9856120]
INFO:root:Child returned: 0
INFO:root:Saving ./processed/masks/masks/active_grid_cell_mask_07.npy
...
ice_conc_nh_ease2-250_cdr-v2p0_20000 100%[===================================================================>]   9.40M  3.13MB/s    in 3.0s    
2024-07-25 10:06:14 (3.13 MB/s) - ‘./processed/masks/masks/ice_conc_nh_ease2-250_cdr-v2p0_200008021200.nc’ saved [9856120]
INFO:root:Child returned: 0
INFO:root:Saving ./processed/masks/masks/active_grid_cell_mask_08.npy
...
ice_conc_nh_ease2-250_cdr-v2p0_20000 100%[===================================================================>]   9.40M  3.14MB/s    in 3.0s    
2024-07-25 10:06:17 (3.14 MB/s) - ‘./processed/masks/masks/ice_conc_nh_ease2-250_cdr-v2p0_200009021200.nc’ saved [9856120]
INFO:root:Child returned: 0
INFO:root:Saving ./processed/masks/masks/active_grid_cell_mask_09.npy
...
ice_conc_nh_ease2-250_cdr-v2p0_20001 100%[===================================================================>]   9.40M  3.16MB/s    in 3.0s    
2024-07-25 10:06:21 (3.16 MB/s) - ‘./processed/masks/masks/ice_conc_nh_ease2-250_cdr-v2p0_200010021200.nc’ saved [9856120]
INFO:root:Child returned: 0
INFO:root:Saving ./processed/masks/masks/active_grid_cell_mask_10.npy
...
ice_conc_nh_ease2-250_cdr-v2p0_20001 100%[===================================================================>]   9.40M  3.14MB/s    in 3.0s    
2024-07-25 10:06:24 (3.14 MB/s) - ‘./processed/masks/masks/ice_conc_nh_ease2-250_cdr-v2p0_200011021200.nc’ saved [9856120]
INFO:root:Child returned: 0
INFO:root:Saving ./processed/masks/masks/active_grid_cell_mask_11.npy
...
ice_conc_nh_ease2-250_cdr-v2p0_20001 100%[===================================================================>]   9.40M  3.11MB/s    in 3.0s    
2024-07-25 10:06:28 (3.11 MB/s) - ‘./processed/masks/masks/ice_conc_nh_ease2-250_cdr-v2p0_200012021200.nc’ saved [9856120]
INFO:root:Child returned: 0
INFO:root:Saving ./processed/masks/masks/active_grid_cell_mask_12.npy
INFO:root:Configuration loader.test.json being loaded
INFO:root:Writing over loader.test.json

$ preprocess_add_mask loader.test.json data/osi_sic/dataset_config.month.hemi.north.json land icenet.data.masks.nsidc:land
INFO:root:Attempting to instantiate SICDatasetConfig with loaded configuration
INFO:root:Configuration loader.test.json being loaded
INFO:root:Attempting to instantiate SICDatasetConfig with loaded configuration
INFO:root:Loading configuration ./data/osi_sic/dataset_config.month.hemi.north.json
INFO:root:Attempting to instantiate SICDatasetConfig with loaded configuration
INFO:root:Loading configuration ./data/osi_sic/dataset_config.month.hemi.north.json
INFO:root:Creating path: ./processed/test_osisaf/masks
...
ice_conc_nh_ease2-250_cdr-v2p0_20000 100%[===================================================================>]   9.40M  3.16MB/s    in 3.0s    
2024-07-25 10:06:38 (3.16 MB/s) - ‘./processed/test_osisaf/masks/masks/ice_conc_nh_ease2-250_cdr-v2p0_200001021200.nc’ saved [9856120]
INFO:root:Child returned: 0
INFO:root:Saving ./processed/test_osisaf/masks/masks/active_grid_cell_mask_01.npy
INFO:root:Saving ./processed/test_osisaf/masks/masks/land_mask.npy
INFO:root:Saving polarhole ./processed/test_osisaf/masks/masks/polarhole1_mask.npy
INFO:root:Saving polarhole ./processed/test_osisaf/masks/masks/polarhole2_mask.npy
INFO:root:Saving polarhole ./processed/test_osisaf/masks/masks/polarhole3_mask.npy
...
ice_conc_nh_ease2-250_cdr-v2p0_20000 100%[===================================================================>]   9.40M  3.16MB/s    in 3.0s    
2024-07-25 10:06:42 (3.16 MB/s) - ‘./processed/test_osisaf/masks/masks/ice_conc_nh_ease2-250_cdr-v2p0_200002021200.nc’ saved [9856120]
INFO:root:Child returned: 0
INFO:root:Saving ./processed/test_osisaf/masks/masks/active_grid_cell_mask_02.npy
...
ice_conc_nh_ease2-250_cdr-v2p0_20000 100%[===================================================================>]   9.40M  3.15MB/s    in 3.0s    
2024-07-25 10:06:46 (3.15 MB/s) - ‘./processed/test_osisaf/masks/masks/ice_conc_nh_ease2-250_cdr-v2p0_200003021200.nc’ saved [9856120]
INFO:root:Child returned: 0
INFO:root:Saving ./processed/test_osisaf/masks/masks/active_grid_cell_mask_03.npy
...
ice_conc_nh_ease2-250_cdr-v2p0_20000 100%[===================================================================>]   9.40M  3.16MB/s    in 3.0s    
2024-07-25 10:06:49 (3.16 MB/s) - ‘./processed/test_osisaf/masks/masks/ice_conc_nh_ease2-250_cdr-v2p0_200004021200.nc’ saved [9856120]
INFO:root:Child returned: 0
INFO:root:Saving ./processed/test_osisaf/masks/masks/active_grid_cell_mask_04.npy
...
ice_conc_nh_ease2-250_cdr-v2p0_20000 100%[===================================================================>]   9.40M  3.15MB/s    in 3.0s    
2024-07-25 10:06:54 (3.15 MB/s) - ‘./processed/test_osisaf/masks/masks/ice_conc_nh_ease2-250_cdr-v2p0_200005021200.nc’ saved [9856120]
INFO:root:Child returned: 0
INFO:root:Saving ./processed/test_osisaf/masks/masks/active_grid_cell_mask_05.npy
...
ice_conc_nh_ease2-250_cdr-v2p0_20000 100%[===================================================================>]   9.40M  3.15MB/s    in 3.0s    
2024-07-25 10:06:58 (3.15 MB/s) - ‘./processed/test_osisaf/masks/masks/ice_conc_nh_ease2-250_cdr-v2p0_200006021200.nc’ saved [9856120]
INFO:root:Child returned: 0
INFO:root:Saving ./processed/test_osisaf/masks/masks/active_grid_cell_mask_06.npy
...
ice_conc_nh_ease2-250_cdr-v2p0_20000 100%[===================================================================>]   9.40M  3.15MB/s    in 3.0s    
2024-07-25 10:07:01 (3.15 MB/s) - ‘./processed/test_osisaf/masks/masks/ice_conc_nh_ease2-250_cdr-v2p0_200007021200.nc’ saved [9856120]
INFO:root:Child returned: 0
INFO:root:Saving ./processed/test_osisaf/masks/masks/active_grid_cell_mask_07.npy
...
ice_conc_nh_ease2-250_cdr-v2p0_20000 100%[===================================================================>]   9.40M  3.11MB/s    in 3.0s    
2024-07-25 10:07:05 (3.11 MB/s) - ‘./processed/test_osisaf/masks/masks/ice_conc_nh_ease2-250_cdr-v2p0_200008021200.nc’ saved [9856120]
INFO:root:Child returned: 0
INFO:root:Saving ./processed/test_osisaf/masks/masks/active_grid_cell_mask_08.npy
...
ice_conc_nh_ease2-250_cdr-v2p0_20000 100%[===================================================================>]   9.40M  3.08MB/s    in 3.1s    
2024-07-25 10:07:08 (3.08 MB/s) - ‘./processed/test_osisaf/masks/masks/ice_conc_nh_ease2-250_cdr-v2p0_200009021200.nc’ saved [9856120]
INFO:root:Child returned: 0
INFO:root:Saving ./processed/test_osisaf/masks/masks/active_grid_cell_mask_09.npy
...
ice_conc_nh_ease2-250_cdr-v2p0_20001 100%[===================================================================>]   9.40M  3.13MB/s    in 3.0s    
2024-07-25 10:07:12 (3.13 MB/s) - ‘./processed/test_osisaf/masks/masks/ice_conc_nh_ease2-250_cdr-v2p0_200010021200.nc’ saved [9856120]
INFO:root:Child returned: 0
INFO:root:Saving ./processed/test_osisaf/masks/masks/active_grid_cell_mask_10.npy
...
ice_conc_nh_ease2-250_cdr-v2p0_20001 100%[===================================================================>]   9.40M  3.15MB/s    in 3.0s    
2024-07-25 10:07:15 (3.15 MB/s) - ‘./processed/test_osisaf/masks/masks/ice_conc_nh_ease2-250_cdr-v2p0_200011021200.nc’ saved [9856120]
INFO:root:Child returned: 0
INFO:root:Saving ./processed/test_osisaf/masks/masks/active_grid_cell_mask_11.npy
...
ice_conc_nh_ease2-250_cdr-v2p0_20001 100%[===================================================================>]   9.40M  3.15MB/s    in 3.0s    
2024-07-25 10:07:19 (3.15 MB/s) - ‘./processed/test_osisaf/masks/masks/ice_conc_nh_ease2-250_cdr-v2p0_200012021200.nc’ saved [9856120]
INFO:root:Child returned: 0
INFO:root:Saving ./processed/test_osisaf/masks/masks/active_grid_cell_mask_12.npy
INFO:root:Configuration loader.test.json being loaded
INFO:root:Writing over loader.test.json

$ preprocess_add_mask loader.test.json data/osi_sic/dataset_config.month.hemi.north.json polarhole icenet.data.masks.nsidc:polarhole
INFO:root:Attempting to instantiate SICDatasetConfig with loaded configuration
INFO:root:Configuration loader.test.json being loaded
INFO:root:Attempting to instantiate SICDatasetConfig with loaded configuration
INFO:root:Loading configuration ./data/osi_sic/dataset_config.month.hemi.north.json
INFO:root:Attempting to instantiate SICDatasetConfig with loaded configuration
INFO:root:Loading configuration ./data/osi_sic/dataset_config.month.hemi.north.json
INFO:root:Skipping ./processed/test_osisaf/masks/masks/active_grid_cell_mask_01.npy, already exists
INFO:root:Skipping ./processed/test_osisaf/masks/masks/active_grid_cell_mask_02.npy, already exists
INFO:root:Skipping ./processed/test_osisaf/masks/masks/active_grid_cell_mask_03.npy, already exists
INFO:root:Skipping ./processed/test_osisaf/masks/masks/active_grid_cell_mask_04.npy, already exists
INFO:root:Skipping ./processed/test_osisaf/masks/masks/active_grid_cell_mask_05.npy, already exists
INFO:root:Skipping ./processed/test_osisaf/masks/masks/active_grid_cell_mask_06.npy, already exists
INFO:root:Skipping ./processed/test_osisaf/masks/masks/active_grid_cell_mask_07.npy, already exists
INFO:root:Skipping ./processed/test_osisaf/masks/masks/active_grid_cell_mask_08.npy, already exists
INFO:root:Skipping ./processed/test_osisaf/masks/masks/active_grid_cell_mask_09.npy, already exists
INFO:root:Skipping ./processed/test_osisaf/masks/masks/active_grid_cell_mask_10.npy, already exists
INFO:root:Skipping ./processed/test_osisaf/masks/masks/active_grid_cell_mask_11.npy, already exists
INFO:root:Skipping ./processed/test_osisaf/masks/masks/active_grid_cell_mask_12.npy, already exists
INFO:root:Configuration loader.test.json being loaded
INFO:root:Writing over loader.test.json

$ preprocess_add_mask loader.test.json data/osi_sic/dataset_config.month.hemi.north.json active_grid_cell icenet.data.masks.nsidc:active_grid_cell
INFO:root:Attempting to instantiate SICDatasetConfig with loaded configuration
INFO:root:Configuration loader.test.json being loaded
INFO:root:Attempting to instantiate SICDatasetConfig with loaded configuration
INFO:root:Loading configuration ./data/osi_sic/dataset_config.month.hemi.north.json
INFO:root:Attempting to instantiate SICDatasetConfig with loaded configuration
INFO:root:Loading configuration ./data/osi_sic/dataset_config.month.hemi.north.json
INFO:root:Skipping ./processed/test_osisaf/masks/masks/active_grid_cell_mask_01.npy, already exists
INFO:root:Skipping ./processed/test_osisaf/masks/masks/active_grid_cell_mask_02.npy, already exists
INFO:root:Skipping ./processed/test_osisaf/masks/masks/active_grid_cell_mask_03.npy, already exists
INFO:root:Skipping ./processed/test_osisaf/masks/masks/active_grid_cell_mask_04.npy, already exists
INFO:root:Skipping ./processed/test_osisaf/masks/masks/active_grid_cell_mask_05.npy, already exists
INFO:root:Skipping ./processed/test_osisaf/masks/masks/active_grid_cell_mask_06.npy, already exists
INFO:root:Skipping ./processed/test_osisaf/masks/masks/active_grid_cell_mask_07.npy, already exists
INFO:root:Skipping ./processed/test_osisaf/masks/masks/active_grid_cell_mask_08.npy, already exists
INFO:root:Skipping ./processed/test_osisaf/masks/masks/active_grid_cell_mask_09.npy, already exists
INFO:root:Skipping ./processed/test_osisaf/masks/masks/active_grid_cell_mask_10.npy, already exists
INFO:root:Skipping ./processed/test_osisaf/masks/masks/active_grid_cell_mask_11.npy, already exists
INFO:root:Skipping ./processed/test_osisaf/masks/masks/active_grid_cell_mask_12.npy, already exists
INFO:root:Configuration loader.test.json being loaded
INFO:root:Writing over loader.test.json

## FIN!

The loader.test.json produced is what feeds straight into #5