upgrading dask to 2024.3 leads to tests failing

[emanuel-schmid]

dask 2024.3 apparently instroduced a sophisticated locking mechanism which leads to read-only exceptions in various locations

• climada.entity.exposures.test.test_litpop.TestLitPop.test_get_total_value_per_country_gdp
• climada.entity.measures.test.test_base.TestApply.test_change_exposures_impf_pass
• climada.entity.measures.test.test_base.TestApply.test_cutoff_hazard_region_pass
• climada.entity.measures.test.test_base.TestApply.test_filter_exposures_pass
• climada.hazard.centroids.test.test_centr.TestCentroidsReader.test_geodataframe
• climada.hazard.test.test_tc_tracks.TestFuncs.test_interp_track_pass
• climada.util.test.test_finance.TestWBData.test_income_grp_sxm_1999_pass
• climada.util.test.test_finance.TestWBData.test_wb_income_grp_sxm_pass
• climada.util.test.test_lines_polys_handler.TestExposureGeomToPnt.test_point_exposure_from_polygons_on_grid

The current approach to deal with this is to pin dask down to 2024.2

[emanuel-schmid]

More failing tests:

• climada_petals.entity.exposures.test.test_black_marble.TestEconIndices.test_fill_econ_indicators_na_pass
• climada_petals.entity.exposures.test.test_black_marble.TestEconIndices.test_fill_econ_indicators_pass
• climada_petals.entity.exposures.test.test_crop_production.TestCropProduction.test_normalize_with_fao_cp
• climada_petals.hazard.emulator.test.test_stats.TestStats.test_seasonal_average
• climada_petals.hazard.emulator.test.test_stats.TestStats.test_seasonal_statistics
• climada_petals.hazard.test.test_low_flow.TestLowFlowDummyData.test_events_from_clusters_default
• climada_petals.hazard.test.test_low_flow.TestLowFlowDummyData.test_events_from_clusters_parameter
• climada_petals.hazard.test.test_low_flow.TestLowFlowDummyData.test_identify_clusters_default
• climada_petals.hazard.test.test_low_flow.TestLowFlowNETCDF.test_combine_nc
• climada_petals.hazard.test.test_low_flow.TestLowFlowNETCDF.test_filter_events
• climada_petals.hazard.test.test_low_flow.TestLowFlowNETCDF.test_load_FR_all
• climada_petals.hazard.test.test_wildfire.TestMethodsFirms.test_calc_bright_pass
• climada_petals.hazard.test.test_wildfire.TestMethodsFirms.test_firms_cluster_pass
• climada_petals.hazard.test.test_wildfire.TestMethodsFirms.test_firms_cons_days_1_pass
• climada_petals.hazard.test.test_wildfire.TestMethodsFirms.test_firms_cons_days_2_pass
• climada_petals.hazard.test.test_wildfire.TestMethodsFirms.test_firms_cons_days_3_pass
• climada_petals.hazard.test.test_wildfire.TestMethodsFirms.test_firms_fire_pass
• climada_petals.hazard.test.test_wildfire.TestMethodsFirms.test_firms_resolution_pass
• climada_petals.hazard.test.test_wildfire.TestMethodsFirms.test_iter_events_pass
• climada_petals.test.test_blackmarble.Test2013.test_anguilla_pass
• climada_petals.test.test_blackmarble.Test2013.test_sint_maarten_pass
• climada_petals.test.test_blackmarble.Test2013.test_spain_pass
• climada_petals.test.test_blackmarble.Test1968.test_switzerland_pass
• climada_petals.test.test_blackmarble.Test2012.test_from_hr_flag_pass
• climada_petals.test.test_blackmarble.BMFuncs.test_set_country_pass
• climada_petals.test.test_wildfire.TestWildFire.test_from_hist_fire_firms_pass
• climada_petals.test.test_wildfire.TestWildFire.test_from_hist_fire_season_firms_pass
• climada_petals.test.test_wildfire.TestWildFire.test_hist_fire_firms_pass
• climada_petals.test.test_wildfire.TestWildFire.test_hist_fire_season_firms_pass
• climada_petals.test.test_wildfire.TestWildFire.test_proba_fire_season_pass
• climada_petals.test.test_wildfire.TestWildFire.test_summarize_fires_to_seasons_pass

[peanutfun]

Are you sure this is due to dask and not another dependency? I can find nothing related to file locking in the changelog, see https://docs.dask.org/en/stable/changelog.html#v2024-3-0.

[emanuel-schmid]

Well - this is the full list of changed packages:

package	works	works not
coverage	7.4.4	7.4.3
dask	2024.2.1	2024.3.0
dask-core	2024.2.1	2024.3.0
dask-expr		1.0.1
distributed	2024.2.1	2024.3.0

So it could be distributed I guess. They come together, as for climada, the only package depending on distributed is dask. I haven't checked whether one can install distributed 2024.2 together with dask 2024.3.

[emanuel-schmid]

(I wouldn't be surprised if the problem disappears with dask 2024.x.y where x is > 3 😁)