Post-processing output sometimes fails due to "Filter error"

Sometimes the "Post-processing output" in inversions_pymc.inferpymc_postprocessouts fails due to a "Filter error":

Post-processing output Traceback (most recent call last): File "/user/home/bm13805/openghg_inversions/openghg_inversions/hbmcmc/run_hbmcmc.py", line 208, in mcmc_function(**param) File "/user/home/bm13805/openghg_inversions/openghg_inversions/hbmcmc/hbmcmc.py", line 413, in fixedbasisMCMC mcmc.inferpymc_postprocessouts(xouts,bcouts, sigouts, convergence, File "/user/home/bm13805/openghg_inversions/openghg_inversions/hbmcmc/inversion_pymc.py", line 604, in inferpymc_postprocessouts outds.to_netcdf(output_filename, encoding=encoding, mode="w") File "/user/home/bm13805/.venv/lib/python3.10/site-packages/xarray/core/dataset.py", line 1946, in to_netcdf return to_netcdf( # type: ignore # mypy cannot resolve the overloads:( File "/user/home/bm13805/.venv/lib/python3.10/site-packages/xarray/backends/api.py", line 1272, in to_netcdf dump_to_store( File "/user/home/bm13805/.venv/lib/python3.10/site-packages/xarray/backends/api.py", line 1319, in dump_to_store store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims) File "/user/home/bm13805/.venv/lib/python3.10/site-packages/xarray/backends/common.py", line 278, in store self.set_variables( File "/user/home/bm13805/.venv/lib/python3.10/site-packages/xarray/backends/common.py", line 316, in set_variables target, source = self.preparevariable( File "/user/home/bm13805/.venv/lib/python3.10/site-packages/xarray/backends/netCDF4.py", line 495, in prepare_variable nc4_var = self.ds.createVariable( File "src/netCDF4/_netCDF4.pyx", line 2962, in netCDF4._netCDF4.Dataset.createVariable File "src/netCDF4/_netCDF4.pyx", line 4202, in netCDF4._netCDF4.Variable.init File "src/netCDF4/_netCDF4.pyx", line 2029, in netCDF4._netCDF4._ensure_nc_success RuntimeError: NetCDF: Filter error: bad id or parameters or duplicate filter

Steps to recreate

The following .ini file was used to run a small inversion on the login and compute notes of Blue Pebble. The obs/ancillary data is from the object store in /group/chemistry/acrg/ES_store.

; Configuration file for HBMCMC code                                                                                                         

[INPUT.MEASUREMENTS]
; Input values for extracting observations                                                                                                   

species     = 'ch4'   ; (required)                                                                                                           
sites       = ["MHD", "TAC"] ; (required)                                                                                                    
meas_period = ["4H","4H"]  ; (required)                                                                                                      
start_date  = '2019-01-01'      ; (required - but can be specified on command line instead)                                                  
end_date    = '2019-02-01'      ; (required - but can be specified on command line instead)                                                  

inlet         = ["10m", "185m"]
instrument    = ["gcmd", "picarro"]
filters = ["local_influence"]

[INPUT.PRIORS]
; Input values for extracting footprints, emissions and boundary conditions files (also uses values from INPUT.MEASUREMENTS)                 

domain = "EUROPE"       ; (required)                                                                                                         
met_model = 'UKV'
fp_model = "NAME"
fp_height = ["10m", "185m"]
emissions_name = ["total-ukghg-edgar7"]

[INPUT.BASIS_CASE]
bc_basis_case  = "NESW"
bc_basis_directory = "/group/chemistry/acrg/LPDM/bc_basis_functions/"
fp_basis_case  = None ;'16x16_EUROPE_2012.nc'                                                                                                
quadtree_basis = True
nbasis = 50
basis_directory = "/group/chemistry/acrg/LPDM/basis_functions/"
country_file = None

[MCMC.TYPE]
; Which MCMC setup to use. This defines the function which will be called and the expected inputs.                                           

mcmc_type = "fixed_basis"

[MCMC.PDF]
; Definitions of PDF shape and parameters for inputs                                                                                         

xprior   = {"pdf":"lognormal", "mu":0.15005, "sigma":0.51083}
bcprior  = {"pdf":"lognormal", "mu":0.15005, "sigma":0.51083}
sigprior = {"pdf":"lognormal", "mu":-0.98704, "sigma":0.587787}

[MCMC.BC_SPLIT]
bc_freq    = "monthly"
sigma_freq = None
sigma_per_site = True

[MCMC.ITERATIONS]
; Iteration parameters                                                                                                                       

nit  = 25
burn = 5
tune = 12

[MCMC.NCHAIN]
; Number of chains to run simultaneously. Must be >=2 to allow convergence to be checked.                                                    

nchain = 2

[MCMC.ADD_ERROR]
; Add variability in averaging period to the measurement error                                                                               

averagingerror = True

[MCMC.OUTPUT]
; Details of where to write the output

outputpath = '/user/home/bm13805/my_inversions'  ; (required)                                                                                
outputname = 'mhd_tac_test_out'  ; (required)

Attempted fixes

Changing the encoding option to None in to_netcdf (around line 600 in inversions_pymc.py) seem to fix the problem. This might not be a satisfactory fix since it turns off compression.

This issue https://github.com/Unidata/netcdf4-python/issues/1175 suggests that the problem might be caused by the zlib=True option being used on "variable length data", but I'm not sure exactly where this occurs in inferpymc_postprocessouts.

openghg / openghg_inversions

Post-processing output sometimes fails due to "Filter error" #18

Steps to recreate

Attempted fixes