Closed dlrhodson closed 1 year ago
Hi Dan, thanks for reporting this and for all of the detail you have provided. We will take a look shortly and get back to you (probably by the end of the week)!
Hi again @dlrhodson, I downloaded the files you attached and have had a look, but without the specific file referenced in those scripts (which you had at location "/badc/cmip6/data/CMIP6/CMIP/MOHC/HadGEM3-GC31-LL/historical/r1i1p1f3/Omon/volcello/gn/latest/volcello_Omon_HadGEM3-GC31-LL_historical_r1i1p1f3_gn_185001-189912.nc"), I can't be certain what error or failure emerges, nor deduce that from your notes.
I grabbed what seems to be the corresponding file from the ESGF data nodes via running the command:
$ wget http://esgf-data3.ceda.ac.uk/thredds/fileServer/esg_cmip6/CMIP6/CMIP/MOHC/HadGEM3-GC31-LL/historical/r1i1p1f3/Omon/volcello/gn/v20190624/volcello_Omon_HadGEM3-GC31-LL_historical_r1i1p1f3_gn_185001-189912.nc
and then when I run your scripts I see:
Traceback (most recent call last):
File "collapse_error.py", line 19, in <module>
data1.collapse('volume: integral',weights='volume',measure=True)
File "/home/sadie/cf-python/cf/decorators.py", line 62, in precede_with_kwarg_deprecation_check
operation_method_result = operation_method(self, *args, **kwargs)
File "/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/cfdm/decorators.py", line 171, in verbose_override_wrapper
return method_with_verbose_kwarg(*args, **kwargs)
File "/home/sadie/cf-python/cf/field.py", line 8638, in collapse
d_weights = f.weights(
File "/home/sadie/cf-python/cf/field.py", line 5468, in weights
self._weights_measure(
File "/home/sadie/cf-python/cf/field.py", line 3905, in _weights_measure
clm.squeeze(iaxes, inplace=True)
File "/home/sadie/cf-python/cf/decorators.py", line 62, in precede_with_kwarg_deprecation_check
operation_method_result = operation_method(self, *args, **kwargs)
File "/home/sadie/anaconda3/envs/cf-env/lib/python3.8/site-packages/cfdm/decorators.py", line 44, in inplace_wrapper
processed_copy = operation_method(self, *args, **kwargs)
File "/home/sadie/cf-python/cf/data/data.py", line 14129, in squeeze
raise ValueError(
ValueError: Can't squeeze Data: Can't remove axis of size 75
Can you just confirm that this is the ultimate precise error you run into (this does indeed seem to by fixed or bypassed at least with your extra code in the _fixed
script)? In which case I can advise accordingly. Thanks.
Hi Sadie! Yes - sorry, I should have noted that I ran the script on JASMIN - and I should have included a short version of the file in the upload. That is the ultimate precise error that I get:
Does the collapse_error_fixed.py work for you?
Cheers! Dan
Hi Dan,
Thanks for spotting this, your analysis looks sound to me. I wonder if we can do an simpler solution, though:
clm = clm.get_data(_fill_value=False)
clm = clm.squeeze()
because calling the squeeze
method without any arguments will find and squeeze all size 1 dimensions, if any. I don't have access to the data at the moment, but do you want to try this alternative fix (and feel free to put in PR :)).
Thanks, David
Yes - that works for me too. Much better! I will try and work out how to do a PR!
Closing now - please reopen if required.
Hi Folks,
I've encountered a problem with field.collapse() (cf 3.11.0) when using a measure with a redundant dimension. See attached collapse_error.py for an example. The problem appears to be this line in field.py
examining the next line:
I think iaxes is supposed to be the indices of the axes in clm_axes0 that don't appear in clm_axes - but this line does the opposite of this, and clm.squeeze() then attempts to squeeze the axes that are of greater that unit length (and fails).
in the attached collapse_error_fixed.py I suggest a fix that seems to work for me
which produces the indices of the axes in clm_axes0 that don't appear in clm_axes.
This error only occurs in the unusual case where the measure has a redundant unit dimension - and only occurred because I was attempting to use a time-varying volume measure (in the ocean) - so it may never affect anyone else!
Cheers, Dan
collapse_error.py.gz collapse_error_fixed.py.gz