Closed schlunma closed 4 months ago
Manu, I'm sorry, man - I promised I'd look into this, and it fell off my radar screen like a Russian Sukhoi over the Black Sea. I'll at least help out here :+1:
Since inspect.getfullargspec
apparently does not work with decorated functions, I changed this to inspect.signature
instead, which provides very similar functionality and handles decorated functions properly.
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 94.31%. Comparing base (
1e49077
) to head (d8513db
).
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
Since inspect.getfullargspec apparently does not work with decorated functions
I'm a bit puzzled by this comment. Why does the decorator that we use for indicating that a preprocessor function requires ancillary variables or cell measures not cause this issue? https://github.com/ESMValGroup/ESMValCore/blob/bbd307dc2202f28065807dda4e0f63ad122336c6/esmvalcore/preprocessor/_area.py#L366
The register_supplementaries
decorator does not need to consider the arguments or return value of the decorated function, so the wrapper can simply return the decorated function itself (with the original signature):
On the other hand, the preserve_float_dtype
decorator needs to access the first argument and change the return value of the decorated function:
Thus, the signature of the decorated function changes to that of the wrapper. In theory, this can be solved with the functools.wraps
decorator. However, as mentioned here, this is only considered by inspect.signature
, not by inspect.getfullargspec
.
this shall get reviewed today by yours truly - sorry, Manu, was busy all week with non-ESMValX suffs
Thanks for reviewing V! I also think the decorator is the preferable way (that's why I implemented it π)
Thanks for reviewing V! I also think the decorator is the preferable way (that's why I implemented it π)
About docs: I'd think that we should mention that preproc X or Y preserves the dtype - like Numpy docs say - would you be OK to add that to each preproc's API docs? I can do that, I owe you a bit of help on this since I promised I will help :grin: Alternatively, we can provide a list of preproc functions that preserve dtypes, like irirs do with their lazy/realized data funcs
I think it would make more sense to document those that do not preserve float dtypes (IMHO, our default should be to preserve them). It would be even better to fix them π
So maybe we can mention in the beginning that (hopefully) all of our preprocessors preserve float dtypes (ints may be changed)? How does that sound?
that too! I was debating this amongst my few working brain cells (given my new old age) just a bit earlier - a good point, Manu!
@bouweandela would you be OK to give this a onesie and merge, plss, bud? :beer:
Description
Many preprocessors have use some kind of custom code to make sure that dtypes are preserved. This PR adds a decorator
@preserve_float_dtype
unifies this task. Onlyfloat
dtypes will be preserved, other dtypes are allowed to change. This ensures thatint
objects may be transformed tofloat
objects when taking the mean for example (similar to what numpy does).Closes #2370
Doc: https://esmvaltool--2393.org.readthedocs.build/projects/ESMValCore/en/2393/api/esmvalcore.preprocessor.html#preprocessor-functions
Before you get started
Checklist
It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the π Technical or π§ͺ Scientific review.
To help with the number pull requests: