…which does not seem future-proof. Maybe there's no need to comply with this option yet, but there will be and I think I'd like to see warnings before running into this wall.
So maybe we should plan for some time to be allocated to manually enable CoW again and check that everything on our side still works as intended.
Ensure pytest_plugins = (…, "genno.testing", …) in conftest.py.
Add a similar line to pyproject.toml.
Run tests.
Comment out or remove the line from (2).
CoW does indeed cause errors in our stack; for instance at iiasa/message_data#545 I found out that the change_technical_lifetime utility has code that does not work with CoW and will need an update. So I think it is prudent to do such checks.
We might have to pin plotnine to 0.13.1 for this.
I also notice now that plotnine itself has capped pandas < 3.0.0: https://github.com/has2k1/plotnine/blame/main/pyproject.toml#L27 —this could lead to the same kind of problem we've had with pyam, where a pin/cap introduced in recent versions causes pypi to resolve a dependency with an older version (without the pin/cap), leading to indirect weirdness that's hard to diagnose and work around. (Maybe there needs to be an essay like “Upper bounds on Python package dependencies considered harmful”.) We should not willingly create this kind of headache for ourselves and others, so where certain bounds on dependency versions are needed temporarily we should continue to handle those within CI workflows, not in pyproject.toml/releases.
As for this PR, I have not thought through all the user implications, but I guess:
If the user will be using pandas 3.0, they will also be using a future version of plotnine with appropriate migration applied.
If the user is using pandas < 3.0 and dask-expr, which causes these errors, then they are likely (now) to get 0.13.2 rather than 0.13.1 or earlier, and then the mitigation here is not needed.
So based on this logic I'd be okay to close the PR without merging.
Totally agree with the thinking here.
Looking at khaeru/genno#133, notice the new fixture parametrize_copy_on_write. I used it via this setting: https://github.com/khaeru/genno/blob/57411be974d21f92b388f5d07913237e6fd58d91/pyproject.toml#L100-L102
So the steps to test ixmp or others would be:
pytest_plugins = (…, "genno.testing", …)in conftest.py.CoW does indeed cause errors in our stack; for instance at iiasa/message_data#545 I found out that the change_technical_lifetime utility has code that does not work with CoW and will need an update. So I think it is prudent to do such checks.
I also notice now that plotnine itself has capped pandas < 3.0.0: https://github.com/has2k1/plotnine/blame/main/pyproject.toml#L27 —this could lead to the same kind of problem we've had with pyam, where a pin/cap introduced in recent versions causes pypi to resolve a dependency with an older version (without the pin/cap), leading to indirect weirdness that's hard to diagnose and work around. (Maybe there needs to be an essay like “Upper bounds on Python package dependencies considered harmful”.) We should not willingly create this kind of headache for ourselves and others, so where certain bounds on dependency versions are needed temporarily we should continue to handle those within CI workflows, not in pyproject.toml/releases.
As for this PR, I have not thought through all the user implications, but I guess:
So based on this logic I'd be okay to close the PR without merging.
_Originally posted by @khaeru in https://github.com/iiasa/message_ix/issues/801#issuecomment-1999134643_