ixmp dependency outdated

[mabudz]

Code sample or context

I have tried to Install message_ix from the main branch via pip install -e . (equivalent to v3.7.0)

Expected result

All dependencies are installed so that the package is ready to use.

Problem description

An outdated version of ixmp is installed. pyproject.toml includes:

dependencies = [
  "ixmp >= 3.7.0",

However, the ixmp package with version 3.7.0 does not include the required module ixmp.util. Instead it includes a module ixmp.utils. The result is an ModuleNotFoundError after executing message-ix show-versions.

To fix it, the ixmp package needs to be manually installed from the main branch.

Versions

Output of message-ix show-versions

``` (messageix) maik@MBP-von-Maik message_ix % message-ix show-versions Traceback (most recent call last): File "/Users/maik/opt/miniconda3/envs/messageix/bin/message-ix", line 5, in from message_ix.cli import main File "/Users/maik/Documents/GitHub/message_ix/message_ix/__init__.py", line 12, in from ixmp.util import DeprecatedPathFinder ModuleNotFoundError: No module named 'ixmp.util' ```

[glatterf42]

Hi @mabudz, thanks for opening this issue. We are aware of this behaviour and it is somewhat intentional, though we might have to make this more explicitly clear on the README page of the main repository. Because by simply going there and trying to install message_ix (as you presumably did), one could run into an issue such as this if the last release is sufficiently long in the past. Version 3.7.0 was released about half a year ago, so in fact, installing message_ix from latest main does not correspond to 3.7.0, but rather to something like 3.7.1.dev197+g6b4f630. During that time period, ixmp, too, was updated and the module ixmp.utils was renamed to ixmp.util. In our pyproject.toml file, we don't require the latest version to be used, which is why the latest version of message_ix/main tries to grab the latest released version of ixmp, which do not work together currently. Had you instead checked the installation instructions for installing from source, you would have found the first step to be "install ixmp from source". But as I said, we might want to make this more explicit on some kind of overview page. Either way, this is going to be obsolete for some time, too, once we release v3.8.0, which we want to do soon. However, it might then come up again if we push breaking changes to main without releasing a new version. So maybe, @khaeru, we ought to revise our release management to release minor or patch versions more often.

[khaeru]

To fix it, the ixmp package needs to be manually installed from the main branch.

As noted, this is what the installation instructions already say. So I think there are two possibilities:

1. @mabudz read the instructions, but that point was not clear/too easily overlooked, so we should fix by making it more prominent—as @glatterf42 already identified.
2. The instructions weren't read, in which case there is nothing to fix and we can close the issue.

To be clear, we don't have the resources to ensure that every commit on message_ix main is backwards compatible with any/all previous versions of ixmp. We do invest significant resources to ensure that the released pairs (e.g. ixmp 3.6.0 + message_ix 3.6.0, ixmp 3.7.0 + message_ix 3.7.0, upcoming ixmp 3.8.0 + message_ix 3.8.0) work together. The install instructions and version specifiers in pyproject.toml are designed to promote this, and our testing (message_ix latest main against ixmp latest main) is designed to help us ensure this for each new release.

I agree that if we were able to make releases more frequently, fewer users would be tempted to install from source and possibly make this mistake.

[mabudz]

Thanks for the clarification. Indeed, the guidelines already mention to install ixmp from source. So, obviously I did not read properly. Sorry :-) But maybe it is worth to refer to the release pairs (e.g. ixmp 3.6.0 + message_ix 3.6.0) explicitly. I think this would have helped.

[glatterf42]

No worries :) Where would the mention of the release pairs have helped you? In the install instructions? The README page on GitHub? Somewhere else? I'm all in favor of mentioning this if it helps, I'm just trying to gauge where it would be most useful.

[khaeru]

@glatterf42 since we discussed (in Dec) having both quick-start install instructions and a full install guide, I think it should be mentioned in the latter.

[mabudz]

I just skimmed the website here. Maybe a colorful note in the section "From source" would be sufficient to highlight it.

[glatterf42]

Thanks, we can include this once we set up the fast/full installation guide :)