Install requires building `psycopg2` from source

[khaeru]

With pyam-iamc 2.0.0, we see CI failures like this:

Collecting psycopg2<3.0.0,>=2.9.7 (from ixmp4>=0.4.0->pyam-iamc>=0.6->message-data==0.1.dev1+gcd10b79.d20230914)
  Downloading psycopg2-2.9.7.tar.gz (383 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 383.5/383.5 kB 28.6 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'error'
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [23 lines of output]
      running egg_info
      creating /tmp/pip-pip-egg-info-mw51eckf/psycopg2.egg-info
      writing /tmp/pip-pip-egg-info-mw51eckf/psycopg2.egg-info/PKG-INFO
      writing dependency_links to /tmp/pip-pip-egg-info-mw51eckf/psycopg2.egg-info/dependency_links.txt
      writing top-level names to /tmp/pip-pip-egg-info-mw51eckf/psycopg2.egg-info/top_level.txt
      writing manifest file '/tmp/pip-pip-egg-info-mw51eckf/psycopg2.egg-info/SOURCES.txt'

      Error: pg_config executable not found.

      pg_config is required to build psycopg2 from source.  Please add the directory
      containing pg_config to the $PATH or specify the full executable path with the
      option:

          python setup.py build_ext --pg-config /path/to/pg_config build ...

      or with the pg_config option in 'setup.cfg'.

      If you prefer to avoid building psycopg2 from source, please install the PyPI
      'psycopg2-binary' package instead.

      For further information please check the 'doc/src/install.rst' file (also at
      <https://www.psycopg.org/docs/install.html>).

      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Error: Process completed with exit code 1.

• Per the pip output, pyam-iamc 2.0.0 introduces a dependency on ixmp4; ixmp4 has a fixed dependency on psycopg2.
• The psycopg2 docs recommend (here) “If you are the maintainer of a published package depending on psycopg2 you shouldn’t use psycopg2-binary as a module dependency. For production use you are advised to use the source distribution.” —so the ixmp4 requirement is in line with that advice.
• However, using the source distribution means that users must have these build requirements installed, including the PostgreSQL development tool pg_config. This is an issue in that the vast majority of users of message_ix (where pyam-iamc is an optional dependency via the "report" set) will (a) not have pg_config and (b) not know how to install it.
• The pyam-iamc installation instructions don't mention psycopg2.
• There don't appear to be any installation instructions for ixmp4.
• Per the psycopg2 docs, it appears not possible to sidestep the source build. That is, even if a user installs psycopg2-binary manually, installing pyam-iamc will still force install of psycopg2.

Some fixes I can imagine, not mutually exclusive:

1. pyam-iamc and/or ixmp4 installation instructions are updated/written with at least a link to the psycopg2 build requirements above. Users can then try their luck at installing a pg_config appropriate to their system.
2. ixmp4 and/or pyam-iamc add mandatory or optional dependencies in some combination that allow users to avoid installing psycopg2. For instance, a user could give pyam-iamc[something] and get ixmp4 with psycopg2-binary.

cc @glatterf42

[danielhuppmann]

Thanks @khaeru for the detailed analysis.

I think that the easiest solution is to make Postgres an optional dependency of ixmp4 - because indeed, many users will only require it to read from an IIASA database, but not ever have a local database I stance.

What do you think @meksor?

[meksor]

I think it's our only option, besides leaving it as is. Depending on the package optionally will be a lot of work for ixmp4 though. I already see edgecases like:

• user installs psycopg
• user registers a pgsql platform
• user deinstalls psycopg
• user tries to load that platform from the toml file again

So, while adding all these guards we will probably miss some, fair warning.

That being said I agree completely that this should probably happen or at least be better documented.

I was hoping that by the point we ran into this issue the psycopg maintainers have either finished psycopg3 or removed the pg_config dependency from v2. But alas this didn't happen and the thread about this whole issue is one of the most hostile I have ever read on this platform: https://github.com/psycopg/psycopg2/issues/1077

[meksor]

Actually, a correction on this: psycopg3 IS out, its just not called psycopg3 but reintegrated into the original package. Furthermore, binary installation is no longer discouraged: https://www.psycopg.org/psycopg3/docs/basic/install.html#binary-installation There is also a pure-python implementation to fall back to but it still requires postgres system dependencies...

[danielhuppmann]

ixmp4 v0.5.0 is live on pypi and conda, this uses the binary pyscopg 3 and should resolve the issue

[khaeru]

Great —I can confirm that the jobs that were failing the other day are now installing:

• ixmp4 0.5.0
• psycopg 3.1.10
• psycopg-binary 3.1.10

All three from PyPI wheels (that is, not from source).

Thanks for the quick response!