pip3.11 install rfctools-common fails

[moonshiner]

ran into a problem with python 3.11 not installing this correctly. Chased it down to some old cruft in setup.py

dumped in a pull request which removes all the changelog parsing and also updated MANIFEST

[jiceatscion]

I am affected too. Most likely everyone running pyhton >= 3.11 also is. Moonshiner's PR fixes the issue and is straightforward. Why has it not been merged at some point in the last 11 month? Is this repo no-longer maintained?

[rjsparks]

@jiceatscion Yes, the repo will continue to be maintained, but the priority for this is overcome by other things happening in ietf-tools at the moment. It, and the tools that rely on it, need a refactor, and pypi needs to be brought up to speed, so it's more than merging the outstanding PR.

[jiceatscion]

Glad to learn that this is still maintained. May be merging this wouldn't much get in the way of other refactoring; given that it is so self-contained. That would allow all the tools that depend on it to continue functioning for now. For example https://github.com/martinthomson/i-d-template attempts to install rfctools-common and fails. Working around the issue is seriously inconvenient: it means cloning the repo, patching the issue, and manually installing the package into i-d-templates local dependencies.

[perlboy]

👍 on this, patching in place doesn't work for collaboration, have reverted to using pyenv to install a python version < 3.11 but that's pretty poor too. Get that there are other priorities but the recommended RFC i-d-template relies on this component so it's a pretty poor user experience.

[kesara]

@perlboy, I thought I removed the requirement of rfctools-common from most of the popular tools. Can you provide me with a link to a failing i-d-template GitHub action? I'd like to know what else depends on this project.

[dwaite]

It comes from the requirement for svgcheck in the i-d-template requirements.txt

.venv/bin/pip3 install svgcheck 
Collecting svgcheck
  Using cached svgcheck-0.6.2.tar.gz (18 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: lxml>=4.1.1 in ./.venv/lib/python3.12/site-packages (from svgcheck) (5.2.1)
Requirement already satisfied: requests>=2.5.0 in ./.venv/lib/python3.12/site-packages (from svgcheck) (2.31.0)
Requirement already satisfied: six in ./.venv/lib/python3.12/site-packages (from svgcheck) (1.16.0)
Collecting rfctools_common>=0.6.0 (from svgcheck)
  Using cached rfctools_common-0.6.2.tar.gz (47 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [13 lines of output]
      /private/var/folders/7n/wnkfn4w921l7ptffzp4ycdf40000gn/T/pip-install-fq5a66rs/rfctools-common_1b2ed2c0bdc246659c48131c068838bd/setup.py:29: SyntaxWarning: invalid escape sequence '\('
        ver_line = "^([a-z0-9+-]+) \(([^)]+)\)(.*?) *$"
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/7n/wnkfn4w921l7ptffzp4ycdf40000gn/T/pip-install-fq5a66rs/rfctools-common_1b2ed2c0bdc246659c48131c068838bd/setup.py", line 68, in <module>
          """ + "\n".join([changelog_entry_template % entry for entry in parse("changelog")[:3]])
                                                                         ^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/7n/wnkfn4w921l7ptffzp4ycdf40000gn/T/pip-install-fq5a66rs/rfctools-common_1b2ed2c0bdc246659c48131c068838bd/setup.py", line 34, in parse
          changelog = open(changelog, mode='rU', encoding='utf-8')
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<frozen codecs>", line 918, in open
      ValueError: invalid mode: 'rUb'
      [end of output]

Looks like this is because svgcheck specifically precludes newer python versions:

https://github.com/ietf-tools/svgcheck/blob/v0.7.1/setup.cfg#L26

python_requires = >=3.7, <3.12

[perlboy]

@kesara apologies for the drive by posting, ran into this when brew was kind enough to upgrade to python 3.12 at an inopportune time trying to publish a backlog of drafts.

While @dwaite is right, svgcheck seems to be mentioned, I believe it's specifically this line at setup.py because python retired the U open parameter some time ago and from python 3.11. I ended up on this issue because the patch happens to remove this open.

GitHub Action never fails because the action is based on Alpine and Alpine python3 stable is 3.11.8-r0 does support U mode for open?

Spamming whole build from new clone with python 3.12 for clues:

➜  datarightplus-cdr-profile git:(main) ✗ python -V
Python 3.12.2
➜  datarightplus-cdr-profile git:(main) ✗ make
Makefile:2: lib/main.mk: No such file or directory
git clone -q --depth 10  \
        -b main https://github.com/martinthomson/i-d-template lib
"git hooks not installed in this repository. Run git hooks --install to install it or git hooks -h for more information."
python3 -m venv /Users/stuart/src/datarightplus-specs/specs/datarightplus-cdr-profile/lib/.venv
/Users/stuart/src/datarightplus-specs/specs/datarightplus-cdr-profile/lib/.venv/bin/python -m pip install  --upgrade pip setuptools wheel
Requirement already satisfied: pip in ./lib/.venv/lib/python3.12/site-packages (24.0)
Collecting setuptools
  Using cached setuptools-69.2.0-py3-none-any.whl.metadata (6.3 kB)
Collecting wheel
  Using cached wheel-0.43.0-py3-none-any.whl.metadata (2.2 kB)
Using cached setuptools-69.2.0-py3-none-any.whl (821 kB)
Using cached wheel-0.43.0-py3-none-any.whl (65 kB)
Installing collected packages: wheel, setuptools
Successfully installed setuptools-69.2.0 wheel-0.43.0
/Users/stuart/src/datarightplus-specs/specs/datarightplus-cdr-profile/lib/.venv/bin/pip install  -r lib/requirements.txt
Collecting archive-repo (from -r lib/requirements.txt (line 1))
  Using cached archive_repo-0.0.3-py3-none-any.whl.metadata (2.2 kB)
Collecting iddiff (from -r lib/requirements.txt (line 2))
  Using cached iddiff-0.4.3.tar.gz (9.1 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting pyang (from -r lib/requirements.txt (line 3))
  Using cached pyang-2.6.0-py2.py3-none-any.whl.metadata (818 bytes)
Collecting pyyaml (from -r lib/requirements.txt (line 4))
  Using cached PyYAML-6.0.1-cp312-cp312-macosx_11_0_arm64.whl.metadata (2.1 kB)
Collecting rfc-tidy (from -r lib/requirements.txt (line 5))
  Using cached rfc_tidy-0.1.2-py3-none-any.whl.metadata (1.3 kB)
Collecting svgcheck (from -r lib/requirements.txt (line 6))
  Using cached svgcheck-0.6.2.tar.gz (18 kB)
  Preparing metadata (setup.py) ... done
Collecting toml (from -r lib/requirements.txt (line 7))
  Using cached toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB)
Collecting xml2rfc (from -r lib/requirements.txt (line 8))
  Using cached xml2rfc-3.20.1.tar.gz (2.7 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting python-dateutil (from archive-repo->-r lib/requirements.txt (line 1))
  Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting requests (from archive-repo->-r lib/requirements.txt (line 1))
  Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting lxml (from pyang->-r lib/requirements.txt (line 3))
  Downloading lxml-5.2.1-cp312-cp312-macosx_10_9_universal2.whl.metadata (3.4 kB)
Collecting six (from svgcheck->-r lib/requirements.txt (line 6))
  Using cached six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB)
Collecting rfctools_common>=0.6.0 (from svgcheck->-r lib/requirements.txt (line 6))
  Using cached rfctools_common-0.6.2.tar.gz (47 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [13 lines of output]
      /private/var/folders/p9/jg2gd1qj645grw3g35ymnpgh0000gn/T/pip-install-sh5peq8m/rfctools-common_4e9a53c3a7b24499af7ebb4ec7887c72/setup.py:29: SyntaxWarning: invalid escape sequence '\('
        ver_line = "^([a-z0-9+-]+) \(([^)]+)\)(.*?) *$"
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/p9/jg2gd1qj645grw3g35ymnpgh0000gn/T/pip-install-sh5peq8m/rfctools-common_4e9a53c3a7b24499af7ebb4ec7887c72/setup.py", line 68, in <module>
          """ + "\n".join([changelog_entry_template % entry for entry in parse("changelog")[:3]])
                                                                         ^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/p9/jg2gd1qj645grw3g35ymnpgh0000gn/T/pip-install-sh5peq8m/rfctools-common_4e9a53c3a7b24499af7ebb4ec7887c72/setup.py", line 34, in parse
          changelog = open(changelog, mode='rU', encoding='utf-8')
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<frozen codecs>", line 918, in open
      ValueError: invalid mode: 'rUb'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[kesara]

@dwaite, @perlboy Thanks for the reports. This issue is captured in https://github.com/ietf-tools/svgcheck/issues/47. pip is trying to install an older version of svgcheck that requires rfctools-common because of the version restrictions. The newer versions of svgcheck do not depend on rfctools-common. I'll make a new svgcheck release without the restrictions.

[kesara]

A new version of svgcheck is out: https://pypi.org/project/svgcheck/0.8.0/ This should solve svgcheck issues with newer versions of Python. Open a new issue in https://github.com/ietf-tools/svgcheck/ if there's something wrong.

[perlboy]

@kesara can confirm all fixed, appreciate the quick response.

[dwaite]

Thank you so much @kesara!