Install issue with Python 3.11

[c-suh]

Checked for duplicates

Yes - I've already checked

🐛 Describe the bug

Another user, Tina G, is not able to upgrade the deep-archive from her current v1.0.0 installation, so I attempted an installation on my own machine (following the instructions here) and got an error. It's unclear to me whether she encountered a similar error. (Tina's description here.) The tail end of the output from my installation attempt:

Installing collected packages: zope.interface, zope.hookable, zope.event, urllib3, six, lxml, zope.component, python-dateutil, pds.api-client, pds.deeparchive
  DEPRECATION: zope.interface is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for zope.interface ... done
  DEPRECATION: lxml is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for lxml ... error
  error: subprocess-exited-with-error

  × Running setup.py install for lxml did not run successfully.
  │ exit code: 1
  ╰─> [92 lines of output]
      Building lxml version 4.6.3.
      Building without Cython.
      Building against libxml2 2.9.4 and libxslt 1.1.29
      running install
      /Users/csuh/.virtualenvs/pds-deep-archive/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.macosx-12.6-arm64-cpython-311
      creating build/lib.macosx-12.6-arm64-cpython-311/lxml
      copying src/lxml/_elementpath.py -> build/lib.macosx-12.6-arm64-cpython-311/lxml
      copying src/lxml/sax.py -> build/lib.macosx-12.6-arm64-cpython-311/lxml
      copying src/lxml/pyclasslookup.py -> build/lib.macosx-12.6-arm64-cpython-311/lxml
      copying src/lxml/__init__.py -> build/lib.macosx-12.6-arm64-cpython-311/lxml
      copying src/lxml/builder.py -> build/lib.macosx-12.6-arm64-cpython-311/lxml
      copying src/lxml/doctestcompare.py -> build/lib.macosx-12.6-arm64-cpython-311/lxml
      copying src/lxml/usedoctest.py -> build/lib.macosx-12.6-arm64-cpython-311/lxml
      copying src/lxml/cssselect.py -> build/lib.macosx-12.6-arm64-cpython-311/lxml
      copying src/lxml/ElementInclude.py -> build/lib.macosx-12.6-arm64-cpython-311/lxml
      creating build/lib.macosx-12.6-arm64-cpython-311/lxml/includes
      copying src/lxml/includes/__init__.py -> build/lib.macosx-12.6-arm64-cpython-311/lxml/includes
      creating build/lib.macosx-12.6-arm64-cpython-311/lxml/html
      copying src/lxml/html/soupparser.py -> build/lib.macosx-12.6-arm64-cpython-311/lxml/html
      copying src/lxml/html/defs.py -> build/lib.macosx-12.6-arm64-cpython-311/lxml/html
      copying src/lxml/html/_setmixin.py -> build/lib.macosx-12.6-arm64-cpython-311/lxml/html
      copying src/lxml/html/clean.py -> build/lib.macosx-12.6-arm64-cpython-311/lxml/html
      copying src/lxml/html/_diffcommand.py -> build/lib.macosx-12.6-arm64-cpython-311/lxml/html
      copying src/lxml/html/html5parser.py -> build/lib.macosx-12.6-arm64-cpython-311/lxml/html
      copying src/lxml/html/__init__.py -> build/lib.macosx-12.6-arm64-cpython-311/lxml/html
      copying src/lxml/html/formfill.py -> build/lib.macosx-12.6-arm64-cpython-311/lxml/html
      copying src/lxml/html/builder.py -> build/lib.macosx-12.6-arm64-cpython-311/lxml/html
      copying src/lxml/html/ElementSoup.py -> build/lib.macosx-12.6-arm64-cpython-311/lxml/html
      copying src/lxml/html/_html5builder.py -> build/lib.macosx-12.6-arm64-cpython-311/lxml/html
      copying src/lxml/html/usedoctest.py -> build/lib.macosx-12.6-arm64-cpython-311/lxml/html
      copying src/lxml/html/diff.py -> build/lib.macosx-12.6-arm64-cpython-311/lxml/html
      creating build/lib.macosx-12.6-arm64-cpython-311/lxml/isoschematron
      copying src/lxml/isoschematron/__init__.py -> build/lib.macosx-12.6-arm64-cpython-311/lxml/isoschematron
      copying src/lxml/etree.h -> build/lib.macosx-12.6-arm64-cpython-311/lxml
      copying src/lxml/etree_api.h -> build/lib.macosx-12.6-arm64-cpython-311/lxml
      copying src/lxml/lxml.etree.h -> build/lib.macosx-12.6-arm64-cpython-311/lxml
      copying src/lxml/lxml.etree_api.h -> build/lib.macosx-12.6-arm64-cpython-311/lxml
      copying src/lxml/includes/xmlerror.pxd -> build/lib.macosx-12.6-arm64-cpython-311/lxml/includes
      copying src/lxml/includes/c14n.pxd -> build/lib.macosx-12.6-arm64-cpython-311/lxml/includes
      copying src/lxml/includes/xmlschema.pxd -> build/lib.macosx-12.6-arm64-cpython-311/lxml/includes
      copying src/lxml/includes/__init__.pxd -> build/lib.macosx-12.6-arm64-cpython-311/lxml/includes
      copying src/lxml/includes/schematron.pxd -> build/lib.macosx-12.6-arm64-cpython-311/lxml/includes
      copying src/lxml/includes/tree.pxd -> build/lib.macosx-12.6-arm64-cpython-311/lxml/includes
      copying src/lxml/includes/uri.pxd -> build/lib.macosx-12.6-arm64-cpython-311/lxml/includes
      copying src/lxml/includes/etreepublic.pxd -> build/lib.macosx-12.6-arm64-cpython-311/lxml/includes
      copying src/lxml/includes/xpath.pxd -> build/lib.macosx-12.6-arm64-cpython-311/lxml/includes
      copying src/lxml/includes/htmlparser.pxd -> build/lib.macosx-12.6-arm64-cpython-311/lxml/includes
      copying src/lxml/includes/xslt.pxd -> build/lib.macosx-12.6-arm64-cpython-311/lxml/includes
      copying src/lxml/includes/config.pxd -> build/lib.macosx-12.6-arm64-cpython-311/lxml/includes
      copying src/lxml/includes/xmlparser.pxd -> build/lib.macosx-12.6-arm64-cpython-311/lxml/includes
      copying src/lxml/includes/xinclude.pxd -> build/lib.macosx-12.6-arm64-cpython-311/lxml/includes
      copying src/lxml/includes/dtdvalid.pxd -> build/lib.macosx-12.6-arm64-cpython-311/lxml/includes
      copying src/lxml/includes/relaxng.pxd -> build/lib.macosx-12.6-arm64-cpython-311/lxml/includes
      copying src/lxml/includes/lxml-version.h -> build/lib.macosx-12.6-arm64-cpython-311/lxml/includes
      copying src/lxml/includes/etree_defs.h -> build/lib.macosx-12.6-arm64-cpython-311/lxml/includes
      creating build/lib.macosx-12.6-arm64-cpython-311/lxml/isoschematron/resources
      creating build/lib.macosx-12.6-arm64-cpython-311/lxml/isoschematron/resources/rng
      copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.macosx-12.6-arm64-cpython-311/lxml/isoschematron/resources/rng
      creating build/lib.macosx-12.6-arm64-cpython-311/lxml/isoschematron/resources/xsl
      copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.macosx-12.6-arm64-cpython-311/lxml/isoschematron/resources/xsl
      copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.macosx-12.6-arm64-cpython-311/lxml/isoschematron/resources/xsl
      creating build/lib.macosx-12.6-arm64-cpython-311/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.macosx-12.6-arm64-cpython-311/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.macosx-12.6-arm64-cpython-311/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.macosx-12.6-arm64-cpython-311/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.macosx-12.6-arm64-cpython-311/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.macosx-12.6-arm64-cpython-311/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.macosx-12.6-arm64-cpython-311/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      running build_ext
      building 'lxml.etree' extension
      creating build/temp.macosx-12.6-arm64-cpython-311
      creating build/temp.macosx-12.6-arm64-cpython-311/src
      creating build/temp.macosx-12.6-arm64-cpython-311/src/lxml
      clang -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/opt/homebrew/opt/openjdk/include -I/opt/homebrew/opt/openjdk@11/include -DCYTHON_CLINE_IN_TRACEBACK=0 -Isrc -Isrc/lxml/includes -I/Users/csuh/.virtualenvs/pds-deep-archive/include -I/Users/csuh/.pyenv/versions/3.11.0/include/python3.11 -c src/lxml/etree.c -o build/temp.macosx-12.6-arm64-cpython-311/src/lxml/etree.o -w -flat_namespace
      src/lxml/etree.c:289:12: fatal error: 'longintrepr.h' file not found
        #include "longintrepr.h"
                 ^~~~~~~~~~~~~~~
      1 error generated.
      Compile failed: command '/usr/bin/clang' failed with exit code 1
      creating var
      creating var/folders
      creating var/folders/lz
      creating var/folders/lz/7mljvnvn7r93pn5tfh9stf8m0000gp
      creating var/folders/lz/7mljvnvn7r93pn5tfh9stf8m0000gp/T
      cc -I/usr/include/libxml2 -c /var/folders/lz/7mljvnvn7r93pn5tfh9stf8m0000gp/T/xmlXPathInitotkleprq.c -o var/folders/lz/7mljvnvn7r93pn5tfh9stf8m0000gp/T/xmlXPathInitotkleprq.o
      cc var/folders/lz/7mljvnvn7r93pn5tfh9stf8m0000gp/T/xmlXPathInitotkleprq.o -lxml2 -o a.out
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> lxml

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

🕵️ Expected behavior

I expected deep-archive to install successfully.

📜 To Reproduce

1. Assuming this is an installation and not an upgrade, create the virtual environment.
2. Verify the required versions for python, libxml2, and libxslt.
3. Install the application with pip install pds.deeparchive.
4. Read output.

🖥 Environment Info

• Operating System: MacOS Ventura 13.4.1, Chip
• python: 3.11.0
• pip: 22.3
• libxml2: 2.9.13
• libxslt: 1.1.35

📚 Version of Software Used

• pds.deeparchive: 1.1.3

🩺 Test Data / Additional context

No response

🦄 Related requirements

🦄 #xyz

⚙️ Engineering Details

No response

[jordanpadams]

@c-suh did you try creating a new venv from scratch to test this? Or was this an existing venv? Also, what version of python is running?

[jordanpadams]

@c-suh the app requires python 3.9 (or at least that is the only version it has been tested with)

[c-suh]

@jordanpadams I created a new virtual environment on my local machine with python 3.11.0 and attempted to install deep archive. I don't have a prior installation to upgrade, so this isn't replicating Tina's situation, pip version and other requirements notwithstanding. I've listed my other versions (e.g. pip, libxml2) in the main description under "Environment Info".

Ah, I mistakenly filled in "Python 3.9 or above". I will try again with 3.9.

[jordanpadams]

@nutjob4life can you look into this and see if it is a Python version issue and/or if there is anything we can do to fix it? ideally we can figure out some workaround so ATM can fix their installation per https://github.com/NASA-PDS/operations/issues/400#issuecomment-1622431545

[nutjob4life]

Hi @jordanpadams, this is a subtle one.

A change to the arrangement of header files introduced in Python 3.11 has broken several Python add-ons, including lxml which is used by the Deep Archive.

The quickest workaround is to downgrade Python from 3.11 → 3.10. Note also that the PDS Engineering Node currently has standardized on Python 3.9, which also works.

[jordanpadams]

Thanks @nutjob4life can we update the docs to include this nuance?

[c-suh]

@nutjob4life Tina just posted her output here from when she upgrades the deep archive. I will ask her to use Python 3.9, but is there anything else I should add? Update: never mind :smile: Jordan just responded in the other ticket.

[nutjob4life]

Thanks @nutjob4life can we update the docs to include this nuance?

We absolutely can.

[nutjob4life]

@nutjob4life Tina just posted her output here from when she upgrades the deep archive. I will ask her to use Python 3.9, but is there anything else I should add? Update: never mind 😄 Jordan just responded in the other ticket.

@jordanpadams is too fast 🏎️💨

[jordanpadams]

Closed per documentation update