cchardet alternatives for python 3.11+

[dcslagel]

Describe the bug For python 3.11 cChardet fails to build. The error message is: cchardet.cpp(196): fatal error C1083: Cannot open include file: 'longintrepr.h': No such file or directory.

In addition, the cchardet doesn't seem to be currently maintained.

This is a problem that will need to be resolved going forward. faust-streaming/cChardet is a possible replacement. It is a fork of cChardet.
https://pypi.org/project/faust-cchardet/ https://github.com/faust-streaming/cChardet

Notes Does not Support Anaconda pyenv

Does not yet install on MacOS Cannot install on arm64 (Apple M1 Sillicon) #12

To Reproduce Steps to reproduce the behavior:

• Add "3.11" to ci-test.yml on a dev branch and push to Github
• GitHub->Actions->Python CI should fail for 3.11 image when trying to build cChardet.

Expected behavior Python 3.11 builds/test on GitHub->Actions->Python CI should should pass

Software versions (please complete the following information):

• Python version: [3.11]
• Python distribution: [GitHub]
• lasio version: 0.30

Additional context

• I think chardet is a slower fall back option. So there is a possibility that Lasio code/packaging can be arranged to use chardet on Python 3.11.

[dcslagel]

Additional Data

Tried install Lasio requirements in a Python 3.11 venv. Below is the full cChardet install error.

The critical failure is:

src/cchardet/_cchardet.cpp:196:12: fatal error: 'longintrepr.h' file not found
        #include "longintrepr.h"

Full Output:

 DEPRECATION: cchardet is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for cchardet ... error
  error: subprocess-exited-with-error

  × Running setup.py install for cchardet did not run successfully.
  │ exit code: 1
  ╰─> [25 lines of output]
      running install
      /Users/dcs/Wrk/Projects/geo-wrk/lasio/venvs/venv-3.11-lasio/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.macosx-13-arm64-cpython-311
      creating build/lib.macosx-13-arm64-cpython-311/cchardet
      copying src/cchardet/version.py -> build/lib.macosx-13-arm64-cpython-311/cchardet
      copying src/cchardet/__init__.py -> build/lib.macosx-13-arm64-cpython-311/cchardet
      running build_ext
      building 'cchardet._cchardet' extension
      creating build/temp.macosx-13-arm64-cpython-311
      creating build/temp.macosx-13-arm64-cpython-311/src
      creating build/temp.macosx-13-arm64-cpython-311/src/cchardet
      creating build/temp.macosx-13-arm64-cpython-311/src/ext
      creating build/temp.macosx-13-arm64-cpython-311/src/ext/uchardet
      creating build/temp.macosx-13-arm64-cpython-311/src/ext/uchardet/src
      creating build/temp.macosx-13-arm64-cpython-311/src/ext/uchardet/src/LangModels
      clang -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk -Isrc/ext/uchardet/src -I/Users/dcs/Wrk/Projects/geo-wrk/lasio/venvs/venv-3.11-lasio/include -I/opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c src/cchardet/_cchardet.cpp -o build/temp.macosx-13-arm64-cpython-311/src/cchardet/_cchardet.o
      src/cchardet/_cchardet.cpp:196:12: fatal error: 'longintrepr.h' file not found
        #include "longintrepr.h"
                 ^~~~~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> cchardet

[kinverarity1]

Maybe we can make it simpler and just use chardet if it is installed, and if not, give an error?

[dcslagel]

Do you mean, "use chardet if it is installed" or "use cchardet if it is installed"?

[dcslagel]

Notes: The pyproject.toml syntax below may work. For now (lasio main), it needs to be in setup.py format. So look that up!

[project.optional-dependencies]
cchardet = [
  'cchardet; python_version < "3.11"',
]

References: https://peps.python.org/pep-0631/ https://peps.python.org/pep-0508/

[kinverarity1]

I meant the former, since the cchardet fork isn't fully cross-platform, so trying to use it or recommend it for python 3.11 would probably create more problems than it helps with? Not sure what to do!

[dcslagel]

I'll make a branch replacing cchardet with chardet as an optional dependency. chardet is still actively maintained which is a good sign.