search

Azure / azure-uamqp-python

AMQP 1.0 client library for Python
MIT License
57 stars 48 forks source link

Python 3.11 issues for uaqmp #334

Closed kashifkhan closed 1 year ago

EpicWink commented 2 years ago

Seems to fail to build due to missing longintrepr.h during build of 'build/temp.linux-x86_64-cpython-311/uamqp/c_uamqp.o' (from 'uamqp/c_uamqp.c')

Reproduction:

docker run -it python:3.11-rc-slim bash
apt update
apt install --no-install-recommends g++ libc-dev cmake make libssl-dev uuid-dev
pip install uamqp

Quick google search gives possibly relevant:

Solution seems to be:

- #include "longintrepr.h"
+ #if PY_MAJOR_VERSION < 3
+   #include "longintrepr.h"
+ #endif

(or drop Python 2 support entirely, and just remove the include)

frenck commented 2 years ago

Build output:

  Building wheel for uamqp (setup.py): started
  Building wheel for uamqp (setup.py): finished with status 'error'
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [209 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-311
      creating build/lib.linux-x86_64-cpython-311/uamqp
      copying uamqp/receiver.py -> build/lib.linux-x86_64-cpython-311/uamqp
      copying uamqp/types.py -> build/lib.linux-x86_64-cpython-311/uamqp
      copying uamqp/compat.py -> build/lib.linux-x86_64-cpython-311/uamqp
  Running setup.py clean for uamqp
      copying uamqp/address.py -> build/lib.linux-x86_64-cpython-311/uamqp
      copying uamqp/message.py -> build/lib.linux-x86_64-cpython-311/uamqp
      copying uamqp/session.py -> build/lib.linux-x86_64-cpython-311/uamqp
      copying uamqp/errors.py -> build/lib.linux-x86_64-cpython-311/uamqp
      copying uamqp/client.py -> build/lib.linux-x86_64-cpython-311/uamqp
      copying uamqp/constants.py -> build/lib.linux-x86_64-cpython-311/uamqp
      copying uamqp/connection.py -> build/lib.linux-x86_64-cpython-311/uamqp
      copying uamqp/utils.py -> build/lib.linux-x86_64-cpython-311/uamqp
      copying uamqp/__init__.py -> build/lib.linux-x86_64-cpython-311/uamqp
      copying uamqp/sender.py -> build/lib.linux-x86_64-cpython-311/uamqp
      copying uamqp/mgmt_operation.py -> build/lib.linux-x86_64-cpython-311/uamqp
      creating build/lib.linux-x86_64-cpython-311/uamqp/async_ops
      copying uamqp/async_ops/connection_async.py -> build/lib.linux-x86_64-cpython-311/uamqp/async_ops
      copying uamqp/async_ops/receiver_async.py -> build/lib.linux-x86_64-cpython-311/uamqp/async_ops
      copying uamqp/async_ops/mgmt_operation_async.py -> build/lib.linux-x86_64-cpython-311/uamqp/async_ops
      copying uamqp/async_ops/sender_async.py -> build/lib.linux-x86_64-cpython-311/uamqp/async_ops
      copying uamqp/async_ops/session_async.py -> build/lib.linux-x86_64-cpython-311/uamqp/async_ops
      copying uamqp/async_ops/utils.py -> build/lib.linux-x86_64-cpython-311/uamqp/async_ops
      copying uamqp/async_ops/__init__.py -> build/lib.linux-x86_64-cpython-311/uamqp/async_ops
      copying uamqp/async_ops/client_async.py -> build/lib.linux-x86_64-cpython-311/uamqp/async_ops
      creating build/lib.linux-x86_64-cpython-311/uamqp/authentication
      copying uamqp/authentication/cbs_auth.py -> build/lib.linux-x86_64-cpython-311/uamqp/authentication
      copying uamqp/authentication/common.py -> build/lib.linux-x86_64-cpython-311/uamqp/authentication
      copying uamqp/authentication/cbs_auth_async.py -> build/lib.linux-x86_64-cpython-311/uamqp/authentication
      copying uamqp/authentication/__init__.py -> build/lib.linux-x86_64-cpython-311/uamqp/authentication
      running egg_info
      writing uamqp.egg-info/PKG-INFO
      writing dependency_links to uamqp.egg-info/dependency_links.txt
      writing requirements to uamqp.egg-info/requires.txt
      writing top-level names to uamqp.egg-info/top_level.txt
      reading manifest file 'uamqp.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      writing manifest file 'uamqp.egg-info/SOURCES.txt'
      copying uamqp/c_uamqp.c -> build/lib.linux-x86_64-cpython-311/uamqp
      running build_ext
      will build uamqp in build/temp.linux-x86_64-cpython-311/cmake
      Building with generator flags: -G "Unix Makefiles"
      calling cmake /tmp/pip-install-lqo3w3o_/uamqp/src/vendor/azure-uamqp-c/ -G "Unix Makefiles" -Duse_openssl:bool=ON -Duse_default_uuid:bool=ON  -Duse_builtin_httpapi:bool=ON  -Dskip_samples:bool=ON -DCMAKE_POSITION_INDEPENDENT_CODE=TRUE -DCMAKE_BUILD_TYPE=Release
      -- The C compiler identification is GNU 9.4.0
      -- The CXX compiler identification is GNU 9.4.0
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      CMake Deprecation Warning at deps/azure-macro-utils-c/CMakeLists.txt:4 (cmake_minimum_required):
        Compatibility with CMake < 2.8.12 will be removed from a future version of
        CMake.

        Update the VERSION argument <min> value or use a ...<max> suffix to tell
        CMake that the project does not need compatibility with older versions.

      CMake Warning (dev) at deps/azure-macro-utils-c/CMakeLists.txt:14 (option):
        Policy CMP0077 is not set: option() honors normal variables.  Run "cmake
        --help-policy CMP0077" for policy details.  Use the cmake_policy command to
        set the policy and suppress this warning.

        For compatibility with older versions of CMake, option is clearing the
        normal variable 'run_int_tests'.
      This warning is for project developers.  Use -Wno-dev to suppress it.

      CMake Deprecation Warning at deps/umock-c/CMakeLists.txt:4 (cmake_minimum_required):
        Compatibility with CMake < 2.8.12 will be removed from a future version of
        CMake.

        Update the VERSION argument <min> value or use a ...<max> suffix to tell
        CMake that the project does not need compatibility with older versions.

      -- Looking for include file stdint.h
      -- Looking for include file stdint.h - found
      -- Looking for include file stdbool.h
      -- Looking for include file stdbool.h - found
      -- target architecture: x86_64
      -- Performing Test CXX_FLAG_CXX11
      -- Performing Test CXX_FLAG_CXX11 - Success
      -- Found OpenSSL: /usr/lib/x86_64-linux-gnu/libcrypto.so (found version "1.1.1f")
      -- target architecture: x86_64
      -- Configuring done
      -- Generating done
      -- Build files have been written to: /tmp/pip-install-lqo3w3o_/uamqp/build/temp.linux-x86_64-cpython-311/cmake
      calling cmake --build . --config Release
      [  0%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/azure_base32.c.o
      [  1%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/azure_base64.c.o
      [  2%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/buffer.c.o
      [  3%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/constbuffer_array.c.o
      [  4%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/constbuffer_array_batcher.c.o
      [  5%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/connection_string_parser.c.o
      [  6%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/constbuffer.c.o
      [  7%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/consolelogger.c.o
      [  8%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/crt_abstractions.c.o
      [  9%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/constmap.c.o
      [ 10%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/doublylinkedlist.c.o
      [ 11%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/gballoc.c.o
      [ 12%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/gbnetwork.c.o
      [ 13%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/gb_stdio.c.o
      [ 14%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/gb_time.c.o
      [ 15%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/gb_rand.c.o
      [ 16%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/hmac.c.o
      [ 17%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/hmacsha256.c.o
      [ 18%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/xio.c.o
      [ 19%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/singlylinkedlist.c.o
      [ 20%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/map.c.o
      [ 21%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/sastoken.c.o
      [ 22%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/sha1.c.o
      [ 23%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/sha224.c.o
      [ 24%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/sha384-512.c.o
      [ 25%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/strings.c.o
      [ 26%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/string_token.c.o
      [ 27%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/string_tokenizer.c.o
      [ 28%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/uuid.c.o
      [ 29%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/urlencode.c.o
      [ 30%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/usha.c.o
      [ 31%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/vector.c.o
      [ 32%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/xlogging.c.o
      [ 33%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/optionhandler.c.o
      [ 34%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/memory_data.c.o
      [ 35%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/adapters/agenttime.c.o
      [ 36%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/adapters/condition_pthreads.c.o
      [ 37%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/adapters/lock_pthreads.c.o
      [ 38%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/adapters/platform_linux.c.o
      [ 39%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/adapters/socketio_berkeley.c.o
      [ 40%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/adapters/tickcounter_linux.c.o
      [ 41%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/adapters/threadapi_pthreads.c.o
      [ 42%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/adapters/uniqueid_stub.c.o
      [ 43%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/adapters/envvariable.c.o
      [ 44%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/dns_resolver_sync.c.o
      [ 45%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/adapters/linux_time.c.o
      [ 46%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/httpapiex.c.o
      [ 47%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/httpapiexsas.c.o
      [ 48%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/httpheaders.c.o
      [ 49%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/adapters/httpapi_compact.c.o
      [ 50%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/http_proxy_io.c.o
      [ 51%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/adapters/tlsio_openssl.c.o
      [ 52%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/adapters/x509_openssl.c.o
      [ 53%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/wsio.c.o
      [ 54%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/uws_client.c.o
      [ 55%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/uws_frame_encoder.c.o
      [ 56%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/utf8_checker.c.o
      [ 57%] Building C object deps/azure-c-shared-utility/CMakeFiles/aziotsharedutil.dir/src/ws_url.c.o
      [ 58%] Linking C static library libaziotsharedutil.a
      [ 58%] Built target aziotsharedutil
      [ 59%] Building C object CMakeFiles/uamqp.dir/src/amqp_definitions.c.o
      [ 60%] Building C object CMakeFiles/uamqp.dir/src/amqp_frame_codec.c.o
      [ 61%] Building C object CMakeFiles/uamqp.dir/src/amqp_management.c.o
      [ 62%] Building C object CMakeFiles/uamqp.dir/src/amqpvalue.c.o
      [ 63%] Building C object CMakeFiles/uamqp.dir/src/amqpvalue_to_string.c.o
      [ 64%] Building C object CMakeFiles/uamqp.dir/src/async_operation.c.o
      [ 65%] Building C object CMakeFiles/uamqp.dir/src/cbs.c.o
      [ 66%] Building C object CMakeFiles/uamqp.dir/src/connection.c.o
      [ 67%] Building C object CMakeFiles/uamqp.dir/src/frame_codec.c.o
      [ 68%] Building C object CMakeFiles/uamqp.dir/src/header_detect_io.c.o
      [ 69%] Building C object CMakeFiles/uamqp.dir/src/link.c.o
      [ 70%] Building C object CMakeFiles/uamqp.dir/src/message.c.o
      [ 71%] Building C object CMakeFiles/uamqp.dir/src/message_receiver.c.o
      [ 72%] Building C object CMakeFiles/uamqp.dir/src/message_sender.c.o
      [ 73%] Building C object CMakeFiles/uamqp.dir/src/messaging.c.o
      [ 74%] Building C object CMakeFiles/uamqp.dir/src/sasl_anonymous.c.o
      [ 75%] Building C object CMakeFiles/uamqp.dir/src/sasl_frame_codec.c.o
      [ 76%] Building C object CMakeFiles/uamqp.dir/src/sasl_mechanism.c.o
      [ 77%] Building C object CMakeFiles/uamqp.dir/src/sasl_server_mechanism.c.o
      [ 78%] Building C object CMakeFiles/uamqp.dir/src/sasl_mssbcbs.c.o
      [ 79%] Building C object CMakeFiles/uamqp.dir/src/sasl_plain.c.o
      [ 80%] Building C object CMakeFiles/uamqp.dir/src/saslclientio.c.o
      [ 81%] Building C object CMakeFiles/uamqp.dir/src/session.c.o
      [ 82%] Building C object CMakeFiles/uamqp.dir/src/socket_listener_berkeley.c.o
      [ 83%] Linking C static library libuamqp.a
      [ 83%] Built target uamqp
      [ 84%] Building C object deps/umock-c/CMakeFiles/umock_c.dir/src/umock_c.c.o
      [ 85%] Building C object deps/umock-c/CMakeFiles/umock_c.dir/src/umock_c_negative_tests.c.o
      [ 86%] Building C object deps/umock-c/CMakeFiles/umock_c.dir/src/umockalloc.c.o
      [ 87%] Building C object deps/umock-c/CMakeFiles/umock_c.dir/src/umockcall.c.o
      [ 88%] Building C object deps/umock-c/CMakeFiles/umock_c.dir/src/umockcallrecorder.c.o
      [ 89%] Building C object deps/umock-c/CMakeFiles/umock_c.dir/src/umocktypename.c.o
      [ 90%] Building C object deps/umock-c/CMakeFiles/umock_c.dir/src/umocktypes.c.o
      [ 91%] Building C object deps/umock-c/CMakeFiles/umock_c.dir/src/umocktypes_bool.c.o
      [ 92%] Building C object deps/umock-c/CMakeFiles/umock_c.dir/src/umocktypes_c.c.o
      [ 93%] Building C object deps/umock-c/CMakeFiles/umock_c.dir/src/umocktypes_stdint.c.o
      [ 94%] Building C object deps/umock-c/CMakeFiles/umock_c.dir/src/umocktypes_charptr.c.o
      [ 95%] Building C object deps/umock-c/CMakeFiles/umock_c.dir/src/umocktypes_wcharptr.c.o
      [ 96%] Building C object deps/umock-c/CMakeFiles/umock_c.dir/src/umockcallpairs.c.o
      [ 97%] Building C object deps/umock-c/CMakeFiles/umock_c.dir/src/umockstring.c.o
      [ 98%] Building C object deps/umock-c/CMakeFiles/umock_c.dir/src/umockautoignoreargs.c.o
      [ 99%] Building C object deps/umock-c/CMakeFiles/umock_c.dir/src/umock_log.c.o
      [100%] Linking C static library libumock_c.a
      [100%] Built target umock_c
      building 'uamqp.c_uamqp' extension
      creating build/temp.linux-x86_64-cpython-311/uamqp
      gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I./src/vendor/inc -I./src/vendor/azure-uamqp-c/deps/azure-macro-utils-c/inc -I./src/vendor/azure-uamqp-c/deps/umock-c/inc -I./src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/pal/inc -I./src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/inc -I./src/vendor/azure-uamqp-c/inc -I./src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/pal/linux -I/home/runner/work/core/core/venv/include -I/opt/hostedtoolcache/Python/3.11.0-rc.2/x64/include/python3.11 -c uamqp/c_uamqp.c -o build/temp.linux-x86_64-cpython-311/uamqp/c_uamqp.o -g -O0 -std=gnu99 -fPIC
      uamqp/c_uamqp.c:268:12: fatal error: longintrepr.h: No such file or directory
        268 |   #include "longintrepr.h"
            |            ^~~~~~~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
kashifkhan commented 2 years ago

Seems to fail to build due to missing longintrepr.h during build of 'build/temp.linux-x86_64-cpython-311/uamqp/c_uamqp.o' (from 'uamqp/c_uamqp.c')

Reproduction:

docker run -it python:3.11-rc-slim bash
apt update
apt install --no-install-recommends g++ libc-dev cmake make libssl-dev uuid-dev
pip install uamqp

Quick google search gives possibly relevant:

Solution seems to be:

- #include "longintrepr.h"
+ #if PY_MAJOR_VERSION < 3
+   #include "longintrepr.h"
+ #endif

(or drop Python 2 support entirely, and just remove the include)

Thanks for this @EpicWink and @frenck ...Looks like the cython requirement needs to be bumped up to fix this.

kashifkhan commented 1 year ago

Wanted to give yall an update, I'm working on generating wheels for 3.11 in this feature branch. Bumping up the cython was indeed the fix, the version we had pinned to had a bug which was fixed subsequently. I am able to build wheels so a simple workaround is to just bump up the version to 0.29.32 to get yall moving :)

potiuk commented 1 year ago

Cool if there is a help you need on that - we are looking forward to it in Apache Airflow too

I know it is eaarly (Python 3.11 has just been released yesterday) but we are hoping in Apache Airflow to a much faster cycle of adding new Python releases - especially that Pyhon 3.11 introduces huge performance improvements (25% is the average number claimed) due to a very focused effort to increase single-threaded Python performance (Specialized interpreter being the core of it but also many other improvements) without actually changing any of the Python code.

The azure-uamqp-python is one of the dependencies of Airlfow that need to support newer Python version so this issue is here to make you aware that we are eaager (also to help if needed) to make it faster, possibly by talking to dependencies of Beam as well (who are oftten also direct dependencies of Airflow) and help them as well. I perfectly understand we need to bubble up support so that your dependencies support them first.

I opened PR in Apache Airflow yesterday and plan to keep it open until it gets green :)

https://github.com/apache/airflow/pull/27264

I think it would be fantastic if we could as the open source community migrate to the new Python much faster.

Looking forward to cooperation on that one :)

kashifkhan commented 1 year ago

Hi @potiuk, Im the in the process of running a few tests on uamqp and making sure everything is working the way we intend it to be, so that everyone has a smooth experience whilst using 3.11 :) We are also quite excited to see the effect of 3.11 on our code base.

As you know macOS runners on Github added 3.11 to their images just a few hours ago, so I need to go through those and some internal checks before pushing out new wheels.

appreciate the offer for help, Ill update this thread if I end up needing help etc.

kashifkhan commented 1 year ago

hello everyone, we just published uamqp ver 1.6.3 that has support for py311

potiuk commented 1 year ago

WOOOHO!!!