search

Azure / azure-sdk-for-python

This repository is for active development of the Azure SDK for Python. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/python/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-python.
MIT License
4.64k stars 2.84k forks source link

Missing Module: azure.eventhub #3863

Closed raygervais closed 5 years ago

raygervais commented 6 years ago

Trying to setup a basic EventsHub POC following the documentation provided here: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-python-get-started-receive

Python Import: from azure.eventhub import EventHubClient, Receiver, Offset

Error: 

Traceback (most recent call last):
  File "Event.py", line 8, in <module>
    from azure.eventhub import EventHubClient, Receiver, Offset
ModuleNotFoundError: No module named 'azure.eventhub'

Latest Container Version(s)

azure-cli (2.0.51)

acr (2.1.9)
acs (2.3.11)
advisor (2.0.0)
ams (0.3.0)
appservice (0.2.7)
backup (1.2.1)
batch (3.4.1)
batchai (0.4.4)
billing (0.2.0)
botservice (0.1.1)
cdn (0.2.0)
cloud (2.1.0)
cognitiveservices (0.2.4)
command-modules-nspkg (2.0.2)
configure (2.0.19)
consumption (0.4.0)
container (0.3.9)
core (2.0.51)
cosmosdb (0.2.5)
dla (0.2.3)
dls (0.1.5)
dms (0.1.1)
eventgrid (0.2.0)
eventhubs (0.3.1)
extension (0.2.3)
feedback (2.1.4)
find (0.2.12)
hdinsight (0.1.0)
interactive (0.4.0)
iot (0.3.4)
iotcentral (0.1.4)
keyvault (2.2.7)
lab (0.1.3)
maps (0.3.2)
monitor (0.2.7)
network (2.2.9)
nspkg (3.0.3)
policyinsights (0.1.0)
profile (2.1.2)
rdbms (0.3.5)
redis (0.3.2)
relay (0.1.2)
reservations (0.4.1)
resource (2.1.6)
role (2.1.10)
search (0.1.1)
servicebus (0.3.2)
servicefabric (0.1.8)
signalr (1.0.0)
sql (2.1.5)
storage (2.2.5)
telemetry (1.0.0)
vm (2.2.8)

Python location '/usr/local/bin/python'
Extensions directory '/root/.azure/cliextensions'

Python (Linux) 3.6.4 (default, Jan 10 2018, 05:20:21)
annatisch commented 6 years ago

Hi @raygervais - it looks like you have the Azure CLI installed - however this does not currently include the package azure-eventhub that is used by the documentation you linked to.

Did you try running the installation command?

pip install azure-eventhub

If so - it may have installed to a different Python version then that used by the CLI that we're seeing in the output above.

raygervais commented 6 years ago

Hi @annatisch, attempting to install results in a few errors (in order along with their fix), and though I am happy to help fix them, I wonder if there a Docker image which already has this setup officially by Microsoft / Azure?

All Errors relate to microsoft/azure-cli:latest

Command "/usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-4xo0sdxd/uamqp/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-czuiv9p9-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-4xo0sdxd/uamqp/
You are using pip version 9.0.3, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Running the suggested command upgrades the version successfully, and produces this error message when trying pip install azure-eventhub again: 

 ----------------------------------------
  Failed building wheel for uamqp
  Running setup.py clean for uamqp
Failed to build uamqp
Installing collected packages: uamqp, azure-eventhub
  Running setup.py install for uamqp ... error
    Complete output from command /usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-g5cpyzbn/uamqp/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-fq4n4aom/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.6
    creating build/lib.linux-x86_64-3.6/uamqp
    copying uamqp/sender.py -> build/lib.linux-x86_64-3.6/uamqp
    copying uamqp/session.py -> build/lib.linux-x86_64-3.6/uamqp
    copying uamqp/compat.py -> build/lib.linux-x86_64-3.6/uamqp
    copying uamqp/utils.py -> build/lib.linux-x86_64-3.6/uamqp
    copying uamqp/message.py -> build/lib.linux-x86_64-3.6/uamqp
    copying uamqp/types.py -> build/lib.linux-x86_64-3.6/uamqp
    copying uamqp/address.py -> build/lib.linux-x86_64-3.6/uamqp
    copying uamqp/receiver.py -> build/lib.linux-x86_64-3.6/uamqp
    copying uamqp/client.py -> build/lib.linux-x86_64-3.6/uamqp
    copying uamqp/constants.py -> build/lib.linux-x86_64-3.6/uamqp
    copying uamqp/connection.py -> build/lib.linux-x86_64-3.6/uamqp
    copying uamqp/__init__.py -> build/lib.linux-x86_64-3.6/uamqp
    copying uamqp/mgmt_operation.py -> build/lib.linux-x86_64-3.6/uamqp
    copying uamqp/errors.py -> build/lib.linux-x86_64-3.6/uamqp
    creating build/lib.linux-x86_64-3.6/uamqp/authentication
    copying uamqp/authentication/common.py -> build/lib.linux-x86_64-3.6/uamqp/authentication
    copying uamqp/authentication/cbs_auth_async.py -> build/lib.linux-x86_64-3.6/uamqp/authentication
    copying uamqp/authentication/cbs_auth.py -> build/lib.linux-x86_64-3.6/uamqp/authentication
    copying uamqp/authentication/__init__.py -> build/lib.linux-x86_64-3.6/uamqp/authentication
    creating build/lib.linux-x86_64-3.6/uamqp/async_ops
    copying uamqp/async_ops/connection_async.py -> build/lib.linux-x86_64-3.6/uamqp/async_ops
    copying uamqp/async_ops/receiver_async.py -> build/lib.linux-x86_64-3.6/uamqp/async_ops
    copying uamqp/async_ops/client_async.py -> build/lib.linux-x86_64-3.6/uamqp/async_ops
    copying uamqp/async_ops/sender_async.py -> build/lib.linux-x86_64-3.6/uamqp/async_ops
    copying uamqp/async_ops/mgmt_operation_async.py -> build/lib.linux-x86_64-3.6/uamqp/async_ops
    copying uamqp/async_ops/__init__.py -> build/lib.linux-x86_64-3.6/uamqp/async_ops
    copying uamqp/async_ops/session_async.py -> build/lib.linux-x86_64-3.6/uamqp/async_ops
    running egg_info
    writing uamqp.egg-info/PKG-INFO
    writing dependency_links to uamqp.egg-info/dependency_links.txt
    writing requirements to uamqp.egg-info/requires.txt
    writing top-level names to uamqp.egg-info/top_level.txt
    reading manifest file 'uamqp.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'uamqp.egg-info/SOURCES.txt'
    copying uamqp/c_uamqp.c -> build/lib.linux-x86_64-3.6/uamqp
    running build_ext
    will build uamqp in build/temp.linux-x86_64-3.6/cmake
    Building with generator: Unix Makefiles
    calling cmake /tmp/pip-install-g5cpyzbn/uamqp/src/vendor/azure-uamqp-c/ -G "Unix Makefiles" -Duse_openssl:bool=ON -Duse_default_uuid:bool=ON  -Duse_builtin_httpapi:bool=ON  -Dskip_samples:bool=ON -DCMAKE_POSITION_INDEPENDENT_CODE=TRUE -DCMAKE_BUILD_TYPE=Release
    /bin/sh: cmake: not found
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-g5cpyzbn/uamqp/setup.py", line 328, in <module>
        python_requires=">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*",
      File "/usr/local/lib/python3.6/site-packages/setuptools/__init__.py", line 129, in setup
        return distutils.core.setup(**attrs)
      File "/usr/local/lib/python3.6/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/local/lib/python3.6/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/usr/local/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python3.6/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/usr/local/lib/python3.6/distutils/command/install.py", line 545, in run
        self.run_command('build')
      File "/usr/local/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/local/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python3.6/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/usr/local/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/local/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/tmp/pip-install-g5cpyzbn/uamqp/setup.py", line 167, in run
        self.build_cmake(ext)
      File "/tmp/pip-install-g5cpyzbn/uamqp/setup.py", line 215, in build_cmake
        subprocess.check_call(joined_cmd, shell=True, universal_newlines=True, env=build_env)
      File "/usr/local/lib/python3.6/subprocess.py", line 291, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command 'cmake /tmp/pip-install-g5cpyzbn/uamqp/src/vendor/azure-uamqp-c/ -G "Unix Makefiles" -Duse_openssl:bool=ON -Duse_default_uuid:bool=ON  -Duse_builtin_httpapi:bool=ON  -Dskip_samples:bool=ON -DCMAKE_POSITION_INDEPENDENT_CODE=TRUE -DCMAKE_BUILD_TYPE=Release' returned non-zero exit status 127.

    ----------------------------------------
Command "/usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-g5cpyzbn/uamqp/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-fq4n4aom/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-g5cpyzbn/uamqp/

Thoughts? Again, happy to get this resolved and working for those who want to use in AzureCLI as well, but really need a working Docker image to use during that time.

To your last comment, is there a specific directory for the AzureCLI / Python SDK version that I should be executing against?

annatisch commented 6 years ago

Thanks @raygervais! We do not currently have a Docker container with azure-eventhub set up. But this is an interesting idea that I will look into - as it could also be useful for our test infrastructure....

Though in the meantime, it's interesting that it looks like it's attempting to build uAMQP from source. Could you try updating pip? Maybe the older version of pip is not picking up the uAMQP wheel. The alternative is that your Linux distro does not like the published manylinux wheel..... which Linux distro are you currently running?

python -m pip install --upgrade pip
raygervais commented 6 years ago

Update for you @annatisch, updated pip using your supplied command above, but it appears to have the same error. The official microsoft/azure-cli Docker image is using the following Linux distro: Alpine Linux 3.7.0.

Dockerfile: https://github.com/Azure/azure-cli/blob/master/Dockerfile

I know Alpine has a common issue with building tooling due to it's minimalistic nature, so I'd be curious to try on an Ubuntu 18.04 LTS Dockerfile (based off of what is already done in the Dockerfile above). Thoughts?

Would love to hear about your test infrastructure too at one point, always open to learning and help to improve my own work alongside the communities.

Update: Also tried on azuresdk/azure-cli-python:latest, same issue. What is the difference between microsoft/azure-cli and the above?

raygervais commented 6 years ago

Wanted to provide an update/working solution using Debian Stretch, but there is still the ask of Alpine.

Dockerfile


FROM python:slim-stretch

# Install Azure Event Hub PythonSDK Module
RUN pip install azure-eventhub

# Remove Unneeded PIP Cache after Install
RUN rm -rf /.cache

Installs as expected, the average image size is 185MB, hence my wanting to get the Alpine variant working.

annatisch commented 6 years ago

Hi @raygervais - yes you're quite right - Alpine is not supported by manylinux, so we do not currently have a wheel available for uAMQP for that distro.

You could try out the instructions here to see whether it's possible to build uAMQP from source for Alpine: https://github.com/Azure/azure-uamqp-python/blob/master/README.rst#installation

Otherwise I have opened an issue here and I will try to take a look at getting wheel support: https://github.com/Azure/azure-uamqp-python/issues/51

raygervais commented 6 years ago

Will try @annatisch the suggestion instructions, thanks for opening the issue in the uampq-python repository! Looking forward to getting this resolved and enabling other developers both Microsoft and alike to utilize the Azure-EventHub Python module in Alpine Docker containers.

powo commented 5 years ago

apk add g++ seemed to at least fix a cmake error in Alpine 3.9 for a build from source

annatisch commented 5 years ago

Closing due to inactivity - alpine support issue will remain open in uamqp.

nsquaretologn commented 2 years ago

Adding these four lines on docker file solved my issue

Find arm64 variant of deb buster


RUN echo "deb http://deb.debian.org/debian buster main" > /etc/apt/sources.list \
    && echo "deb http://deb.debian.org/debian  buster/updates main" >> /etc/apt/sources.list \
    && echo "deb http://deb.debian.org/debian buster-updates main" >> /etc/apt/sources.list \
    && apt-get --quiet update --yes && apt-get --quiet upgrade --yes \
    && apt-get install -y build-essential libssl-dev uuid-dev cmake libcurl4-openssl-dev pkg-config