Running with python, help needed

[DanBmh]

I could install this project with python bindings, and run the example pub/sub scripts, but pub and sub are not exchanging messages.

The output of pub:

/dps-for-iot/py_scripts# PYTHONPATH=../build/dist/py python simple_pub.py 
Publisher is listening on [::]:57575
Pub UUID f4d1c799-8bcb-2574-6155-f11c9e9f8074(1)
Pub UUID f4d1c799-8bcb-2574-6155-f11c9e9f8074(2)
Traceback (most recent call last):
  File "simple_pub.py", line 185, in <module>
    dps.destroy_publication(pub)
TypeError: destroy_publication expected 2 arguments, got 1

The output of sub:

/dps-for-iot/py_scripts# PYTHONPATH=../build/dist/py python simple_sub.py 
Subscriber is listening on [::]:48070

I did start the subscriber before the publisher and I think i'm missing some network options. What do I need to do to get this work?

And another question, I would like to use it with python3 instead of python2, but in this case i get following error:

Traceback (most recent call last):
  File "simple_sub.py", line 2, in <module>
    import dps
  File "/dps-for-iot/build/dist/py/dps.py", line 24, in <module>
    _dps = swig_import_helper()
  File "/dps-for-iot/build/dist/py/dps.py", line 23, in swig_import_helper
    return importlib.import_module('_dps')
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: /dps-for-iot/build/dist/py/_dps.so: undefined symbol: PyInstance_Type

Is python3 supported or do you plan to support it?

[malsbat]

I've repro'd your first issue, I'll have a fix out shortly. The test automation should've caught this so there's at least two problems here.

python3 is supported, and I can't repro the problem you are seeing. Are you running on Windows or Linux?

[DanBmh]

I'm running it on linux in a container. My containerfile:

FROM ubuntu:18.04

ARG DEBIAN_FRONTEND=noninteractive
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8

RUN apt-get update && apt-get upgrade -y
RUN apt-get install -y --no-install-recommends nano wget curl
RUN apt-get install -y --no-install-recommends git
RUN apt-get install -y --no-install-recommends python-pip python-dev
RUN apt-get install -y --no-install-recommends python3-pip python3-dev

RUN apt-get install -y --no-install-recommends gcc build-essential
RUN apt-get install -y --no-install-recommends scons
RUN apt-get install -y --no-install-recommends libuv-dev
RUN apt-get install -y --no-install-recommends swig

RUN pip install --no-cache-dir --upgrade cryptography
RUN pip3 install --no-cache-dir --upgrade cryptography

RUN git clone https://github.com/intel/dps-for-iot
RUN git config --global user.email "jaco@mail.de" && git config --global user.name "Jaco"
RUN cd dps-for-iot && scons bindings=python

CMD ["/bin/bash"]

[malsbat]

I pushed up a fix for the TypeError: destroy_publication expected 2 arguments, got 1 error.

I see that you're installing both python2 and python3, so I suspect that https://github.com/intel/dps-for-iot/blob/7d7ae100d3708e46ea6e45dbe7c3d531289548d9/SConstruct#L124 points to the python2 headers, and you're trying to run against python3.

[DanBmh]

I did add the python2 installation because without it I get following error:

Language subdirectory: python
[...]
Generating wrappers...
g++ -o build/obj/swig/py/dps_wrap.os -c -march=native -fPIC -ggdb -O3 -DNDEBUG -fstack-protector-strong -fPIE -Werror -Wall -Wextra -Wformat-security -Wno-unused-parameter -Wno-unused-function -Wno-type-limits -Wno-missing-braces -Wno-attributes -Wno-ignored-qualifiers -Wno-cast-function-type -fPIC -DDPS_USE_UDP -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -DMBEDTLS_USER_CONFIG_FILE=\"mbedtls_config.h\" -D_GNU_SOURCE -Iinc -Iext/safestring/include -Iext -Iext/mbedtls/include -Iext/libuv/include -Ibuild/obj/swig -Iswig -Ibuild/obj/swig/py -Iswig/py build/obj/swig/py/dps_wrap.cc
build/obj/swig/py/dps_wrap.cc:174:11: fatal error: Python.h: No such file or directory
 # include <Python.h>
           ^~~~~~~~~~

[malsbat]

It looks like either the SConstruct line referenced above didn't get executed or python-config did not return the expected value.

For example, here is the same build line on my machine

Generating wrappers...
g++ -o build/obj/swig/py/dps_wrap.os -c -march=native -fPIC -ggdb -O3 -DNDEBUG -fstack-protector-strong -fPIE -Werror -Wall -Wextra -Wformat-security -Wno-unused-parameter -Wno-unused-function -Wno-type-limits -Wno-missing-braces -Wno-attributes -Wno-ignored-qualifiers -Wno-cast-function-type -fPIC -DDPS_USE_UDP -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -DMBEDTLS_USER_CONFIG_FILE=\"mbedtls_config.h\" -D_GNU_SOURCE -Iinc -Iext/safestring/include -Iext -Iext/mbedtls/include -I/usr/include/python3.7m -I/usr/include/python3.7m -Ibuild/obj/swig -Iswig -Ibuild/obj/swig/py -Iswig/py build/obj/swig/py/dps_wrap.cc

The key part here is -I/usr/include/python3.7m, which is missing from your run.

Here is what the SConstruct is expecting to see

$ python-config --includes
-I/usr/include/python3.7m -I/usr/include/python3.7m

$ ls /usr/include/python3.7m/Python.h
/usr/include/python3.7m/Python.h

[DanBmh]

The problem is that I cant run python-config --includes

python-config --includes
bash: python-config: command not found

python3 -m python-config --includes
/usr/bin/python3: No module named python-config

I did install a package with the same name like this:

pip3 install setuptools
pip3 install python-config

Above installation threw an error in python 3.6 so i switched to python 3.8 which comes with ubuntu:20.04, you need to change libuv-dev to libuv1-dev with the new image.

[malsbat]

Did that resolve your issues? If so then it sounds like the root issue is that the build system isn't properly handling Ubuntu 18.04 and 20.04.

[DanBmh]

No, it only solved an issue with the pip installation of python-config. But i still can not run python-config --includes because python-config is not found.

[malsbat]

I'm trying to reproduce your issue with Ubuntu 18.04 using the automated builds, I'll post another update once I make some progress.

[malsbat]

Ok, it looks like Ubuntu makes some interesting choices around python-config. I'd recommend using pyenv to install and select the version you want to use.

It also looks like Ubuntu 20.04 has python-dev-is-python3 which will symlink python-config to python3-config, but I have not verified that solution.