Trouble installing Python wrapper

[joe-marrone]

Hello again, I have another issue. The backend installed OK after some effort. When installing the Python wrapper everything appeared to install OK. But when I tried to run the takeoff_and_landing script it threw this error:

joe@joe-VirtualBox:~/src/Firmware/DronecodeSDK-Python$ examples/takeoff_and_land.py
Traceback (most recent call last):
File "examples/takeoff_and_land.py", line 5, in 
from dronecode_sdk import connect as dronecode_sdk_connect
ModuleNotFoundError: No module named 'dronecode_sdk'

What next? Joe

PS. I asked this same question in another thread that was closed, so it's probably best to create a new thread.

[joe-marrone]

It looks like the apt_pkg issue has been solved. Then I started to go thru the steps and got hung up on curl https://bootstrap.pypa.io/get-pip.py | python3.6 This throws the error:

joe-pl@joe-PL:~$ sudo curl https://bootstrap.pypa.io/get-pip.py | python3.6
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1622k  100 1622k    0     0  3223k      0 --:--:-- --:--:-- --:--:-- 3226k
Collecting pip
  Using cached https://files.pythonhosted.org/packages/c2/d7/90f34cb0d83a6c5631cf71dfe64cc1054598c843a92b400e55675cc2ac37/pip-18.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 8.1.1
    Uninstalling pip-8.1.1:
Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/bin/pip'
Consider using the `--user` option or check the permissions.

I researched this error and found a few ways around it using sudo apt-get install python-virtualenv and virtualenv --python=/usr/bin/python3.6 venv but this did not fix the problem.

I will continue to look for a fix, but do you have any other ideas? anks, Joe Th

[JonasVautherin]

I guess I would run sudo curl https://bootstrap.pypa.io/get-pip.py | python3.6.

[joe-marrone]

Same thing happens running sudo curl https://bootstrap.pypa.io/get-pip.py | python3.6 It's got something to do with permissions, so I looked at some of the files involved, and I have rwx on them, so this seems to be OK.

Joe

[JonasVautherin]

Oh, my bad. Probably more curl https://bootstrap.pypa.io/get-pip.py | sudo python3.6.

Basically the goal is to run python3.6 get-pip.py, so you could also download the file manually and run it then.

[joe-marrone]

@JonasVautherin : Sorry, but cannot get past step 4 $ curl https://bootstrap.pypa.io/get-pip.py | python3.6

I added sudo to the last part of the command as Julian suggested, but I still get errors related to permissions. I have looked at all the directories on the /home/joe-pl/.cache/pip/http path and cannot find the offending directory. Here's the terminal output for this:

joe-pl@joe-PL:~$ curl https://bootstrap.pypa.io/get-pip.py | sudo python3.6
[sudo] password for joe-pl:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
Sorry, try again.
[sudo] password for joe-pl: 
100 1622k  100 1622k    0     0  28828      0  0:00:57  0:00:57 --:--:-- 28829
The directory '/home/joe-pl/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/joe-pl/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting pip
  Downloading https://files.pythonhosted.org/packages/c2/d7/90f34cb0d83a6c5631cf71dfe64cc1054598c843a92b400e55675cc2ac37/pip-18.1-py2.py3-none-any.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 1.7MB/s 
Installing collected packages: pip
  Found existing installation: pip 18.1
    Uninstalling pip-18.1:
      Successfully uninstalled pip-18.1
Successfully installed pip-18.1

-Joe

[JonasVautherin]

Successfully installed pip-18.1

Looks like it worked to me :sweat_smile:. Above lines say that the cache has been disabled, but well, should not be an issue.

What if you do $ pip3.6 --version now? Does it exist?

[joe-marrone]

joe-pl@joe-PL:~$ pip3.6 --version pip 18.1 from /usr/local/lib/python3.6/dist-packages/pip (python 3.6)

Looks like it's there. Now when I go to step 5 $ cd <path/to/dronecodesdk_python>/proto/pb_plugins $ pip3.6 install -r requirements.txt here's what happens:

joe-pl@joe-PL:/src/DronecodeSDK/DronecodeSDK-Python/proto/pb_plugins$ pip3.6 install -r requirements.txt
Collecting protobuf (from -r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/c2/f9/28787754923612ca9bfdffc588daa05580ed70698add063a5629d1a4209d/protobuf-3.6.1-cp36-cp36m-manylinux1_x86_64.whl (1.1MB)
    100% |████████████████████████████████| 1.1MB 940kB/s 
Requirement already satisfied: jinja2 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 2)) (2.8)
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from protobuf->-r requirements.txt (line 1)) (20.7.0)
Requirement already satisfied: six>=1.9 in /usr/lib/python3/dist-packages (from protobuf->-r requirements.txt (line 1)) (1.10.0)
Requirement already satisfied: MarkupSafe in /usr/lib/python3/dist-packages (from jinja2->-r requirements.txt (line 2)) (0.23)
Installing collected packages: protobuf
Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/local/lib/python3.6/dist-packages/protobuf-3.6.1-py3.6-nspkg.pth'
Consider using the `--user` option or check the permissions.

For some files it says 'Requirement already satisfied ' yet down at the end it says "could not install packages ... error 13". So at this point I can't tell what's been done and what's not.

-Joe

[JonasVautherin]

[Errno 13] Permission denied: '/usr/local/lib/python3.6/dist-packages/protobuf-3.6.1-py3.6-nspkg.pth'

It is trying install the packages on your system (/usr/local/lib/python3.6/...), and your user doesn't have the permission for that. Try sudo:

sudo pip3.6 install -r requirements.txt

[joe-marrone]

I think steps 1-5 are complete, so I tried step 6 in the instructions above and this was the result:

joe-pl@joe-PL:/src/DronecodeSDK/DronecodeSDK-Python/proto/pb_plugins$ sudo pip3.6 install -r requirements.txt -r requirements-dev.txt
The directory '/home/joe-pl/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/joe-pl/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Could not open requirements file: [Errno 2] No such file or directory: 'requirements-dev.txt'

Still trying to get this all resolved. Thanks, Joe

[joe-marrone]

Re: the last post. The file requirements-dev.txt is not in the directory:

joe-pl@joe-PL:/src/DronecodeSDK/DronecodeSDK-Python/proto/pb_plugins$ ls
dcsdkgen  README.rst  requirements.txt  setup.py  test

The other file is there. Any idea why the file needed for this pip3 command is not there?

-Joe

[julianoes]

You are in the directory DronecodeSDK-Python/proto/pb_plugins but you should be just in DronecodeSDK-Python/.

[joe-marrone]

Continued progress: I found the requirements-dev.txt file in a different directory, and then the command line worked. Turned out I was using the wrong directory.

-Joe

[joe-marrone]

Continued progress: I found the requirements-dev.txt file in a different directory, and then the command line worked. Turned out I was using the wrong directory.

-Joe

[joe-marrone]

Now that everything is place, I tried the script, but it didn't work.

joe-pl@joe-PL:~/src/DronecodeSDK/DronecodeSDK-Python$ python3.6 examples/takeoff_and_land.py
Traceback (most recent call last):
  File "examples/takeoff_and_land.py", line 7, in <module>
    drone = dronecode_sdk_connect(host="127.0.0.1")
  File "/src/DronecodeSDK/DronecodeSDK-Python/dronecode_sdk/__init__.py", line 49, in connect
    globals()[plugin](plugin_manager)
KeyError: 'Action'

Any other ideas about what's wrong?

Thanks, Joe

[JonasVautherin]

:thinking:

@xzvf: Would you by any chance have an idea about that KeyError: 'Action'?

[joe-marrone]

Happy New Year. I was wondering if anything had been done about the KeyError: 'Action' error?

Joe

[JonasVautherin]

Hi @joe-marrone! Happy new year, and sorry for the delay!

Can you show me the output of $ tree from /src/DronecodeSDK/DronecodeSDK-Python/dronecode_sdk?

[joe-marrone]

@JonasVautherin : Here it is. Obviously its missing, so I will add it. joe-pl@joe-PL:~/src/DronecodeSDK/DronecodeSDK-Python/dronecode_sdk$ tree The program 'tree' is currently not installed. You can install it by typing: sudo apt install tree Joe

[JonasVautherin]

Right. Please paste the output of it after you have installed it ($ sudo apt install tree) :-).

[joe-marrone]

OK, here is the tree output.

` joe-pl@joe-PL:~/src/DronecodeSDK/DronecodeSDK-Python/dronecode_sdk$ tree . ├── async_plugin_manager.py ├── _base.py ├── generated │   ├── action_pb2_grpc.py │   ├── action_pb2.py │   ├── calibration_pb2_grpc.py │   ├── calibration_pb2.py │   ├── camera_pb2_grpc.py │   ├── camera_pb2.py │   ├── core_pb2_grpc.py │   ├── core_pb2.py │   ├── discovery_pb2_grpc.py │   ├── discovery_pb2.py │   ├── gimbal_pb2_grpc.py │   ├── gimbal_pb2.py │   ├── info_pb2_grpc.py │   ├── info_pb2.py │   ├── init.py │   ├── mission_pb2_grpc.py │   ├── mission_pb2.py │   ├── telemetry_pb2_grpc.py │   └── telemetry_pb2.py ├── init.py └── plugins └── init.py

2 directories, 23 files `

Joe

[JonasVautherin]

~/src/DronecodeSDK/DronecodeSDK-Python/dronecode_sdk/plugins should not be empty. It means that run_protoc.sh failed.

Can you verify that the python calls in run_protoc.sh are python3.6 (and not python3), and run run_protoc.sh again?

[joe-marrone]

This is the run_protoc.sh file in my system. I installed this several weeks ago when I started this thread.

Joe

`

!/bin/bash

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" WORK_DIR="${SCRIPT_DIR}/../../" PROTO_DIR="${WORK_DIR}/proto" GENERATED_DIR="${WORK_DIR}/dronecode_sdk/generated" PLUGIN_DIR="${WORK_DIR}/dronecode_sdk/plugins" PLUGIN_INIT="${PLUGIN_DIR}/init.py" export TEMPLATE_PATH="${WORK_DIR}/other/templates/"

function generate { echo -e "# -- coding: utf-8 --\n" > $PLUGIN_INIT

for PROTO_FILE in `find ${PROTO_DIR} -name "*.proto" -type f`; do

    # Generate bindings for each file individually
    python3 -m grpc_tools.protoc -I${GENERATED_DIR} \
                                 --proto_path=$(dirname ${PROTO_FILE}) \
                                 --python_out=${GENERATED_DIR} \
                                 --grpc_python_out=${GENERATED_DIR} \
                                 ${PROTO_FILE}

    # For some reason the import is broken with Python3.5.x and works fine
    # with Python3.6.x, set an absolute path and everything is fine
    PROTO_IMPORT_NAME="$(basename -- ${PROTO_FILE%.*})_pb2"

    # We need to create the .original backup files, otherwise we're not compatible with
    # BSD sed.
    sed -i'.sedoriginal' -e "s/import ${PROTO_IMPORT_NAME}/from . import ${PROTO_IMPORT_NAME}/" \
        "${GENERATED_DIR}/${PROTO_IMPORT_NAME}_grpc.py"
    # Clean up the backup files.
    find ${GENERATED_DIR} -name '*.sedoriginal' -delete
    echo " -> [+] Generated protobuf and gRPC bindings for ${PROTO_IMPORT_NAME%_*}"

    # Generate plugin
    python3 -m grpc_tools.protoc -I$(dirname ${PROTO_FILE}) \
                                 --plugin=protoc-gen-custom=$(which dcsdkgen) \
                                 --custom_out=${PLUGIN_DIR} \
                                 --custom_opt=py \
                                 ${PROTO_FILE}

    WANTED_PLUGIN_NAME="$(echo ${PROTO_FILE} | sed "s#.*/\(.*\).proto#\1#g").py"
    # protoc generates java like filenames, we don't want that with python

    # @TODO: cleanup this script and figure out a way to make it cross-os friendly perhaps rewrite in python
    # capilalization as trivial as this needs a workaround for macos bash 3.2
    # this solution avoids using bash ^ substitution (from bash >4.0) and is using python 3 instead
    CAPITALIZED_PLUGIN_NAME=$(python3 -c "import sys;print(sys.argv[1].capitalize())" "$WANTED_PLUGIN_NAME")
    mv ${PLUGIN_DIR}/${CAPITALIZED_PLUGIN_NAME} ${PLUGIN_DIR}/${WANTED_PLUGIN_NAME}
    # Add to imports
    echo "from .${WANTED_PLUGIN_NAME%.py} import *" >> $PLUGIN_INIT
    echo " -> [+] Generated plugin for ${PROTO_IMPORT_NAME%_*}"

done

} function install_dcsdkgen { cd ${PROTO_DIR}/pb_plugins if [[ "$VIRTUAL_ENV" != "" ]] then pip3 install . else pip3 install --user . fi } echo "[+] Installing the DronecodeSDK autogenerator" install_dcsdkgen echo "[+] Done" echo "[+] Generating plugins from " generate echo "[+] Done" `

[JonasVautherin]

Ok. So part of what you did before was to install python3.6 and pip3.6. On your system, you should now have multiple versions of Python. python is some Python 2 (maybe 2.7?), python3 is Python 3.X (maybe 3.5?), and python3.6 is Python 3.6 (the one you just installed).

The script run_protoc.sh calls python3 and pip3, because on more modern systems like Ubuntu 18.04, that would be fine. But on your Ubuntu 16.04, you need the script to explicitly call python3.6 and pip3.6.

So you should open run_protoc.sh, and replace python3 with python3.6 everywhere. And also replace pip3 with pip3.6 everywhere. Then save your changes and run run_protoc.sh.

[joe-marrone]

I made the changes and ran run_protoc.sh. It installed OK. But now I have another issue preventing me from using the simulator. It seems that my ubuntu 16.04 install does not like the graphics card on my PC because of incompatibilities with GNOME, so I am using the standard graphics shell from Microsoft, which does not run the sim. Never a dull day!

I tried some NVidia driver updates but they didn't work either. I'm kind of stuck right now, and not sure how to proceed. Do you have any suggestions?

Thanks for all your help so far.

Joe

[julianoes]

If you're using Gazebo, you can try to run it without graphics:

HEADLESS=1 make px4_sitl gazebo

[JonasVautherin]

If you haven't installed Gazebo yet, and have Docker, you can also try to run it directly from this docker image (have a look at the documentation there).

[joe-marrone]

Unfortunately, I have not used either Gazebo or Docker yet. Joe

[JonasVautherin]

I see. I guess you may want to try Gazebo (see Julian's comment above).

I'm closing this issue now, as the Python wrapper got installed.