Old version of Pip installed?

[flatsiedatsie]

I'm working on an add-on, and it gave an error when loading a dependency.

Collecting hermes-python (from -r requirements.txt (line 2))
  Could not find a version that satisfies the requirement hermes-python (from -r requirements.txt (line 2)) (from versions: )
No matching distribution found for hermes-python (from -r requirements.txt (line 2))

I looked it up, and I learnt that this can happen if the version of Pip on the system is not up to date.

I then tried to upgrade Pip with pip3 install --upgrade pip

Then, starting the python add-on gave a different error:

Usage:   
  /usr/bin/python3 -m pip install [options] <requirement specifier> [package-index-options] ...
  /usr/bin/python3 -m pip install [options] -r <requirements file> [package-index-options] ...
  /usr/bin/python3 -m pip install [options] [-e] <vcs project url> ...
  /usr/bin/python3 -m pip install [options] [-e] <local project path> ...
  /usr/bin/python3 -m pip install [options] <archive url/path> ...

no such option: --system
Unable to install dependencies: Command '/usr/bin/python3 -m pip install --system --install-option="--prefix=" -t lib -r requirements.txt' returned non-zero exit status 2.

This is on image 0.9, Raspberry Pi 4

[mrstegeman]

The RPi uses a custom version of pip with some default options. The bootstrap.py scripts that some add-ons use expect it to work that way.

https://github.com/createcandle/Candle-manager-addon/blob/master/bootstrap.py#L22-L24

If you run the proper pip command (i.e. remove the --system flag), does your install then work?

[flatsiedatsie]

If I remove --system from the command I get:

/home/pi/.local/lib/python3.7/site-packages/pip/_internal/commands/install.py:243: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.
  cmdoptions.check_install_build_global(options)
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple

Some details:

• It does find the other dependencies from requirements.txt. Just not this one.

  Collecting fuzzywuzzy (from -r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/81/54/586e9f34401dc7f5248589765bb67d49b837e2f309f25a33e82e896cba0a/fuzzywuzzy-0.17.0.tar.gz

• hermes-python is actually installed already:

pip3 install hermes-python
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Requirement already satisfied: hermes-python in /home/pi/.local/lib/python3.7/site-packages (0.7.0)
Requirement already satisfied: enum34 in /home/pi/.local/lib/python3.7/site-packages (from hermes-python) (1.1.6)
Requirement already satisfied: future in /home/pi/.local/lib/python3.7/site-packages (from hermes-python) (0.17.1)
Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from hermes-python) (1.12.0)
Requirement already satisfied: typing in /home/pi/.local/lib/python3.7/site-packages (from hermes-python) (3.7.4)

[mrstegeman]

Do a pip3 uninstall hermes-python and then try again. You'll need it to be downloaded into the add-on's directory.

[mrstegeman]

Also, depending on what you're doing, you may have issues because hermes-python doesn't distribute tarballs, only wheels: https://pypi.org/project/hermes-python/#files

[flatsiedatsie]

Uninstalling hermes-python-0.7.0:
  Would remove:
    /home/pi/.local/lib/python3.7/site-packages/hermes_python-0.7.0.dist-info/*
    /home/pi/.local/lib/python3.7/site-packages/hermes_python/*
Proceed (y/n)? 

After uninstalling and running python3 bootstrap.pyagain I get the same error. I also tried to see if the problem was gone if I turned it into a package, but that made no difference. The same error showed up in the internal logs.

[flatsiedatsie]

Also, depending on what you're doing, you may have issues because hermes-python doesn't distribute tarballs, only wheels: https://pypi.org/project/hermes-python/#files

I don't know what that means :-)

[mrstegeman]

Wheels are pre-compiled libraries, packaged up as a zip. Typically, we want pip to pull down the tarball and compile, rather than doing that. Some of the options passed into pip might be preventing the use of wheels (.whl files).

[mrstegeman]

Regardless, I don't think there's an issue with our version of pip.

[flatsiedatsie]

Any idea how I could work around this? I could add a command shell that installs it, but I suspect this whole construction is so that other processes are not impacted by the gateway? You want to store the dependencies in a 'lib' folder? I can't find that lib folder by the way. Isn't that supposed to be in the addon's folder? (_BASE_DIR = /home/pi/.mozilla-iot/addons/my-addon-name)

I'd like to follow the 'proper' way of doing it. But how..

[mrstegeman]

Typically, the bootstrap.py script creates that lib directory and installs all the dependencies there via pip.

Instead, you could just build the package yourself and not use bootstrap.py anymore. We'd actually prefer that. It's especially easy if you're only really targeting one platform (like linux-arm). You'd then include your prebuilt libraries in the package.

Example: https://github.com/mozilla-iot/tplink-adapter/blob/master/package.sh

[flatsiedatsie]

I just saw this, could be related? https://github.com/snipsco/hermes-protocol/issues/133

[mrstegeman]

(I'm closing this issue since it's not actually an issue. Feel free to continue discussion here.)

[mrstegeman]

I just saw this, could be related? snipsco/hermes-protocol#133

No. See the log you posted:

[...]: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.

[flatsiedatsie]

OK, I like the idea that the add-on comes with the required files in it already. No need to download anything else.

So.. I should create a file in a /libdirectory that is the 'hermes-python' package, but as a.. binary? What extension does a file like this have? I'll look into how to do it.

I don't see a /lib in the TP-Link adapter?

[flatsiedatsie]

Ah, I found an example in the tar.gz of the add-on.

[flatsiedatsie]

What does the pyHS100 in pip3 install -r requirements.txt -t lib --no-binary pyHS100 --prefix "" do? it seems you are telling Pip that you don't want a binary generated for it?

(how) should I change that to my use case? I currently have three things in requirements.txt.

fuzzywuzzy
hermes-python
pyalsaaudio

Should I add them all inline? If so, then what's the use of requirements.txt?

[flatsiedatsie]

No luck trying to download hermes-python to the lib folder. It's the same error again.

pip3 install -r requirements.txt -t lib --no-binary hermes-python --prefix ""
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting hermes-python (from -r requirements.txt (line 1))
  ERROR: Could not find a version that satisfies the requirement hermes-python (from -r requirements.txt (line 1)) (from versions: none)
ERROR: No matching distribution found for hermes-python (from -r requirements.txt (line 1))

I was able to download the others though.

[mrstegeman]

The --no-binary option tells pip to not use the wheel.

The problem with wheels is that they are architecture specific, but also specific to the Python version. If you're only targeting the Raspberry Pi on Buster, then just build that. Another option is to include the hermes-python tarball and build that with the bootstrap.py script.

[flatsiedatsie]

Removing the --no-binary option allows something to be downloaded.

Will the add-on work with the 'normal' download?

Another option is to include the hermes-python tarball and build that with the bootstrap.py script. You mean to download the source code from github and then build it on device? I guess what would allow it to run on more devices, as you say.

But doesn't that add the question of requiring other things to compile it? At the Hermes github page it says you need to have rust and cargo installed to do that. So would the bootstrap.py have to install rust+cargo?

[mrstegeman]

Ah, didn't realize the hefty dependencies for that library. Yes, those would have to be installed.

You can include the wheel in your package. You could even include multiple wheels, if you want, for different Python versions and architectures. You'll have to add the correct one to your sys.path at runtime.

For example: https://github.com/createcandle/voco/blob/master/main.py#L10

You'll need to also append the proper *.whl file.

[flatsiedatsie]

I ran into a similar bump when trying to add pyalsaaudio dependency. The package script can only run if I first add

sudo apt-get install libasound2-dev

So that would mean that this would have to be added via bootstrap.py. But then at least that part could be compiled. But I know that add-ons that install packages are also frowned upon.

What to do in this case? What would you prefer?

A. Install this requirement, and Rust, and make it fully locally compilable? or B. download the binaries to the lib folder.

[mrstegeman]

You're right that we don't really like having add-ons install system dependencies.

There are 2 routes you could take:

1. Submit a PR to the gateway which adds the required packages to the base image and via OTA updates.
2. Host your own add-on list and do whatever you want in your add-ons.

It seems like with this add-on and another one you proposed which adds a desktop environment to the gateway, option 2 might be the way for you to go, that way you don't have to try to patch the gateway all the time for things you need for your use case.

Alternatively, instead of making this an add-on, you could make it a standalone application which talks to the gateway via the REST API, and provide an easy way to install it.

[mrstegeman]

Or, yes, you can pull in the wheels into your lib folder. That's an easy short-term solution as well.

[flatsiedatsie]

I was just thinking that. If this is a linux-only add-on, wouldn't it be possible to just let pip3do it's glorious thing?

[mrstegeman]

Yes. Just remember to add the downloaded wheel to your sys.path.

[flatsiedatsie]

I thought: why not just do what package.shdoes in bootstrap?

I could manually check if the required files were in the /libfolder, and if not, let pip download them to that folder.

pip3 install -r requirements.txt -t lib --no-binary fuzzywuzzy --prefix ""

I guess I basically just described what bootstrap already does.

[mrstegeman]

You sure did. 😄