Sabnzbd-testing Import Error at running python scripts

[WebBuddha]

Few days ago, I've updated the sabnzb-testing package and the Synology Firmware update. I've installed pip and request again and tried to let my sabnzb script running, but recieved following error ImportError: No module named urllib3 I've runned follwing command via ssh sudo /usr/local/sabnzbd-testing/env/bin/pip install urllib3 After that, I've got ImportError: No module named chartet next command... sudo /usr/local/sabnzbd-testing/env/bin/pip install chartet next error... ImportError: No module named certifi next command... sudo /usr/local/sabnzbd-testing/env/bin/pip install certifi

Script is running aggain. sudo pip install request or one of that modules didn't fixed the problem.

[ymartin59]

@Safihre May you please have a look at this request

[Safihre]

Is this different from previous behavior? These modules weren't available in older packages either I think?

[WebBuddha]

I thought that last times it was only the module "requests" which was missing.

[Safihre]

It's strange indeed, by installing the requests package it should have also pulled in the dependencies. I did update the version of Requests with the latest sabnzbd spk update.. Maybe this caused a problem? https://github.com/SynoCommunity/spksrc/blob/master/spk/sabnzbd-testing/src/requirements.txt But then I guess the building of the package should have failed. I didn't get errors, did you @ymartin59?

[BenjV]

When you install the requests module with pip all the other modules will be installed also. Except of course chartet that should be chardet.

You can check the installed modules with this command. sudo /usr/local/sabnzbd-testing/env/bin/pip freeze

Maybe you made a mistake and forget to use the complete path. In that case the modules will be installed in the python environment of DSM and not in the virtual environment of SabNzbd.

By the way it is requests and not request.

[Safihre]

When sabnzbd calls a post processing script, it just calls python script.py. Is that linked to the same python as the one running sabnzbd or to the system python?

[BenjV]

No the SynoCommunity packages create a virtual environment for Python packages.

So if a post processingscript calls python it will get a shell environment in that virtual environment and use the python of the SynoCommunity. In this virtual environment the correct path to the SynoCommunity pyhon is added at the beginning of the PATH.

Virtual environments are used so that evry package can have its own library module. This is to prevent problems when for example package A needs requests version X and Package B needs request version Y. With a virtual environment both packages can use their own version of requests while sharing the same python.

[Safihre]

Then it's weird that @WebBuddha is getting the error. Because urllib3 is a dependency of requests, so it should be there (since it's build with the package) when SABnzbd calls the script, right?

[BenjV]

I don't know the structure of Sabnzbd. Maybe they just included libraries like request as a source and import them with a full path. The virtual environment is added by the SynoCommunity package and not by SabNzbd, so most likely the module is not added to the python sitepackages (sloppy developers). In that case you have to add the module yourself with pip or import them with the full path like Sab most likely does.

Good python developers should call pip from within their application and install the module if it is missing. In that case it would have been added to the virtual environment.

What happend here is probably that @WebBuddha added the module to the wrong python.

[Safihre]

@BenjV I think you are missing something, SABnzbd does not need requests but it is included (to be helpful for extensions scripts) in the spk via the method provided by SynoCommunity : https://github.com/SynoCommunity/spksrc/wiki/Using-wheels-to-distribute-Python-packages

So if requests is not available within the python that is running SABnzbd, it's a problem of the way the SynoCommunity builder includes it.

---

EDIT: This is a bit weird statement:

Good python developers should call pip from within their application and install the module if it is missing.

On most Unix platforms installing of pip modules requires sudo-rights, so just running it from a normal python application is not possible at all and I don't think I've ever seen this in any python application.

[ymartin59]

Moreover installing pip module "on runtime" may require compilation chain on system... what DSM does not have. It is important to determine all runtime dependencies of a Python application to properly publish it as package. For instance, beets package is delivered without acoustid plugin because of lacking dependency on chromaprint... anyone may complain but it is documented as non available.

[BenjV]

Your missing the point. I did not say installing pip. Pip is installed automatically if there is a virtual environment and if not it will be there as part of the python installation. What I said was installing modules from within python with the use of pip. By the way just do a google on "pip.main" to find out how that works.

Python developers should anticipate the use of a virtual environment and install standard module if needed like requests from within their code by calling pip install. In case of a virtual environment there will be no need for sudo rights to be able to do that, that's the beauty of it. I would argue that always using a virtual environment would be the way to go for every python application.

But back to the problem now. I did get the point that the topic poster needs the requests module for a postproject script. I did not know that is was already supplied by Sabnzbd via the spkrsc I am neither familiar with Sabnzbd nor with the spksrc framework but I know a little of python. Depending on how SABnzbd is spawning the postprocess it is possible that the postprocess gets a new Shell and not the virtual environment. In which case it will use the system python and the topic poster should add the request package to that python.

Normally a postprocess would just get a complete commandline from an .ini file to spawn a process, leaving it to the user to choose the python (or other tool) he wants to use and not add the call to python to it by Sabnzbd. That way they can python or other tools they want like just a shell script.

[Safihre]

@ymartin59 Could you run that /usr/local/sabnzbd-testing/env/bin/pip freeze command on the build SABnzbd spk (testing or regular version)? Then we know what is inside :)

[Safihre]

The new Python package will now include requests and as such also these dependencies. Can be found in #3138 for now. We hope to publish the packages to the official repo soon and appreciate all testing and reports on the functioning of these packages.