Exception on Keypirinha start

[psistorm]

Hi there,

unfortunately there is an Error thrown when trying to install your new version or restarting Keypirinha with the new version in InstalledPackage folder.

07:42:59.743 Python: Traceback (most recent call last):
07:42:59.743 Python:   File "keypirinha_private.py", line 307, in reload_module
07:42:59.743 Python:   File "lib\importlib\__init__.py", line 126, in import_module
07:42:59.743 Python:   File "<frozen importlib._bootstrap>", line 986, in _gcd_import
07:42:59.743 Python:   File "<frozen importlib._bootstrap>", line 969, in _find_and_load
07:42:59.743 Python:   File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
07:42:59.743 Python:   File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
07:42:59.743 Python:   File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
07:42:59.743 Python:   File "keypirinha_private.py", line 205, in load_module
07:42:59.743 Python:   File "C:\Tools\Keypirinha\portable\Profile\InstalledPackages\Kill.keypirinha-package\kill.py", line 5, in <module>
07:42:59.743 Python:   File "keypirinha_private.py", line 205, in load_module
07:42:59.743 Python:   File "C:\Tools\Keypirinha\portable\Profile\InstalledPackages\Kill.keypirinha-package\lib\psutil.py", line 124, in <module>
07:42:59.743 Python:   File "keypirinha_private.py", line 205, in load_module
07:42:59.743 Python:   File "C:\Tools\Keypirinha\portable\Profile\InstalledPackages\Kill.keypirinha-package\lib\psutil\_pswindows.py", line 15, in <module>
07:42:59.743 Python: ImportError: cannot import name '_psutil_windows'

[polyvertex]

I would throw in a guess that it's because part of the code of the psutil module cannot be loaded due to it being in a .pyd file. .pyd files are DLL files and cannot be loaded natively (well, at least natively).

As a more general remark, importing non-pure python modules (i.e.: that are not shipped with KP) is not recommended because of potential ABI incompatibilities. For example, if your .pyd dll hasn't been compiled with the same version of the same compiler than KP, you may run into trouble.

[polyvertex]

On Windows platforms, DLL loading from memory is not supported by Microsoft so while being "doable", it's a hack. So no matter the solution, .pyd file has to be extracted out of the zip archive first, and put in a folder that is referenced by Python's sys.path.

Unfortunately, you're a bit ahead of the game here since I haven't had time to work on a clean and automated solution for that in KP.

What you can try is to copy the .pyd file to your package's Live directory from the on_start method of your plugin if it's not present. I'm not sure it will work but you can give it a try as it's the cleanest potential solution for now. EDIT: However, note that you won't be able to import the psutil module before that operation obviously.

[psistorm]

Well even copying the .pyd to python/dlls as suggested doesn't work (also not work python/dlls/psutil). I extracted the package to the Live Packages folder which throws the same error for me here.

I have already done such copying in the on_start for a package that is not open yet which does apparently work pretty well using the load_binary_resource function in kpu. I used the cache folder, but I reference the needed .exe there directly. Which is probably something different than using the import mechanism.

[polyvertex]

EDIT Well, I just have to work on it in order to find a clean and automated solution that deals with versions collisions and all that stuff... Sorry for that.

[psistorm]

I just have found out the Error message changed a little bit when placing the files in suggested folders. Just have to find out which folder was working that way. Error message was:

2016-08-25 09:10:27.955 [i] Python: Traceback (most recent call last):
2016-08-25 09:10:27.955 [i] Python:   File "keypirinha_private.py", line 307, in reload_module
2016-08-25 09:10:27.955 [i] Python:   File "lib\importlib\__init__.py", line 126, in import_module
2016-08-25 09:10:27.955 [i] Python:   File "<frozen importlib._bootstrap>", line 986, in _gcd_import
2016-08-25 09:10:27.955 [i] Python:   File "<frozen importlib._bootstrap>", line 969, in _find_and_load
2016-08-25 09:10:27.955 [i] Python:   File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
2016-08-25 09:10:27.955 [i] Python:   File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
2016-08-25 09:10:27.955 [i] Python:   File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
2016-08-25 09:10:27.955 [i] Python:   File "keypirinha_private.py", line 205, in load_module
2016-08-25 09:10:27.955 [i] Python:   File "C:\Tools\Keypirinha\portable\Profile\Packages\Kill\kill.py", line 5, in <module>
2016-08-25 09:10:27.955 [i] Python:     from .lib import psutil
2016-08-25 09:10:27.955 [i] Python:   File "keypirinha_private.py", line 205, in load_module
2016-08-25 09:10:27.955 [i] Python:   File "C:\Tools\Keypirinha\portable\Profile\Packages\Kill\lib\psutil\__init__.py", line 124, in <module>
2016-08-25 09:10:27.955 [i] Python:     from . import _pswindows as _psplatform
2016-08-25 09:10:27.955 [i] Python:   File "keypirinha_private.py", line 205, in load_module
2016-08-25 09:10:27.955 [i] Python:   File "C:\Tools\Keypirinha\portable\Profile\Packages\Kill\lib\psutil\_pswindows.py", line 15, in <module>
2016-08-25 09:10:27.955 [i] Python:     from . import _psutil_windows as cext
2016-08-25 09:10:27.955 [i] Python: ImportError: cannot import name '_psutil_windows'

[polyvertex]

You mean you put the .pyd file in the portable/Profile/Packages/Kill/lib/psutil/ folder?

In that case, have you removed the pyd from the .keypirinha-package archive? It's possible that the import machinery tries to load the pyd from it otherwise.

[ueffel]

Does it at least work for you guys, if you extract the files from the .keypirinha-package into \Packages\Kill?

[polyvertex]

@ueffel Just did a simple test by extracting the package into its Live folder and it seems to work like a charm.

[psistorm]

I tried different combinations. Maybe I forgot to remove it from the archive. Nevertheless, removing all occurences of the package and only extracting it to \Packages\Kill leads to the following error:

10:29:51.717 Python: Traceback (most recent call last):
10:29:51.717 Python:   File "keypirinha_private.py", line 307, in reload_module
10:29:51.717 Python:   File "lib\importlib\__init__.py", line 126, in import_module
10:29:51.717 Python:   File "<frozen importlib._bootstrap>", line 986, in _gcd_import
10:29:51.717 Python:   File "<frozen importlib._bootstrap>", line 969, in _find_and_load
10:29:51.717 Python:   File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
10:29:51.717 Python:   File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
10:29:51.717 Python:   File "<frozen importlib._bootstrap_external>", line 665, in exec_module
10:29:51.717 Python:   File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
10:29:51.717 Python:   File "C:\Tools\Keypirinha\portable\Profile\Packages\Kill\kill.py", line 5, in <module>
10:29:51.717 Python:     from .lib import psutil
10:29:51.717 Python:   File "C:\Tools\Keypirinha\portable\Profile\Packages\Kill\lib\psutil\__init__.py", line 124, in <module>
10:29:51.717 Python:     from . import _pswindows as _psplatform
10:29:51.717 Python:   File "C:\Tools\Keypirinha\portable\Profile\Packages\Kill\lib\psutil\_pswindows.py", line 15, in <module>
10:29:51.717 Python:     from . import _psutil_windows as cext
10:29:51.717 Python: ImportError: DLL load failed: Das angegebene Modul wurde nicht gefunden.

The last error message line says something like DLL load failed: The request module can not be found.

Edit: My current workplace is Windows 7, x64, Enterprise Edition if that is of interest.

[polyvertex]

@psistorm The same test worked for me but I had to restart KP due to some ghosting of module(s) in Python interpreter. Have you tried to restart KP?

[psistorm]

@polyvertex Yes I cleaned all directories (also python/dlls from stuff of Kill package), closed Keypirinha, copied the contents of the package to said directory (portable/Profile/Packages/Kill) and the error message seen above occurred.

[polyvertex]

@psistorm Stupid question: do you actually have the pyd file present in your folder (portable/Profile/Packages/Kill/lib/psutil/_psutil_windows.cp35-win_amd64.pyd)? You might have accidentally removed it from the archive for example during previous tests or something like that?

[psistorm]

@polyvertex There are no stupid questions :smile: Yes I have the file actually available.

[polyvertex]

@ueffel Would you mind doing the exact same test (extracting Kill into its live folder on a clean KP install, then start KP and see what happens)?

[ueffel]

@polyvertex Extracted KP 2.9.1 in new directory, extracted contents of Kill.keypirinha-package in portable\Profile\Packages\Kill => works.

i have the sneaky suspicion that the keypirinha packaged python gets the missing stuff from installed Python version.

[psistorm]

Well that could be an explanation. I also have a Python version installed (Python 3.5.1) but it is a clean Python without any further packages installed.

[polyvertex]

i have the sneaky suspicion that the keypirinha packaged python gets the missing stuff from installed Python version.

No way if modules do not try to override the import machinery in some way (standard behavior). psutil seems to be clean for that matter.

[ueffel]

maybe not. i delete the python\scripts directory from my path but it still works...

[polyvertex]

@ueffel read my comment right above (we posted at the same time)

[ueffel]

@polyvertex ok. then i'm quite baffled with the error @psistorm gets...

in the meantime i tried to copy the library to the cache path but loading the files as bytes fails i tried the following code snippet in my on_start method

        resources = self.find_resources('*.*')
        self.dbg(resources)
        for resource in resources:
            try:
                binary = self.load_binary_resource(resource)
                self.dbg("binary with size %d bytes loaded" % len(binary))
            except Exception as exp:
                self.dbg("{} binary not loaded".format(exp))

and it drops me that in the console:

11:26:31.976 Kill.Kill(91): DEBUG: ['kill.ini', 'kill.py', 'README.md', 'lib/psutil/_common.py', 'lib/psutil/_compat.py', 'lib/psutil/_psbsd.py', 'lib/psutil/_pslinux.py', 'lib/psutil/_psosx.py', 'lib/psutil/_psposix.py', 'lib/psutil/_pssunos.py', 'lib/psutil/_psutil_windows.cp35-win_amd64.pyd', 'lib/psutil/_pswindows.py', 'lib/psutil/__init__.py']
11:26:31.977 Kill.Kill(95): DEBUG: binary with size 602 bytes loaded
11:26:31.978 Kill.Kill(95): DEBUG: binary with size 9404 bytes loaded
11:26:31.979 Kill.Kill(95): DEBUG: binary with size 584 bytes loaded
11:26:31.979 Kill.Kill(97): DEBUG: Resource not found: Kill/lib/psutil/_common.py binary not loaded
11:26:31.980 Kill.Kill(97): DEBUG: Resource not found: Kill/lib/psutil/_compat.py binary not loaded
11:26:31.981 Kill.Kill(97): DEBUG: Resource not found: Kill/lib/psutil/_psbsd.py binary not loaded
11:26:31.982 Kill.Kill(97): DEBUG: Resource not found: Kill/lib/psutil/_pslinux.py binary not loaded
11:26:31.982 Kill.Kill(97): DEBUG: Resource not found: Kill/lib/psutil/_psosx.py binary not loaded
11:26:31.983 Kill.Kill(97): DEBUG: Resource not found: Kill/lib/psutil/_psposix.py binary not loaded
11:26:31.987 Kill.Kill(97): DEBUG: Resource not found: Kill/lib/psutil/_pssunos.py binary not loaded
11:26:31.988 Kill.Kill(97): DEBUG: Resource not found: Kill/lib/psutil/_psutil_windows.cp35-win_amd64.pyd binary not loaded
11:26:31.990 Kill.Kill(97): DEBUG: Resource not found: Kill/lib/psutil/_pswindows.py binary not loaded
11:26:31.991 Kill.Kill(97): DEBUG: Resource not found: Kill/lib/psutil/__init__.py binary not loaded

what am i doing wrong?

[polyvertex]

@ueffel At first glance you haven't done anything wrong I would say. This part of the API has been slightly modified in 2.9 (due to Keypirinha/Keypirinha#73), as far as zip archives are concerned, so it might be a bug in KP. I'll investigate and report.

[polyvertex]

Ah. OK @ueffel's last comment aside, I found out why the pyd gets loaded on my machine but not @psistorm's one: DLL dependency.

The pyd @ueffel provides has been compiled with the VS2015 so the OS has to find VS2015 runtime DLLs. KP on the other hand has been compiled with VS2013 (and so its embedded Python) so the MSVC DLLs bundled with KP do not fit (they are named very differently anyway) and OS has to find them beyond KP's scope. In that regard, I'm pretty sure @psistorm's machine does not have VS2015 runtime DLLs installed (EDIT but you had a different message so I'm a bit skeptical still).

Well, we're back to my initial remark about shipping non-pure Python modules in your package...

If you get the big picture right, even if I implement an API that help dealing with pyd files, still, we're entering in the scope of Python's dependencies nightmare (i.e. version/ABI collisions; which actually becomes Windows DLLs nightmare). Imagine I write a plugin that needs a different version of psutil... Both DLLs cannot be instantiated at the same time and we end up having the wrong DLL for at least one of the plugin and all the issues that may occur because of that.

[polyvertex]

Bottom line is: sorry @ueffel, unluckily you still have to find a viable solution to list running processes.

[ueffel]

I will cry myself to sleep tonight :(

[polyvertex]

Geeez @ueffel you've found a gigantic and long-standing bug in the resource finding/loading API (check out Keypirinha/Keypirinha#111). I'm pretty sure that was your revenge for your struggles with developing your plugin :smile:

[polyvertex]

@ueffel Could you confirm you have VS2015 runtime files installed on your machine?

[ueffel]

@polyvertex i have Visual Studio Community 2015 installed at home, so i guess yes? if you mean the Microsoft Visual C++ 2015 Redistributables, i have them too

[polyvertex]

Yes either covers it. So it seems that was the right reason why it works for you as well.

[ueffel]

Removed the psutil library and the corresponding build. its not a good solution to ship the psutil binary with the package. i have to find another way.