Import of psutil errors with "import _psutil_mswindows .. DLL load failed"

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Install python-3.3.0 and psutil 0.6.1.win32 on windows-XP, 32bit
2. start python cli
3. import psutil

What is the expected output?

succesful import of psutil

What do you see instead?
>python
Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:55:48) [MSC v.1600 32 bit (In
tel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\Python33\lib\site-packages\psutil\__init__.py", line 76, in <module>
    import psutil._psmswindows as _psplatform
  File "c:\Python33\lib\site-packages\psutil\_psmswindows.py", line 16, in <modu
le>
    import _psutil_mswindows
ImportError: DLL load failed: The specified procedure could not be found.
>>>

What version of psutil are you using? What Python version?
psutil 0.6.1
python 3.3

On what operating system? Is it 32bit or 64bit version?
Windows XP, 32bit, SP-3

Please provide any additional information below.

I have installed same python 3.3 and psutil 0.6.1 on a windows 7 systems and it 
works.

Original issue reported on code.google.com by Drydercs...@gmail.com on 2 Jan 2013 at 1:56

[GoogleCodeExporter]

I don't have time to look into this now but a quick google search suggests this 
a problem common to other projects as well.

Original comment by g.rodola on 4 Jan 2013 at 4:55

[GoogleCodeExporter]

I checked _psutil_mswindows.pyd with Dependency Walker. It's looking for the 
following functions in Kernel32.dll:

K32EnumProcesses
K32GetMappedFileNameA
K32GetProcessImageFileNameA
K32GetProcessMemoryInfo

These were added to Kernel32 in NT 6.1+, such as Windows 7. If you're using XP 
(NT 5.1) or Vista (NT 6.0), they're instead in Psapi.dll, without the K32 
prefix. The following blog discusses a workaround to bypass the K32
macros by setting PSAPI_VERSION=1:

http://blogs.msdn.com/b/vcblog/archive/2009/08/27/windows-sdk-v7-0-v7-0a-incompa
tibility-workaround.aspx

For now, in Vista I used Visual Studio 2010 Express to build from source.

Original comment by eryksun on 19 Mar 2013 at 8:42

[GoogleCodeExporter]

Thanks for the insight.
Can someone please try the patch in attachment? I'm still unable to reproduce 
this issue.

Original comment by g.rodola on 19 Mar 2013 at 10:37

Attachments:

• issue348.patch

[GoogleCodeExporter]

Thanks for the insight.
Can someone please try the patch in attachment? I'm still unable to reproduce 
this issue.

Original comment by g.rodola on 19 Mar 2013 at 10:37

Attachments:

• issue348.patch

[GoogleCodeExporter]

It's already conditionally defined in Psapi.h:

#ifndef PSAPI_VERSION
#if (NTDDI_VERSION >= NTDDI_WIN7)
#define PSAPI_VERSION 2
#else
#define PSAPI_VERSION 1
#endif
#endif

I can successfully build from source on Vista. The issue is with the version 
distributed in the downloads. You have the option to force PSAPI_VERSION=1 when 
building so as to also support XP and Vista in a single build. This incurs a 
performance penalty because the linker uses Psapi.dll instead of the K32 
functions in kernel32.dll. Instead you could instead distribute separate 
versions for XP/Vista and Windows 7+.

Original comment by eryksun on 22 Mar 2013 at 12:53

[GoogleCodeExporter]

You mean performance penalty when compiling?
Distributing different installers for XP/7+ is a non option since we're already 
providing 8 installers for Windows.

Original comment by g.rodola on 22 Mar 2013 at 6:16

[GoogleCodeExporter]

The old Psapi.dll still exists in 7+. They've just reimplemented it in 
kernel32.dll for performance. You could test linking to kernel32.dll vs 
psapi.dll, but I doubt there's much difference in Python, in which case it 
makes sense to force PSAPI_VERSION=1 for the build that you distribute. You can 
leave it as an option for someone on Windows 7+ building their own copy.

Original comment by eryksun on 22 Mar 2013 at 10:15

[GoogleCodeExporter]

Fine. Can anybody confirm that the patch I attached earlier fixes this problem?

Original comment by g.rodola on 23 Mar 2013 at 11:22

[GoogleCodeExporter]

For Windows 7, WINVER == _WIN32_WINNT == 0x0601, and the test would be WINVER 
>= 0x0601. 

But why not modify your macro definition in setup.py to ('_WIN32_WINNT', 
'0x501'), which targets XP and implicitly sets the psapi, or instead add 
('PSAPI_VERSION', 1)?

Windows version macros:
http://msdn.microsoft.com/en-us/library/aa383745

Original comment by eryksun on 23 Mar 2013 at 4:17

[GoogleCodeExporter]

> For Windows 7, WINVER == _WIN32_WINNT == 0x0601, and 
> the test would be WINVER >= 0x0601. 

Right, thanks. How about now? Can someone try the patch in attachment?

> But why not modify your macro definition in setup.py to 
> ('_WIN32_WINNT', '0x501')

Because we're relying on _WIN32_WINNT in different parts of _psutil_mswindows.c.

Original comment by g.rodola on 26 Mar 2013 at 12:13

Attachments:

• issue348.patch

[GoogleCodeExporter]

What did you need this tested on? XP? I don't have an XP box I can test on at 
the moment, just Win 2003 I think. 

Original comment by jlo...@gmail.com on 26 Mar 2013 at 2:02

[GoogleCodeExporter]

The original OP's message was talking about Win XP. I have Win XP but I cannot 
reproduce the issue though so basically what I need is someone who does.

Original comment by g.rodola on 26 Mar 2013 at 2:13

[GoogleCodeExporter]

Regarding the patch, you call EnumProcesses in process_info.c, so you'd have to 
add it there too. Also, you're only defining the _WIN32_WINNT macro in 
setup.py, so you need to test _WIN32_WINNT instead. The value of WINVER gets 
set to _WIN32_WINNT in the headers included by Windows.h. However, rather than 
define this all over the place, you could just do it once in setup.py.

I still don't grok this approach, though. Currently, if someone builds psutil 
himself with SDK 7 (Python 3.3 needs this version), he won't see this extension 
module import problem. Your setup.py sets _WIN32_WINNT to the build system's 
version of NT, which indirectly chooses the correct PSAPI_VERSION. However, if 
he builds an installer on NT 6.1+ using SDK 7, the extension module won't work 
on NT 5.x/6.0 -- as setup.py currently stands. 

It seems the simplest approach is to target the pre-built extension module to 
the widest audience possible. It's not as if an XP/Vista consumer of the 
pre-built 3.3 extension cares about optimizations for a particular _WIN32_WINNT 
version at *build* time. If the build system is NT 6.1+, it's just liable to 
break things for them. You can add an option to override the default 0x0501 
target for the rare user on Windows that actually builds C extensions from 
source.

Original comment by eryksun on 26 Mar 2013 at 4:22

[GoogleCodeExporter]

I'm not sure I'm completely following you here. My knowledge about Windows, 
especially when C code is involved is rather limited.
If the problem is clear to you and you can reproduce the issue maybe you can 
provide a patch?
The ideal solution would be to fix the problem on all Windows 
versions/compilers transparently without involving a cmdline option for 
setup.py (if that is what you meant in the first place).

Original comment by g.rodola on 26 Mar 2013 at 4:29

• Added labels: OpSys-Windows

[GoogleCodeExporter]

We were able to fix this for the one user who was seeing it by copying the two 
files

_psutil_mswindows.pyc
_psutil_mswindows.pyd

into dist/library/psutil.

See https://groups.google.com/forum/?fromgroups#!topic/psutil/gQa0z872pJI

Original comment by makdaddy98102@gmail.com on 29 Apr 2013 at 7:05

[GoogleCodeExporter]

That's not the best solution though.
I was hoping you could try the patch above, compile psutil, and tell us if that 
fixes the issue.

Original comment by g.rodola on 2 May 2013 at 2:56

[GoogleCodeExporter]

Original comment by g.rodola on 3 May 2013 at 3:20

• Added labels: Priority-High
• Removed labels: Priority-Medium

[GoogleCodeExporter]

hi! any news on this one? it's still broken in 1.1.3 on my windows xp box (with 
python 3.3). this is a stopper for me. :-(

Original comment by radekpod...@gmail.com on 18 Nov 2013 at 12:44

[GoogleCodeExporter]

I proposed a solution (issue348.patch) asking if somebody could try the patch 
and report back but no one did.
I cannot know by myself whether issue348.patch fixes the issue because I cannot 
reproduce it.

Original comment by g.rodola on 18 Nov 2013 at 1:12

[GoogleCodeExporter]

i'd be happy to test it but i'm unable to compile it myself... could you 
prepare a package so i can just install and test it?

Original comment by radekpod...@gmail.com on 18 Nov 2013 at 1:25

[GoogleCodeExporter]

Ok, I can do that later today when I'll be back home.
What Python version? 32 or 64 bit?

Original comment by g.rodola on 18 Nov 2013 at 1:31

[GoogleCodeExporter]

great! it's python 3.3 (32bit), win xp sp-3 32bit

Original comment by radekpod...@gmail.com on 18 Nov 2013 at 1:33

[GoogleCodeExporter]

I currently cannot compile for Python 3.3. 
Here's an installer for Python 2.7 32-bit (please install that one).

Original comment by g.rodola on 18 Nov 2013 at 6:17

Attachments:

• psutil-2.0.0.win32-py2.7.exe

[GoogleCodeExporter]

imports fine (and cpu_percent() works) so i'd call it fixed. thanks!

also, if there's nothing else stopping a new release, please do so (our 
company's product just migrated to python3 and this turns out to be a 
show-stopper). thank you very much.

Original comment by radekpod...@gmail.com on 19 Nov 2013 at 9:59

[GoogleCodeExporter]

I will definitively release a version soon, although I'm not sure when (I hope 
someday during this week).

Original comment by g.rodola on 19 Nov 2013 at 4:12

[GoogleCodeExporter]

Fixed in version 1.2.0 which I released a moment ago.

Original comment by g.rodola on 20 Nov 2013 at 8:33

• Changed state: Fixed
• Added labels: Milestone-1.2.0

[GoogleCodeExporter]

just installed the 1.2.0 version under python 3.3 and the import fails just as 
with the old version. :-( ...so this is unfortunately NOT fixed and seems to be 
not only windows xp related but also python 3 related...

Original comment by radekpod...@gmail.com on 21 Nov 2013 at 12:03

[GoogleCodeExporter]

btw, should you need a test-monkey i'd be happy to run and test anything you 
give me (compiled). ;-)

Original comment by radekpod...@gmail.com on 21 Nov 2013 at 12:45

[GoogleCodeExporter]

Shit!
What about Python 2.7? Does it fail too?

Original comment by g.rodola on 21 Nov 2013 at 1:05

[GoogleCodeExporter]

no, the 1.2.0 version imports fine under python 2.7.

Original comment by radekpod...@gmail.com on 21 Nov 2013 at 1:28

[GoogleCodeExporter]

That doesn't make any sense to me. They were compiled form the same source code 
base.

Original comment by g.rodola on 21 Nov 2013 at 1:35

[GoogleCodeExporter]

Can someone else try 1.2.0 on Windows XP?

Original comment by g.rodola on 21 Nov 2013 at 1:36

• Changed state: ReOpened

[GoogleCodeExporter]

is it compiled using the same compiler (same version)? i thought python3 is 
linked to a newer version of msvcr or something (i'm really not familiar with 
the details)...

Original comment by radekpod...@gmail.com on 21 Nov 2013 at 1:46

[GoogleCodeExporter]

Refer to comment #13. Your latest build links to kernel32.dll K32EnumProcesses, 
instead of psapi.dll EnumProcesses. K32EnumProcesses does not exist prior to 
Windows 7. 

I don't agree with patching this separately in every compilation unit. Plus, 
even if it is done that way, it should be testing the _WIN32_WINNT version that 
you define in setup.py. WINVER doesn't get updated to _WIN32_WINNT until after 
you include Windows.h.

IMO, just set _WIN32_WINNT to '0x0501' (XP) in setup.py when building for the 
distribution that needs to support XP/Vista. 

http://code.google.com/p/psutil/source/browse/setup.py?name=release-1.2.0#87

That gives you the old PSAPI without any further mucking around, and it still 
runs in all versions of Windows. Or just unconditionally define PSAPI_VERSION 
to 1 when building for distribution. Either way, someone can always build a 
version targeting a newer Windows version if they so choose.

Original comment by eryksun on 21 Nov 2013 at 2:25

[GoogleCodeExporter]

eryksun: thanks for the help and sorry for not considering your comment #13 but 
that's because I don't have a proper understanding of the problem.
Please take a look at file "348.patch" I'm attaching. Is this what you were 
suggesting?

radekpodgorny: attached are two compiled versions for Python 2.7 and 3.3 which 
were produced after applying the patch in attachment. Could you please try them 
and report back?

Original comment by g.rodola on 21 Nov 2013 at 9:45

• Added labels: Milestone-1.2.1
• Removed labels: Milestone-1.2.0

Attachments:

• psutil-1.2.1.win32-py2.7.exe
• psutil-1.2.1.win32-py3.3.exe
• 348.patch

[GoogleCodeExporter]

sorry to say bad news but it's still the same: works under 2.7 and fails under 
3.3.

(yes, i've checked 1.2.1 was actually installed for both pythons in control 
panel)

Original comment by radekpod...@gmail.com on 21 Nov 2013 at 10:40

[GoogleCodeExporter]

p.s.: i see you're probably compiling with mingw. don't you have a tutorial on 
how to set up the compiling environment you have so can play with it myself? 
thank you!

Original comment by radekpod...@gmail.com on 21 Nov 2013 at 10:43

[GoogleCodeExporter]

Take a look at the INSTALL file. 
It contains instructions on how to build using mingw32.

Original comment by g.rodola on 21 Nov 2013 at 10:46

[GoogleCodeExporter]

1) installed mingw (actually found some old install on my system)
2) cloned hg version on psutil
3) built it as recommended (-c mingw32) (had to tweak distutils not to pass 
-mno-cygwin)
4) installed (setup.py install)
5) succesfully imported in python (3.3)

...so i has to be something related to your build system.

Original comment by radekpod...@gmail.com on 21 Nov 2013 at 11:26

[GoogleCodeExporter]

I use Visual Studio in order to compile for Python 3.3, that's why.

Original comment by g.rodola on 21 Nov 2013 at 11:41

[GoogleCodeExporter]

The extension module in the 1.2.1 build is back to using the kernel32 K32* 
functions that aren't available in XP/Vista. In patch 348 if it's not building 
on XP (Vista is NT 6.0), _WIN32_WINNT is set to the build system Windows 
version returned by get_winver(). But if it's building on XP this isn't even an 
issue, and get_winver() would return '0x0501' anyway.

What I proposed was to unconditionally define ('_WIN32_WINNT', '0x0501') or 
('PSAPI_VERSION', 1). At least do this for the builds on PyPI. 

Using the old PSAPI will work fine on new versions of Windows -- at least for 
as long as Microsoft supports psapi.dll. That should be at least until Vista is 
no longer supported.

Original comment by eryksun on 21 Nov 2013 at 11:54

[GoogleCodeExporter]

I added ('PSAPI_VERSION', 1) in setup.py. 
radekpodgorny here's the new exes.

Original comment by g.rodola on 22 Nov 2013 at 8:28

Attachments:

• psutil-1.2.1.win32-py2.7.exe
• psutil-1.2.1.win32-py3.3.exe

[GoogleCodeExporter]

works! thank you!

Original comment by radekpod...@gmail.com on 23 Nov 2013 at 10:45

[GoogleCodeExporter]

Committed in revision 195e1eaf4a9c.

Original comment by g.rodola on 25 Nov 2013 at 6:46

• Changed state: FixedInHG

[GoogleCodeExporter]

Closing.

Original comment by g.rodola on 26 Nov 2013 at 8:52

[GoogleCodeExporter]

Original comment by g.rodola on 26 Nov 2013 at 8:52

• Changed state: Fixed