Could not import psutil on FreeBSD 9

[GoogleCodeExporter]

What steps will reproduce the problem?

1. Install with pip on FreeBSD 9.0-RELEASE-p3, Python 2.7.3, in virtualenv.
2. Open python shell.
3. import psutil

What is the expected output?
The module is imported.

What do you see instead?

Python 2.7.3 (default, Sep 13 2012, 11:46:06)
[GCC 4.2.1 20070831 patched [FreeBSD]] on freebsd9
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/home/voodoo_people/python/lib/python2.7/site-packages/psutil/__init__.py", line 88, in <module>
    import psutil._psbsd as _psplatform
  File "/usr/home/voodoo_people/python/lib/python2.7/site-packages/psutil/_psbsd.py", line 28, in <module>
    TOTAL_PHYMEM = _psutil_bsd.get_virtual_mem()[0]
SystemError: error return without exception set

What version of psutil are you using? What Python version?
psutil 0.6.1, Python 2.7.3.

On what operating system? Is it 32bit or 64bit version?

Please provide any additional information below.

Original issue reported on code.google.com by sunrize...@gmail.com on 13 Sep 2012 at 4:34

[GoogleCodeExporter]

Have you tried latest SVN revision?

Original comment by g.rodola on 13 Sep 2012 at 4:36

[GoogleCodeExporter]

The last SVN version:

Python 2.7.3 (default, Sep 13 2012, 11:46:06)
[GCC 4.2.1 20070831 patched [FreeBSD]] on freebsd9
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
/usr/home/voodoo_people/python/lib/python2.7/site-packages/psutil-0.6.2-py2.7-fr
eebsd-9.0-RELEASE-p3-amd64.egg/_psutil_bsd.py:3: UserWarning: Module 
_psutil_bsd was already imported from 
/usr/home/voodoo_people/python/lib/python2.7/site-packages/psutil-0.6.2-py2.7-fr
eebsd-9.0-RELEASE-p3-amd64.egg/_psutil_bsd.pyc, but 
/usr/home/wyse/build/psutil-read-only is being added to sys.path
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "psutil/__init__.py", line 88, in <module>
    import psutil._psbsd as _psplatform
  File "psutil/_psbsd.py", line 28, in <module>
    TOTAL_PHYMEM = _psutil_bsd.get_virtual_mem()[0]
SystemError: error return without exception set
>>>

Original comment by sunrize...@gmail.com on 13 Sep 2012 at 5:05

[GoogleCodeExporter]

Ok, something is definitively wrong then.
I won't be able to look into this any time soon though.

Original comment by g.rodola on 13 Sep 2012 at 5:07

[GoogleCodeExporter]

Just got it running with 0.4.1 version.

Original comment by sunrize...@gmail.com on 13 Sep 2012 at 5:19

[GoogleCodeExporter]

Getting back to this.
Unfortunately I'm not able to reproduce the issue.

Would you be able to try to debug the problem yourself in order to figure out 
what's the function which is causing the error?

You should look here:
http://code.google.com/p/psutil/source/browse/trunk/psutil/_psutil_bsd.c?spec=sv
n1550&r=1550#581
Basically, what's going on in there is that one of those sysctl() or 
sysctlbyname() calls is failing and the first thing to do is figure out which 
one.
The weird thing is that contrarily to what stated in the man pages errno is not 
set.

Original comment by g.rodola on 13 Dec 2012 at 2:31

• Changed state: Started
• Added labels: Crash, OpSys-FreeBSD

[GoogleCodeExporter]

Does psutil work fine with GENERIC kernel? This could be 
http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/172803

Original comment by dstew...@gmail.com on 28 Jan 2013 at 1:12

[GoogleCodeExporter]

Not sure what you mean by GENERIC kernel but we might try the patch proposed in 
that ticket.
Can someone who's able to reproduce this issue try this patch (save it into a 
file and apply with "patch -p0 < file.patch")?

Index: psutil/_psutil_bsd.c
===================================================================
--- psutil/_psutil_bsd.c    (revisione 1560)
+++ psutil/_psutil_bsd.c    (copia locale)
@@ -584,11 +584,16 @@
 static PyObject*
 get_virtual_mem(PyObject* self, PyObject* args)
 {
-    unsigned int   total, active, inactive, wired, cached, free, buffers;
+    unsigned int   total, active, inactive, wired, cached, free;
     size_t        size = sizeof(total);
    struct vmtotal vm;
    int            mib[] = {CTL_VM, VM_METER};
    long           pagesize = getpagesize();
+#if __FreeBSD_version > 702101
+    long buffers;
+#else
+    int buffers;
+#endif

     if (sysctlbyname("vm.stats.vm.v_page_count", &total, &size, NULL, 0))
         goto error;
@@ -602,7 +607,7 @@
         goto error;
     if (sysctlbyname("vm.stats.vm.v_free_count", &free, &size, NULL, 0))
         goto error;
-    if (sysctlbyname("vfs.bufspace", &buffers, &size, NULL, 0))
+    if (sysctlbyname("vfs.bufspace", &buffers, sizeof(buffers), NULL, 0))
         goto error;

     size = sizeof(vm);

Original comment by g.rodola on 28 Jan 2013 at 1:29

[GoogleCodeExporter]

Sorry, there's a typo.
Here's the correct patch (not tested):

Index: psutil/_psutil_bsd.c
===================================================================
--- psutil/_psutil_bsd.c    (revisione 1560)
+++ psutil/_psutil_bsd.c    (copia locale)
@@ -584,11 +584,16 @@
 static PyObject*
 get_virtual_mem(PyObject* self, PyObject* args)
 {
-    unsigned int   total, active, inactive, wired, cached, free, buffers;
+    unsigned int   total, active, inactive, wired, cached, free;
     size_t        size = sizeof(total);
    struct vmtotal vm;
    int            mib[] = {CTL_VM, VM_METER};
    long           pagesize = getpagesize();
+#if __FreeBSD_version > 702101
+    long buffers;
+#else
+    int buffers;
+#endif
+size_t buffers_size = sizeof(buffers);

     if (sysctlbyname("vm.stats.vm.v_page_count", &total, &size, NULL, 0))
         goto error;
@@ -602,7 +607,7 @@
         goto error;
     if (sysctlbyname("vm.stats.vm.v_free_count", &free, &size, NULL, 0))
         goto error;
-    if (sysctlbyname("vfs.bufspace", &buffers, &size, NULL, 0))
+    if (sysctlbyname("vfs.bufspace", &buffers, &buffers_size, NULL, 0))
         goto error;

     size = sizeof(vm);

Original comment by g.rodola on 28 Jan 2013 at 1:31

[GoogleCodeExporter]

Hi again, guys. I was unable to reproduce this issue today.

[sunrize@PRWC013 ~]$ uname -a
FreeBSD PRWC013.local 9.0-RELEASE-p3 FreeBSD 9.0-RELEASE-p3 #0: Tue Jun 12 
02:52:29 UTC 2012     
root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
[sunrize@PRWC013 ~]$ source psutils_test/bin/activate
(psutils_test)[sunrize@PRWC013 ~]$ pip install psutil
Downloading/unpacking psutil
  Downloading psutil-0.6.1.tar.gz (138kB): 138kB downloaded
  Running setup.py egg_info for package psutil

Installing collected packages: psutil
  Running setup.py install for psutil
    building '_psutil_bsd' extension
    cc -fno-strict-aliasing -O2 -pipe -fno-strict-aliasing -DNDEBUG -O2 -pipe -fno-strict-aliasing -fPIC -I/usr/local/include/python2.7 -c psutil/_psutil_bsd.c -o build/temp.freebsd-9.0-RELEASE-p3-amd64-2.7/psutil/_psutil_bsd.o
    cc -fno-strict-aliasing -O2 -pipe -fno-strict-aliasing -DNDEBUG -O2 -pipe -fno-strict-aliasing -fPIC -I/usr/local/include/python2.7 -c psutil/_psutil_common.c -o build/temp.freebsd-9.0-RELEASE-p3-amd64-2.7/psutil/_psutil_common.o
    cc -fno-strict-aliasing -O2 -pipe -fno-strict-aliasing -DNDEBUG -O2 -pipe -fno-strict-aliasing -fPIC -I/usr/local/include/python2.7 -c psutil/arch/bsd/process_info.c -o build/temp.freebsd-9.0-RELEASE-p3-amd64-2.7/psutil/arch/bsd/process_info.o
    cc -shared -pthread build/temp.freebsd-9.0-RELEASE-p3-amd64-2.7/psutil/_psutil_bsd.o build/temp.freebsd-9.0-RELEASE-p3-amd64-2.7/psutil/_psutil_common.o build/temp.freebsd-9.0-RELEASE-p3-amd64-2.7/psutil/arch/bsd/process_info.o -ldevstat -o build/lib.freebsd-9.0-RELEASE-p3-amd64-2.7/_psutil_bsd.so
    building '_psutil_posix' extension
    cc -fno-strict-aliasing -O2 -pipe -fno-strict-aliasing -DNDEBUG -O2 -pipe -fno-strict-aliasing -fPIC -I/usr/local/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.freebsd-9.0-RELEASE-p3-amd64-2.7/psutil/_psutil_posix.o
    cc -shared -pthread build/temp.freebsd-9.0-RELEASE-p3-amd64-2.7/psutil/_psutil_posix.o -o build/lib.freebsd-9.0-RELEASE-p3-amd64-2.7/_psutil_posix.so

Successfully installed psutil
Cleaning up...
(psutils_test)[sunrize@PRWC013 ~]$ python
Python 2.7.3 (default, Oct 24 2012, 15:09:17) 
[GCC 4.2.1 20070831 patched [FreeBSD]] on freebsd9
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> psutil.__version__
'0.6.1'
>>> 

Original comment by sunrize...@gmail.com on 28 Feb 2013 at 9:21

[GoogleCodeExporter]

Should i test this patch anyway?

Original comment by sunrize...@gmail.com on 28 Feb 2013 at 9:26

[GoogleCodeExporter]

> I was unable to reproduce this issue today.

Damn. Is it the exact same system you were using in the first place? Is 
something changed? Point is if we're not able to reproduce the original issue 
there's no point in committing the patch.

Original comment by g.rodola on 28 Feb 2013 at 9:44

[GoogleCodeExporter]

Yes, everything was the same: same os, same psutil and python version. But it 
was not the same machine. Ok, i'll try to figure out, whether that server is 
available now.

Original comment by sunrize...@gmail.com on 28 Feb 2013 at 9:54

[GoogleCodeExporter]

This is now mitigated by revision d6cf2f8ba8c0 which makes psutil raise an 
exception later than sooner, so that at least we don't crash at import time.
Instead we'll get a TypeError on get_memory_percent().
Did you get the chance to put hands on that machine?

Original comment by g.rodola on 5 Mar 2013 at 9:58

[GoogleCodeExporter]

I just experienced this, and so did another user.

I've committed the patch to the FreeBSD ports tree, because it works well.

http://www.freebsd.org/cgi/query-pr.cgi?pr=172803

Chris

Original comment by utis...@gmail.com on 13 Apr 2013 at 11:55

[GoogleCodeExporter]

Oh nice! Can you confirm the attached patch fixes the issue?

Original comment by g.rodola on 13 Apr 2013 at 12:29

Attachments:

• 325.patch

[GoogleCodeExporter]

I cannot reproduce the issue but it seems I'm not going to receive any 
confirmation about the effectiveness of the patch any time soon either, so 
given this is a high priority issue I just committed the patch in revision 
9b6e780ea6b5.

I'm going to consider this fixed for now.

Original comment by g.rodola on 2 May 2013 at 2:45

• Changed state: FixedInHG
• Added labels: Milestone-0.7.1

[GoogleCodeExporter]

Hi, I'm using psutil 0.7.0 on FreeBSD 8.2 (amd64) and Python 2.7.1. 
Getting same problem on import:

>>> import psutil
python2.7/site-packages/psutil-0.7.0-py2.7-freebsd-8.2-RELEASE-amd64.egg/psutil/
_psbsd.py:38: RuntimeWarning: couldn't determine platform's TOTAL_PHYMEM
  warnings.warn("couldn't determine platform's TOTAL_PHYMEM", RuntimeWarning)

Most things work though (psutil.swap_memory() for example). But not 
psutil.virtual_memory():

>>> psutil.swap_memory()
swap(total=34359607296, used=0, free=34359607296, percent=0.0, sin=0, 
sout=328114176)

>>> psutil.virtual_memory()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/AppDocs/local/lib/python2.7/site-packages/psutil-0.7.0-py2.7-freebsd-8.2-RELEASE-amd64.egg/psutil/__init__.py", line 1154, in virtual_memory
    return _psplatform.virtual_memory()
  File "/AppDocs/local/lib/python2.7/site-packages/psutil-0.7.0-py2.7-freebsd-8.2-RELEASE-amd64.egg/psutil/_psbsd.py", line 66, in virtual_memory
    mem =  _psutil_bsd.get_virtual_mem()
SystemError: error return without exception set

Original comment by jtim.arn...@gmail.com on 2 May 2013 at 7:01

[GoogleCodeExporter]

Please try latest repository version and confirm it's fixed:

hg clone https://g.rodola@code.google.com/p/psutil/ 
cd psutil
python setup.py install
python -c "import psutil; psutil.virtual_memory()"

Original comment by g.rodola on 3 May 2013 at 10:15

[GoogleCodeExporter]

thanks, it does work. 
Let me know if you ever need help with FreeBSD 8 and Python2.7, I'll be glad to 
test.

Original comment by jtim.arn...@gmail.com on 3 May 2013 at 1:48

[GoogleCodeExporter]

Original comment by g.rodola on 3 May 2013 at 3:43

• Changed state: Fixed