mk-fg / python-pulse-control

Python high-level interface and ctypes-based bindings for PulseAudio (libpulse)
https://pypi.org/project/pulsectl/
MIT License
170 stars 36 forks source link

Manual download .zip, can't run test_with_dummy_instance.py #69

Closed pippim closed 2 years ago

pippim commented 2 years ago

Background

My python music player uses pactl to turn up/down volume over 1 second for Play/Pause toggle. The command-line call is inefficient so, a direct call to libpulse0 using pulsectl makes sense.

Steps Taken

Unfortunately I couldn't find an apt list candidate for pulsectl so I manually downloaded the .zip and extracted it to a local directory ~/python/pulsectl. Then chmod a+x *.py to make all python programs executable.

Error

When running test_with_dummy_instance.py the following error occurs:

$ test_with_dummy_instance.py

from: can't read /var/mail/__future__

^C^C^C./test_with_dummy_instance.py: line 12: syntax error near unexpected token `1,'
./test_with_dummy_instance.py: line 12: `   sys.path.insert(1, os.path.join(__file__, *['..']*2))'

Initially only the error from: can't read /var/mail/__future__ appears and program appears to be in infinite loop. So, Ctrl + C must be repeated used to get command prompt back.

A /var/mail directory does exist but, it is empty.

Looking at test_with_dummy_instance.py source around line 12:

try: import pulsectl
except ImportError:
    sys.path.insert(1, os.path.join(__file__, *['..']*2))
    import pulsectl

Unfortunately the os.path.join(__file__, *['..']*2)) is above my pay grade.

Any ideas how to get over this hurtle?


Environment

TL;DR Just in case they're needed, here are some environment details.

$ pulseaudio --version
pulseaudio 8.0

:~/python/pulsectl$ grep version setup.py
    version = '22.1.0',

$ apt list | grep libpulse

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libpulse-dev/xenial-updates,xenial-security 1:8.0-0ubuntu3.15 amd64
libpulse-java/xenial,xenial 2.4.7-1 all
libpulse-jni/xenial 2.4.7-1 amd64
libpulse-mainloop-glib0/xenial-updates,xenial-security,now 1:8.0-0ubuntu3.15 amd64 [installed]
libpulse-ocaml/xenial 0.1.2-1build3 amd64
libpulse-ocaml-dev/xenial 0.1.2-1build3 amd64
libpulse0/xenial-updates,xenial-security,now 1:8.0-0ubuntu3.15 amd64 [installed]
libpulsedsp/xenial-updates,xenial-security,now 1:8.0-0ubuntu3.15 amd64 [installed]

$ apt list | grep alsa | grep installed

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

alsa-base/xenial,xenial,now 1.0.25+dfsg-0ubuntu5 all [installed]
alsa-tools/xenial,now 1.1.0-0ubuntu1 amd64 [installed]
alsa-tools-gui/xenial,now 1.1.0-0ubuntu1 amd64 [installed]
alsa-utils/xenial,now 1.1.0-0ubuntu5 amd64 [installed]
gstreamer1.0-alsa/xenial-updates,xenial-security,now 1.8.3-1ubuntu0.3 amd64 [installed]
python-alsaaudio/xenial,now 0.7-1 amd64 [installed]
mk-fg commented 2 years ago

I think simple advice would be to just not run that .py file directly, use e.g. python2 -m unittest discover or python2 -m pulsectl.tests.test_with_dummy_instance from the same dir where setup.py file is (probably ~/python/pulsectl or the single subdir under it), as also suggested in the README - https://github.com/mk-fg/python-pulse-control/#tests

pippim commented 2 years ago

@mk-fg Thanks for the quick response.

Following first suggestion: python2 -m unittest discover yields:

$ python2 -m unittest discover

E
======================================================================
ERROR: test_with_dummy_instance (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_with_dummy_instance
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/rick/python/pulsectl/test_with_dummy_instance.py", line 10, in <module>
    try: import pulsectl
  File "/home/rick/python/pulsectl/pulsectl.py", line 9, in <module>
    from . import _pulsectl as c
ValueError: Attempted relative import in non-package

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)

Following second suggestion: python2 -m pulsectl.tests.test_with_dummy_instance, yields:

$ python2 -m pulsectl.tests.test_with_dummy_instance

Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 163, in _run_module_as_main
    mod_name, _Error)
  File "/usr/lib/python2.7/runpy.py", line 102, in _get_module_details
    loader = get_loader(mod_name)
  File "/usr/lib/python2.7/pkgutil.py", line 464, in get_loader
    return find_loader(fullname)
  File "/usr/lib/python2.7/pkgutil.py", line 474, in find_loader
    for importer in iter_importers(fullname):
  File "/usr/lib/python2.7/pkgutil.py", line 430, in iter_importers
    __import__(pkg)
  File "pulsectl.py", line 9, in <module>
    from . import _pulsectl as c
ValueError: Attempted relative import in non-package
mk-fg commented 2 years ago

Wrt initial issue - this part that you mentioned above in tests:

try: import pulsectl
except ImportError:
    sys.path.insert(1, os.path.join(__file__, *['..']*2))
    import pulsectl

...is intended to try importing "pulsectl" module normally, for which it should be in PYTHONPATH (env var) or that sys.path list (same thing), which is the list of dirs where python searches for its modules to import by name like that, e.g. /usr/lib/python2.7/site-packages and similar dist-packages dirs on Debian/Ubuntu.

But for tests to work without having to install the package, just from the unpacked-archive dir, there's that sys.path.insert(1, os.path.join(__file__, *['..']*2)) fallback, which is supposed to start from __file__, which would be something/something/pulsectl/tests/test_with_dummy_instance.py, then append two .. at the end to go up to a dir which contains while "pulsectl" module, like something/something in earlier example, and prepend that to sys.path, then try to re-import module.

Apparently I've moved the tests in the past however without tweaking that fallback, and it should have *3 now, which I've just fixed in 7541bb9.

mk-fg commented 2 years ago

And I think that "ValueError: Attempted relative import in non-package" might be because you're trying to run those commands from the dir where pulsectl.py file is, which should produce these errors. That's not the directory that I meant above, i.e. you want one dir above that (one where setup.py file is).

Though don't know if that's actually the case, might be something else too, just seem to look like it.

mk-fg commented 2 years ago

Or - come to think of it - it can also be due to that broken sys.path.insert(), as it'd insert that wrong subdir into sys.path as well, though I think python2 -m unittest discover shouldn't run that fallback at all, but not sure.

mk-fg commented 2 years ago

Anyhow, try grabbing latest zip and double-check the dir, let me know if issue is still there. You should also be able to run python2 pulsectl/tests/test_with_dummy_instance.py with it, though I don't think it should actually be related to error that you've posted in initial msg.

mk-fg commented 2 years ago

Unfortunately I couldn't find an apt list candidate for pulsectl so I manually downloaded the .zip and extracted it to a local directory ~/python/pulsectl. Then chmod a+x *.py to make all python programs executable.

When running test_with_dummy_instance.py the following error occurs:

$ test_with_dummy_instance.py

from: can't read /var/mail/future

^C^C^C./test_with_dummy_instance.py: line 12: syntax error near unexpected token 1,' ./test_with_dummy_instance.py: line 12: sys.path.insert(1, os.path.join(file, ['..']2))'

Initially only the error from: can't read /var/mail/future appears and program appears to be in infinite loop. So, Ctrl + C must be repeated used to get command prompt back.

Re-reading this just now, I believe problem here is that you use python module incorrectly:

With fix mentioned above you can run it via python2 ..., and that should work, but that's still not really how python modules (like this one) are supposed to work - you always have these in python's "path" somewhere, which typically includes current dir ("."), and either "import" them in your script or use via "python -m ...".

mk-fg commented 2 years ago

Also, on an unrelated note - you seem to be using python2 and an old ubuntu release (xenial from 2016), and while this module should work there, I think both python2 and that OS release are not supported upstream by now, so there might be issues with installing anything relatively-modern there, and you might want to update to a more recent release/distro in general, if possible. (which would probably have python 3.8-3.10 instead of 2.7 as well)

pippim commented 2 years ago

@mk-fg Yes I'm planning on writing a python program to "suck in" all of 16.04 partition to a fresh install of 22.04 on a new partition. Ubuntu 22.04 LTS should take me off Python 2.7.12 to Python 3.10 I think. As 22.04 is still a few months away there didn't seem to be any rush developing the 16.04 conversion program. I imagine many problems with converting Python programs even though they already use future print, with, subprocess32, Tk vs tk (For Tkinter), etc. As such I see a lot of sed usage in conversion program.

I need to reread all your suggestions on the core issue with running the unit tests. I guess I have a bad habit of putting the shebang #!/usr/bin/env python at the top of each file and never looking back to consider other people don't do that.

I haven't used pip yet, just apt install and adding a ppa every now and then. Thank you very much for your extremely quick and very thorough analysis.

mk-fg commented 2 years ago

I guess I have a bad habit of putting the shebang #!/usr/bin/env python at the top of each file

I do that with actual executable python scripts as well of course, it just doesn't apply to modules, and tests are kinda part of those. Though guess it shouldn't be a problem to add it to that test-suite-file here, especially since it has if __name__ == '__main__': unittest.main() at the bottom for that to work anyway...

pippim commented 2 years ago

@mk-fg I patched in your changes. Hopefully I didn't miss any:

$ diff test_with_dummy_instance_2022_1-22.py test_with_dummy_instance.py
0a1
> #!/usr/bin/env python
12c13,14
<   sys.path.insert(1, os.path.join(__file__, *['..']*2))
---
>   sys.path.insert( 1,
>       os.path.abspath(os.path.join(__file__, *['..']*3)) )

Still had errors and then I realized your repo has /tests subdirectory so I used:

$ mkdir tests

$ cd tests

$ cp ../test_with_dummy_instance.py .

Now running the program works perfectly:


$ test_with_dummy_instance.py

EEE
======================================================================
ERROR: setUpClass (__main__.DummyTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test_with_dummy_instance.py", line 228, in setUpClass
    cls.instance_info = dummy_pulse_init()
  File "./test_with_dummy_instance.py", line 85, in dummy_pulse_init
    try: _dummy_pulse_init(info)
  File "./test_with_dummy_instance.py", line 122, in _dummy_pulse_init
    s.bind((addr, p))
  File "/usr/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 99] Cannot assign requested address

======================================================================
ERROR: test_crash_after_connect (__main__.PulseCrashTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test_with_dummy_instance.py", line 653, in test_crash_after_connect
    info = dummy_pulse_init()
  File "./test_with_dummy_instance.py", line 85, in dummy_pulse_init
    try: _dummy_pulse_init(info)
  File "./test_with_dummy_instance.py", line 122, in _dummy_pulse_init
    s.bind((addr, p))
  File "/usr/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 99] Cannot assign requested address

======================================================================
ERROR: test_reconnect (__main__.PulseCrashTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test_with_dummy_instance.py", line 665, in test_reconnect
    info = dummy_pulse_init()
  File "./test_with_dummy_instance.py", line 85, in dummy_pulse_init
    try: _dummy_pulse_init(info)
  File "./test_with_dummy_instance.py", line 122, in _dummy_pulse_init
    s.bind((addr, p))
  File "/usr/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 99] Cannot assign requested address

----------------------------------------------------------------------
Ran 2 tests in 0.004s

FAILED (errors=3)

I've only done a little bit of socket programming to paint Left & Right VU Meters in the music player. So, I wonder if I need to install one of these uninstalled packages for pulsectrl?:

gstreamer0.10-pulseaudio/xenial-updates,xenial-security 0.10.31-3+nmu4ubuntu2.16.04.3 amd64
libcanberra-pulse-dbg/xenial 0.30-2.1ubuntu1 amd64
libpulse-dev/xenial-updates,xenial-security 1:8.0-0ubuntu3.15 amd64
libpulse-java/xenial,xenial 2.4.7-1 all
libpulse-jni/xenial 2.4.7-1 amd64
libpulse-ocaml/xenial 0.1.2-1build3 amd64
libpulse-ocaml-dev/xenial 0.1.2-1build3 amd64
libsox-fmt-pulse/xenial-updates,xenial-security 14.4.1-5+deb8u4ubuntu0.1 amd64
liquidsoap-plugin-pulseaudio/xenial 1.1.1-7.1 amd64
osspd-pulseaudio/xenial 1.3.2-7 amd64
pulseaudio-esound-compat/xenial-updates,xenial-security 1:8.0-0ubuntu3.15 amd64
pulseaudio-module-droid/xenial-updates,xenial-security 1:8.0-0ubuntu3.15 amd64
pulseaudio-module-gconf/xenial-updates,xenial-security 1:8.0-0ubuntu3.15 amd64
pulseaudio-module-jack/xenial-updates,xenial-security 1:8.0-0ubuntu3.15 amd64
pulseaudio-module-lirc/xenial-updates,xenial-security 1:8.0-0ubuntu3.15 amd64
pulseaudio-module-raop/xenial-updates,xenial-security 1:8.0-0ubuntu3.15 amd64
pulseaudio-module-trust-store/xenial-updates,xenial-security 1:8.0-0ubuntu3.15 amd64
pulseaudio-module-zeroconf/xenial-updates,xenial-security 1:8.0-0ubuntu3.15 amd64
pulseview/xenial 0.2.0-1.1 amd64
snd-gtk-pulse/xenial 16.1-5 amd64
xfce4-pulseaudio-plugin/xenial 0.2.4-1 amd64
xmms2-plugin-pulse/xenial 0.8+dfsg-14build3 amd64

I tried sudo apt install libpulse-dev which seemed a likely candidate but that didn't make the errors go away...

pippim commented 2 years ago

Did a little digging and lines 111:115 contain:

# Pick some random available localhost ports
if not info.get('sock_unix'):
bind = (
    ['127.0.0.1', 0, socket.AF_INET], ['::1', 0, socket.AF_INET6],
    ['127.0.0.1', 0, socket.AF_INET], ['127.0.0.1', 0, socket.AF_INET] )

Because IPv6 is turned off (not much luck with that for me) and only IPv4 is used, this might be cause of the errors being reported?

mk-fg commented 2 years ago

Still had errors and then I realized your repo has /tests subdirectory so I used:

Not sure what you mean here, it shouldn't be needed at all. And you were in the "pulsectl/tests" directory when running that "mkdir" command.

And again, it's weird that you seem to insist on running that file directly - everything should work from the repo root directory, it's the one where setup.py and README.rst files are, really should. Trying to chmod and run random single files internal to the module is kinda strange.

mk-fg commented 2 years ago

Now running the program works perfectly:

Except for test errors of course.

I've only done a little bit of socket programming to paint Left & Right VU Meters in the music player. So, I wonder if I need to install one of these uninstalled packages for pulsectrl?:

No no, it's all just stuff in python.

mk-fg commented 2 years ago

Because IPv6 is turned off (not much luck with that for me) and only IPv4 is used, this might be cause of the errors being reported?

Yeah, gotta be that, will check where it's used in tests, probably easy to disable if bind fails.

mk-fg commented 2 years ago

Yeah, gotta be that, will check where it's used in tests, probably easy to disable if bind fails.

Added a check and skip for one connect-test that uses that ipv6 socket explitictly in eeae4b5, should fix the issue, I think. Thanks for pointing it out.

Also, don't think you need to download apply these patches manually - maybe clone the git repo, and then applying them would be easy via "git pull" - it's what git is there for after all :) Though should work either way, of course.

pippim commented 2 years ago

Not sure what you mean here, it shouldn't be needed at all.
And you were in the "pulsectl/tests" directory when running that "mkdir" command.

When I wrote the steps taken to fix the problem I copy and pasted from history after the fact. When downloading the zip and extracting, everything went into ~/python/pulsectl directory. There was no /tests/ subdirectory like in the repo. So I created /tests/ manually:

drwxrwxr-x 3 rick rick  4096 Jan 22 16:21 ./
drwxrwxr-x 7 rick rick 12288 Jan 23 08:22 ../
-rw-rw-r-- 1 rick rick  2180 Jan 17 13:13 CHANGES.rst
-rw-rw-r-- 1 rick rick  1123 Jan 17 13:13 COPYING
-rw-rw-r-- 1 rick rick    37 Jan 17 13:13 .gitignore
-rwxr-xr-x 1 rick rick   418 Jan 17 13:13 .git.prepare-commit-msg-hook*
-rwxrwxr-x 1 rick rick     0 Jan 17 13:13 __init__.py*
-rw-rw-r-- 1 rick rick   125 Jan 22 14:56 __init__.pyc
-rwxrwxr-x 1 rick rick  4385 Jan 17 13:13 lookup.py*
-rw-rw-r-- 1 rick rick    39 Jan 17 13:13 MANIFEST.in
-rwxrwxr-x 1 rick rick 22448 Jan 17 13:13 _pulsectl.py*
-rwxrwxr-x 1 rick rick 41062 Jan 17 13:13 pulsectl.py*
-rw-rw-r-- 1 rick rick 23944 Jan 22 16:21 _pulsectl.pyc
-rw-rw-r-- 1 rick rick 55863 Jan 22 12:58 pulsectl.pyc
-rw-rw-r-- 1 rick rick 16392 Jan 17 13:13 README.rst
-rw-rw-r-- 1 rick rick    26 Jan 17 13:13 setup.cfg
-rwxrwxr-x 1 rick rick  1247 Jan 17 13:13 setup.py*
drwxrwxr-x 2 rick rick  4096 Jan 22 14:56 tests/
-rwxrwxr-x 1 rick rick 25414 Jan 22 14:53 test_with_dummy_instance_2022_1-22.py*
-rwxrwxr-x 1 rick rick 25457 Jan 22 14:55 test_with_dummy_instance.py*
-rwxrwxr-x 1 rick rick 25414 Jan 17 13:13 test_with_dummy_instance.py~*
-rw-rw-r-- 1 rick rick 28517 Jan 22 12:58 test_with_dummy_instance.pyc

Anyway everything works PERFECTLY now for the needed pulseaudio information as code snippet below shows:

# After calling 11 times crashes so call outside of loop
pulse = pulsectl.Pulse()

def sink_master():
    # Some pulsectl testing
    # pulse.server_info()
    for sink in pulse.sink_input_list():
        print("\n=======  ", sink.index, sink.volume, sink.name)
        print(sink.proplist['application.name'])
        #print(sink.proplist)
        #pulse_dict = sink.proplist
        #for i in pulse_dict:
        #    print("key:", i, "value:", pulse_dict[i])

As noted in the comment above, the program crashes if pulse = pulsectl.Pulse() is called ~11 times. Just a beginner's mistake though and others won't have the same problem.

Using git pull in a working directory and copying only the required files to the PyCharm project directory is a great idea. Currently PyCharm is lagging with 25,000 lines in the music player plus 75,000 lines of other stuff the music player doesn't use. I will move mserve (music player) and all it's dependencies into it's own directory out of python directory.

You can close this issue now after posting any closing thoughts. I must say your response time and thoroughness in answers is second-to-none! Thank you very much :)

mk-fg commented 2 years ago

There was no /tests/ subdirectory like in the repo. So I created /tests/ manually: ... -rw-rw-r-- 1 rick rick 26 Jan 17 13:13 setup.cfg -rwxrwxr-x 1 rick rick 1247 Jan 17 13:13 setup.py* drwxrwxr-x 2 rick rick 4096 Jan 22 14:56 tests/ ...

^^^ there you created "tests" directory next to setup.py file.

vvv here are repository contents, as displayed by github:

scr-20220123212149

Downloading the .zip file and checking its contents:

% curl -LO https://github.com/mk-fg/python-pulse-control/archive/refs/heads/master.zip
%  unzip -l master.zip
Archive:  master.zip
eeae4b50d21b421b63704c55cecaa924f5686c88
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2022-01-23 05:56   python-pulse-control-master/
      418  2022-01-23 05:56   python-pulse-control-master/.git.prepare-commit-msg-hook
       37  2022-01-23 05:56   python-pulse-control-master/.gitignore
     2180  2022-01-23 05:56   python-pulse-control-master/CHANGES.rst
     1123  2022-01-23 05:56   python-pulse-control-master/COPYING
       39  2022-01-23 05:56   python-pulse-control-master/MANIFEST.in
    16392  2022-01-23 05:56   python-pulse-control-master/README.rst
        0  2022-01-23 05:56   python-pulse-control-master/pulsectl/
      665  2022-01-23 05:56   python-pulse-control-master/pulsectl/__init__.py
    22448  2022-01-23 05:56   python-pulse-control-master/pulsectl/_pulsectl.py
     4385  2022-01-23 05:56   python-pulse-control-master/pulsectl/lookup.py
    41062  2022-01-23 05:56   python-pulse-control-master/pulsectl/pulsectl.py
        0  2022-01-23 05:56   python-pulse-control-master/pulsectl/tests/
        0  2022-01-23 05:56   python-pulse-control-master/pulsectl/tests/__init__.py
    25645  2022-01-23 05:56   python-pulse-control-master/pulsectl/tests/test_with_dummy_instance.py
       26  2022-01-23 05:56   python-pulse-control-master/setup.cfg
     1247  2022-01-23 05:56   python-pulse-control-master/setup.py
---------                     -------
   115667                     17 files

There's no "tests" directory next to setup.py in the repo, nor there is in the zip file. There is one INSIDE "pulsectl" dir, but that's not where you seem to be creating it at all, that's inside the module dir, not next to setup.py. So pretty sure you just misremember where you've seen that dir, and it should totally be where it's supposed to be when you clone the repo or unpack that zip file (as per unzip -l output above).

mk-fg commented 2 years ago

# After calling 11 times crashes so call outside of loop pulse = pulsectl.Pulse() ... As noted in the comment above, the program crashes if pulse = pulsectl.Pulse() is called ~11 times. Just a beginner's mistake though and others won't have the same problem.

Not exactly sure why it crashes, and there's no test for creating 11 concurrent libpulse contexts quickly in a loop, but yeah, that's not really how it's meant to be used - you usually need no more than one at a time.

Also, I'd suggest using it like very first example in the README illustrates, and as is idiomatic in python scripts, using context managers and "with" statement:

with pulsectl.Pulse('my-client') as pulse:
  ... do stuff with "pulse"

That way, once you go out of that "with" context, "pulse" object is properly closed. It's same as you'd typically work with files or any other stateful objects in python.

Putting it into a global var should also work, but might not be closing libpulse context properly, but it's unlikely to be a an issue, since presumably that happens when script exits anyway.

mk-fg commented 2 years ago

You can close this issue now after posting any closing thoughts. I must say your response time and thoroughness in answers is second-to-none! Thank you very much :)

Thanks for testing a bunch of rarely-used stuff too.

Suspect no one tried to run those tests directly in a long time, or ever on a system with IPv6 disabled :) Also, your ubuntu release is about as old as this module/repo, so it's interesting and kinda nice that it even still works there.