pa_jack.h dependency on macOS

[shakfu]

Hi,

It's me again. I was checking out your project which has moved along nicely. (-:

Just wanted to highlight something: on a macOS if one installs portaudio via the typical brew install portaudio, then note that only portaudio.h is installed. Therefore a simple python3 setup.py build of your library will stop due to not finding pa_jack.h. If one manually includes this header copying it from the portaudio repo (as I did) then it builds fine.

[shakfu]

Just following up on the last post. It seems that the hackish inclusion of the pa_jack.h, its still not finding the symbol:

In [1]: import cysounddevice
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-081ae6209e10> in <module>
----> 1 import cysounddevice

/usr/local/lib/python3.9/site-packages/cython_sounddevice-0.0.1-py3.9-macosx-10.15-x86_64.egg/cysounddevice/__init__.py in <module>
----> 1 from . import types, devices, streams, buffer
      2 PortAudio = devices.PortAudio

ImportError: dlopen(/usr/local/lib/python3.9/site-packages/cython_sounddevice-0.0.1-py3.9-macosx-10.15-x86_64.egg/cysounddevice/devices.cpython-39-darwin.so, 2): Symbol not found: _PaJack_SetClientName
  Referenced from: /usr/local/lib/python3.9/site-packages/cython_sounddevice-0.0.1-py3.9-macosx-10.15-x86_64.egg/cysounddevice/devices.cpython-39-darwin.so
  Expected in: flat namespace
 in /usr/local/lib/python3.9/site-packages/cython_sounddevice-0.0.1-py3.9-macosx-10.15-x86_64.egg/cysounddevice/devices.cpython-39-darwin.so

[shakfu]

I should have read the code before posting, it looks like Jack is indeed required for cysounddevice.

[nocarryr]

I should have read the code before posting, it looks like Jack is indeed required for cysounddevice.

As it is at the moment, yes. The runtime libraries as well as development headers for both portaudio and jack are required.

It seems there is no homebrew package for portaudio-dev (which would include the headers and make them locatable). That also seems to be the case for jack.

For both, I would recommend building from source by following their respective installation guides. (and removing any existing installation files).

MacOS was last on my list to tackle the build process on. I hit such major roadblocks and headaches trying to compile for Windows that I wound up losing all momentum on the project. Hopefully I'll be able to pick it up again soon.

[shakfu]

Thanks for your reply and shedding some light on the issue.

I think you'll probably find the macOS part of the build process (minus the issues highlighted here) much more friendly to someone coming in from linux-land than Windows. I've personally given up on developing on Windows and have switched completely to OS X and Linux, although the constant struggle with codesigning, notarization with OS X incline me to the latter.

S

[nocarryr]

Agreed. Just getting something to compile in Windows is hard enough, but Python extensions are very picky about the specific compiler, version and settings. Add search paths to that with trying to package the lib and headers for everything in a Python wheel and.... well, you see how I lost hope.

I have a few Macs at work which should make things easier, plus GH actions supports linux, mac and win workflows so it shouldn't be a huge undertaking

[shakfu]

Hi,

I saw that you have a branch rtfd-config which does not depends on jack. I managed to get it to build without errors on my mac (x86_64) using brew-installed portaudio after a tweak to the setup.py:

import sys
import os
from setuptools import setup, find_packages
from Cython.Build import cythonize
from Cython.Build.Dependencies import default_create_extension

try:
    import numpy
except ImportError:
    numpy = None

INCLUDE_PATH=['/usr/local/include']
if numpy:
    INCLUDE_PATH.append(numpy.get_include())

LIB_PATH = ['/usr/local/lib']

def build_pa_lib():
    from tools import build_portaudio
    lib_base = build_portaudio.main()
    INCLUDE_PATH.append(str(lib_base / 'include'))
    LIB_PATH.append(str(lib_base / 'lib'))

RTFD_BUILD = 'READTHEDOCS' in os.environ.keys()
if RTFD_BUILD:
    build_pa_lib()
    print('INCLUDE_PATH: ', INCLUDE_PATH)

USE_CYTHON_TRACE = False
if '--use-cython-trace' in sys.argv:
    USE_CYTHON_TRACE = True
    sys.argv.remove('--use-cython-trace')

def my_create_extension(template, kwds):
    name = kwds['name']
    #if RTFD_BUILD:
    if True:
        kwds['library_dirs'] = LIB_PATH
        kwds['include_dirs'] = INCLUDE_PATH
        kwds['runtime_library_dirs'] = LIB_PATH
        print(kwds)
    if USE_CYTHON_TRACE:
        kwds['define_macros'] = [('CYTHON_TRACE_NOGIL', '1'), ('CYTHON_TRACE', '1')]

    return default_create_extension(template, kwds)

ext_modules = cythonize(
    ['cysounddevice/**/*.pyx'],
    include_path=INCLUDE_PATH,
    annotate=True,
    # gdb_debug=True,
    compiler_directives={
        'linetrace':True,
        'embedsignature':True,
        'binding':True,
    },
    create_extension=my_create_extension,
)

setup(
    ext_modules=ext_modules,
)

Looks promising so far. May I ask what you had in mind for this branch?

[nocarryr]

@shakfu that branch was merged in #3

I had to resort to manually compiling portaudio in readthedocs builds instead of installing the packages from the package manager (apt-get). IIRC, it had something to do with dependencies which couldn't be resolved on the container.

The build script can be found here: https://github.com/nocarryr/cython-sounddevice/blob/ef0a0db1221732d928dda2aa051ec4841c98faec/tools/build_portaudio.py

I do remember thinking at the time though that this method should be made available to others to make things easier, but I was only able to test on Ubuntu linux (and Windows, but that's a completely different story).

In order to make the build script more "cross-platformy" I think it'd be better to have it use a user directory for libraries rather than system-wide (so no "sudo" would be needed). Good to know at least though that it works as-is on Mac OS

[nocarryr]

Correction: https://github.com/nocarryr/cython-sounddevice/blob/ef0a0db1221732d928dda2aa051ec4841c98faec/tools/build_portaudio.py#L11

The script is already using a user library directory. At least one that's used in Linux

[shakfu]

Ok I see that you merged it into the main branch which is still has the jack dependency.

For a moment I thought that the rtfd-config branch was an effort to make portaudio host api agnostic, but it looks like it just preceded the jack dependent code.

I'm revisiting my libpd cython code base again so perhaps I'll circle back here and see if I can experiment and incorporate your jack-dependent code as a start.

[nocarryr]

Hmm... I may have a solution to that. If you're really just wanting to skip the jack dependency, I just found a way to inject compile-time variables to avoid it...

Check out this branch and pass "--no-jack" to setup.py.

Note, I've only tested compilation, nothing further at this point

[shakfu]

Thanks, it worked for me (-:

I did have to make the setup.py home-brew friendly so I could compile as follows:

$ HOMEBREW_BUILD=1 python3 setup.py --no-jack install

Here's the modified setup.py file which just adds the home-brew option without disturbing your jack logic:

import sys
import os
from setuptools import setup, find_packages
from Cython.Build import cythonize
from Cython.Build.Dependencies import default_create_extension

try:
    import numpy
except ImportError:
    numpy = None

INCLUDE_PATH=['/usr/local/include']
if numpy:
    INCLUDE_PATH.append(numpy.get_include())

LIB_PATH = ['/usr/local/lib']

def build_pa_lib():
    from tools import build_portaudio
    lib_base = build_portaudio.main()
    INCLUDE_PATH.append(str(lib_base / 'include'))
    LIB_PATH.append(str(lib_base / 'lib'))

HOMEBREW_BUILD = 'HOMEBREW_BUILD' in os.environ.keys()

RTFD_BUILD = 'READTHEDOCS' in os.environ.keys()
if RTFD_BUILD:
    build_pa_lib()
    print('INCLUDE_PATH: ', INCLUDE_PATH)

USE_CYTHON_TRACE = False
if '--use-cython-trace' in sys.argv:
    USE_CYTHON_TRACE = True
    sys.argv.remove('--use-cython-trace')

COMPILE_TIME_ENV = {'CYSOUNDDEVICE_USE_JACK': True}

USE_JACK_AUDIO = True
if '--no-jack' in sys.argv:
    USE_JACK_AUDIO = False
    sys.argv.remove('--no-jack')
    COMPILE_TIME_ENV['CYSOUNDDEVICE_USE_JACK'] = False

def my_create_extension(template, kwds):
    name = kwds['name']
    if RTFD_BUILD or HOMEBREW_BUILD:
        kwds['library_dirs'] = LIB_PATH
        kwds['include_dirs'] = INCLUDE_PATH
        kwds['runtime_library_dirs'] = LIB_PATH
        print(kwds)
    if USE_CYTHON_TRACE:
        kwds['define_macros'] = [('CYTHON_TRACE_NOGIL', '1'), ('CYTHON_TRACE', '1')]

    return default_create_extension(template, kwds)

ext_modules = cythonize(
    ['cysounddevice/**/*.pyx'],
    include_path=INCLUDE_PATH,
    annotate=True,
    # gdb_debug=True,
    compiler_directives={
        'linetrace':True,
        'embedsignature':True,
        'binding':True,
    },
    create_extension=my_create_extension,
    compile_time_env=COMPILE_TIME_ENV,
)

setup(
    ext_modules=ext_modules,
)

[nocarryr]

@shakfu glad that worked. The build process definitely needs some work and there are other compile-time options that would be helpful to add in, but that would be better as a separate issue/PR for a later time.

Thanks!