[BUG]: Import error on Intel Mac / Ventura 13.5

[jackparmer]

Description of the bug

We've noticed this import error upon from tm_devices import DeviceManager on Intel core I5 Macs with the Ventura 13.4 operating system

Error Domain=KMErrorDomain Code=71 "Insufficient permissions for action: Unable to load 'com.highpoint-tech.kext.HighPointIOP' in unprivileged request." UserInfo={NSLocalizedDescription=Insufficient permissions for action: Unable to load 'com.highpoint-tech.kext.HighPointIOP' in unprivileged request.}

Steps To Reproduce

1. Obtain a Mac with an Intel chipset
2. pip install tm_devices
3. Run python3 to start the Python CLI
4. In the Python CLI, run from tm_devices import DeviceManager

Environment Information

Python 3.11

Additional Information

No response

[nfelt14]

Is there a Python traceback when this happens?

This could help us understand what code is causing this issue.

[jackparmer]

Hi Nicholas!

Sadly, there's no traceback, only the Error Domain=KMErrorDomain Code=71 "Insufficient permissions ... error that I posted above 😿

I cloned the repo and am poking around this file https://github.com/tektronix/tm_devices/blob/main/src/tm_devices/device_manager.py Please let me know if I can run anything in the Python CLI that might help troubleshoot.

I also tried what was suggested in this SO post (granting NI's developer ID access Mac kernel extensions), but no dice.

[kidpyg]

I've got an error, and I was told it was related. My Mac is on Ventura 13.6.1

Obtain a Mac with an Intel chipset Run Mac installer package Open Flojoy studio

Python 3.11 is installed

[adametry]

Do you have a VISA driver installed (e.g., NI-VISA, PyVISA-py)?

[kidpyg]

I do not--I just followed this:

And installed miniconda for my Mac from the second here link:

Do I need one?

EDIT: I searched and I guess I do have some? I attached a screenshot if this helps.

[adametry]

kidpyg, let's investigate your Flojoy Studio error as a separate Issue to avoid overloading this thread. If these errors are related, we can note that in either Issue.

[adametry]

@jackparmer, do you see the same error if you replace step 4 with import pyvisa as visa?

[jackparmer]

Hi Adam!

So,

import pyvisa as visa
rm = visa.ResourceManager('@py')

... works fine for me (the "Python native PyVISA backend").

However,

import pyvisa as visa
rm = visa.ResourceManager()

... results in the same MacOS kernel extensions error that I reported above.

In poking around, device_manager.py, I noticed that L796 instantiates a VISA ResourceManager without explicitly specifying the VISA backend. It seems that setting the __visa_library property to @py would fix the issue for me, but it does not seem like __visa_library has been plumbed into the global config manager yet (it's currently set to NotImplemented).

I created the YAML config file and set standalone to true as described in the docs here, but it did not seem to have an effect.

Please let me know if there are more ways that I can help troubleshoot!

[adametry]

That's very helpful!

It sounds like PyVISA has found a backend IVI-VISA library installed on your machine, which is throwing the exception once it attempts to load the HighPoint IOP kernel ext. Do you have a RAID device attached perhaps?

As an alternative to configuring standalone, try setting an env var with export TM_DEVICES_VISA_LIBRARY="@py" before step 3, and see if the exception persists. Meanwhile, I'll look into why the config may not have loaded...

[nfelt14]

@jackparmer, one other thing to try would be commenting out this section of code: functions.py#L32-L40

The get_available_devices() method is not run on initialization, so that shouldn't be what is causing the issue. However, the code from helpers/functions.py is executed on startup, it runs a function imported from a third-party library that may be causing the issue.

[nfelt14]

@adametry, the reason the config setting wouldn't have an effect is because that code creating the ResourceManager isn't actually executed on import or initialization, so the config setting is never actually used.

[nfelt14]

@jackparmer, one other thing to try would be commenting out this section of code: functions.py#L32-L40

The get_available_devices() method is not run on initialization, so that shouldn't be what is causing the issue. However, the code from helpers/functions.py is executed on startup, it runs a function imported from a third-party library that may be causing the issue.

@jackparmer, you will need to only comment out lines 36 and 40, if you comment out the other lines things will definitely break.

[nfelt14]

@jackparmer, one other thing to try would be commenting out this section of code: functions.py#L32-L40 The get_available_devices() method is not run on initialization, so that shouldn't be what is causing the issue. However, the code from helpers/functions.py is executed on startup, it runs a function imported from a third-party library that may be causing the issue.

@jackparmer, you will need to only comment out lines 36 and 40, if you comment out the other lines things will definitely break.

There may be a warning about GPIB libraries missing, but that is fine.

[nfelt14]

@jackparmer, If commenting out the above lines doesn't work, you could also try to comment out functions.py#L45. This is the only other call that I can find/remember that accesses VISA stuff when things are imported.

[jackparmer]

Thanks guys!

As an alternative to configuring standalone, try setting an env var with export TM_DEVICES_VISA_LIBRARY="@py" before step 3, and see if the exception persists. Meanwhile, I'll look into why the config may not have loaded...

@adametry I tried this, but am still getting the same error on importing tm_devices (eg, the IVI-VISA backend is still being loaded, rather than @py backend). So it seems this env var may not be fully plumbed in yet.

Do you have a RAID device attached perhaps?

I think that the "RAID" element of the error may a red herring. There's never been a RAID device or software installed on this machine. Furthermore, if I start the Python CLI with sudo, then enter import tm_devices, I get the same error that references 2 NI drivers rather than a RAID extension:

Jacks-MacBook-Air:~ jackparmer$ sudo python3.11
Python 3.11.6 (v3.11.6:8b6ee5ba3b, Oct  2 2023, 11:18:21) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
>>> import traceback
>>> try:
...     from tm_devices import DeviceManager
... except:
...     print(trace.format_exc())
... 
Error Domain=KMErrorDomain Code=27 "Extension with identifiers com.ni.driver.NiViPciK,com.ni.driver.ni488k,com.ni.driver.nipalk not approved to load. Please approve using System Preferences." UserInfo={NSLocalizedDescription=Extension with identifiers com.ni.driver.NiViPciK,com.ni.driver.ni488k,com.ni.driver.nipalk not approved to load. Please approve using System Preferences.}
Abort trap: 6
Jacks-MacBook-Air:~ jackparmer$ 

As background, Flojoy users began encountering this error on certain machines (only some MacOS Intel machines so far) because we're importing tm_devices into Flojoy's Python runtime at startup (since last Friday's release).

If tm_devices were configurable to load the PyVISA resources manager as rm = visa.ResourceManager('@py') (rather than rm = visa.ResourceManager()), that would be a sufficient fix. Beyond Flojoy, at the moment, any machine that has an issue with the IVI-VISA PyVISA backend will fail on import tm_devices. From the docs and the code, it looks like configurability of the PyVISA backend was part of the original design, but may not have been fully integrated.

Anyway, I'll try commenting out the lines that Nicolas mentioned this evening and report back what I learn. Thanks for you help!

[nfelt14]

As I mentioned in a previous comment, the code that opens a ResourceManager object is not run when things are imported. It respects the configured VISA backend (from the env var or config settings). Since this code is not run when tm_devices is imported, something else must be triggering this error. My guess is that one of the sections I mentioned from helpers/functions.py is causing the issue since those are executed when tm_devices is imported.

[jackparmer]

You're right! If I comment out lines #L32-L40, I no longer get the error originally reported and the Python session remains open.

If it's possible to patch this, we'd like to keep tm_devices in the Flojoy distribution. We're currently using it to add first-class support for Tek's product line (here for example, https://github.com/flojoy-ai/blocks/pulls).

Please let me know how I can help troubleshoot further!

[nfelt14]

Does commenting out just line 45 make a difference?

[nfelt14]

If we can narrow this down to a single line, that would be great. Some of those lines are imports that are currently needed for other things.

[jackparmer]

Yep! Commenting out only line 45 results in a working tm_devices import with no error. Nice sleuthing! I'll be available if there's anything else you'd like me to test.

[nfelt14]

Since this seems to only be happening on systems with System Integrity Protection enabled, we will need a way to reliably detect if that is enabled. (We have had reports of Macs that do not show this issue, presumably they don't have System Integrity Protection enabled.)

@jackparmer, can you post the output of the csrutil status command? We will probably need to use that command to detect if System Integrity Protection is enabled. (Based on issue 1555478 from the NI-VISA known issues page.) It appears that we won't be able to use python code to catch the exception being thrown, so we need to detect the status before we call the function causing the crash.

[jackparmer]

Sure thing. Here is the output of csrutil status on my machine:

Jacks-MacBook-Air:~ jackparmer$ csrutil status
System Integrity Protection status: enabled.
Jacks-MacBook-Air:~ jackparmer$ 

[nfelt14]

@jackparmer, I raised #109 to hopefully fix the issue. If you want, you could check out the branch I made the change on (nfelt14/fix-import-error-on-mac-with-system-integrity-protection) to validate that it works. Or you can just wait for it to be merged to the main branch and test it out there.

[nfelt14]

I re-opened this since the changes have not been released yet, just merged to the main branch.

[nfelt14]

@jackparmer the changes merged to main. If you could test them out and let me know if it works that would be great.

[nfelt14]

@jackparmer have you been able to test the latest updates on the main branch?

[nfelt14]

@jackparmer, @kidpyg, I went ahead and released v1.0.1, please let me know if this fixes the issue for you.