Macbook M1, serial issue

[theoroborus]

Hello guys. I'm a Macbook M1 user. I've been able to install and run the app using rosetta and Azul ARM Java build. The serial doesn't seem to be working. Led is red with green every 3 secs. However in my case plugging the oscilloscope after starting the software doesn't solve the issue. From terminal with "ioreg -p IOUSB" I get:

at com.owon.uppersoft.vds.core.usb.CDevice.init(CDevice.java:195) at com.owon.uppersoft.vds.core.usb.CDevice.getDevices(CDevice.java:209) at com.owon.uppersoft.dso.source.usb.USBSourceManager.refreshUSBPort(USBSourceManager.java:79) at com.owon.uppersoft.dso.source.comm.detect.USBLoopChecker.checkUSBDevice(USBLoopChecker.java:74) at com.owon.uppersoft.dso.source.comm.USBDaemonHelper.onNotConnecting(USBDaemonHelper.java:35) at com.owon.uppersoft.vds.machine.InfiniteDaemonTiny0.onNotConnecting(InfiniteDaemonTiny0.java:30) at com.owon.uppersoft.dso.source.comm.Flow.run(Flow.java:74) at com.owon.uppersoft.dso.global.ControlAppsTiny$1.run(ControlAppsTiny.java:38) java.lang.NoClassDefFoundError: Could not initialize class ch.ntb.usb.LibusbJavajava.lang.NoClassDefFoundError: Could not initialize class ch.ntb.usb.LibusbJava

[florentbr]

Since you are using a Mac M1, my guess is that you needs an "arm64" version of libusb which is not included in this project. To make it work, you'll have to find or build yourself libusb for mac arm and place it in lib/mac/arm64/ so it can be properly installed. Strange, didn't the install script install-mac.sh mentioned "Architecture not supported: arm64" ?

[theoroborus]

I managed to I install it using a rosetta-based version of the terminal. That's why ! :) Can be done by launching the terminal with rosetta, or duplicating it and checking the rosetta option in the properties. So I tried copying libiusb from /opt/local/lib and pasting in in lib/mac/arm64/ inside the freshly downloaded OWON-VDS1022 install folder. I relaunched the install-mac.sh script from a rosetta terminal. Same issues as before. I retried from a non-rosetta terminal : same issue. However now, using "ioreg -p IOUSB", I'm not seeing any issue with the Owon device, but it's also not being displayed as an owon device in the list of connected devices ; " | | +-o U@01144420 <class IOUSBHostDevice, id 0x100002baa, registered, !matched, active, busy 1 (107 ms), retain 19>"

[florentbr]

For your platform, the app requires all the following librarires compilled as arm64 : libusb-0.1.4.dylib libusb-1.0.0.dylib libusbJava.jnilib

The lib librxtxSerial.jnilib can be ignored. It's loaded at some point but it's not used by the app. The native libusb-0.1.4.dylib and libusb-1.0.0.dylib are available at Homebrew : https://formulae.brew.sh/formula/libusb-compat#default

As for libusbJava.jnilib, you'll have to build the Java jni yourself : https://sourceforge.net/projects/libusbjava/

I don't own a Mac M1, so I can't build or test any of them.

[theoroborus]

Thanks. I had already done that from your previous post. I installed libusb with homebrew and. simply copy/pasted these files in "lib/mac/arm64/" from the libusb directory found in "/opt/local/lib". For some reason "libusbJava.jnilib" was already in the folder.

[florentbr]

Java ARM can only load a libusbJava.jnilib compiled for arm64. The one you found is likely compiled for amd64.

[florentbr]

The source to build libusbJava.jnilib for amr64: LibusbJava.zip

[jpangburn]

For M1 users, I didn't try to get it going using arm64 (because I hadn't read this page first so hadn't seen the author's suggestion to compile LibusbJava.zip), instead I used a Java x86_64 installation which I had installed for another application. To do this, it's super easy, just edit /Applications/OWON VDS1022 Oscilloscope.app/Contents/MacOS/owon-vds-tiny. Look for this line at the top /usr/libexec/java_home --exec java\ and change it to /usr/libexec/java_home -a x86_64 --exec java\. Save the file. Then just launch the app normally (double click on the application icon).

I used Amazon's Corretto 11 x86_64 JDK, but probably any x86_64 JRE would work. Obviously, I have Rosetta 2 installed so x86_64 stuff can run but anyone reading this probably knows that.

[jpangburn]

I tried to make an M1 version of libusbJava.jnilib, but didn't have any luck. The LibusbJava.zip that you linked @florentbr is not the same project as found at https://sourceforge.net/projects/libusbjava/. Which one is the right one?

The zip file seems to have an Ant target for Mac but the usb.h that it references includes windows.h so seems like it may have been modified to only work on Windows? It also includes poppack.h and another one that I don't know about, and wonder if they're windows specific too.

The project at Sourceforge uses cmake, so I tried that but am too much of a cmake noob to debug the errors that come up. For example, it tries to use javah but that's gone since Java 10 so I don't have that. Converted it to use javac instead, but then there's a linker error. So wondering if I'm on the wrong path anyway since the project is not the same as the zip file you attached?

Also, you can make Universal binaries from an Intel-based Mac https://developer.apple.com/documentation/apple-silicon/building-a-universal-macos-binary. You just can't test the arm64 binary. But you can check them like this:

% lipo libusb-1.0.0.dylib -info
Architectures in the fat file: libusb-1.0.0.dylib are: x86_64 arm64

It lets you see what architectures are provided inside the library/executable.

[florentbr]

@jpangburn

I tried to make an M1 version of libusbJava.jnilib, but didn't have any luck. The LibusbJava.zip that you linked @florentbr is not the same project as found at https://sourceforge.net/projects/libusbjava/. Which one is the right one?

It belongs to the same project on an older branche : https://sourceforge.net/p/libusbjava/code/HEAD/tree/branches/libusb_api_0v1/LibusbJava/

The zip file seems to have an Ant target for Mac but the usb.h that it references includes windows.h so seems like it may have been modified to only work on Windows?

The included usb.h is for Windows only. The one for Linux and Mac is provided by libusb once installed. There's also a release of libusbjava on Linux. The code is slightly different, more recent and likely more suited for Mac: https://packages.debian.org/sid/libusb-java-lib

The project at Sourceforge uses cmake, so I tried that but am too much of a cmake noob to debug the errors that come up. For example, it tries to use javah but that's gone since Java 10 so I don't have that. Converted it to use javac instead, but then there's a linker error. So wondering if I'm on the wrong path anyway since the project is not the same as the zip file you attached?

There's no need to rebuild the jar. If you wish to compile the native library on Mac:

• Intall a JDK, GCC and libusb-compat
• Extract LibusbJava.c and LibusbJava.h from libusb-java_0.8+ztex20090101-9.debian.tar.xz
• Extract usb.h from libusb-compat-0.1.5
• Compile with GCC :

JAVAHOME=$(/usr/libexec/java_home)

gcc -dynamiclib -fPIC -std=c99 -Wall\
 -I "$JAVAHOME/include"\
 -I "$JAVAHOME/include/darwin"\
 -Wl,-rpath,@loader_path/\
 -o libusbJava.dylib\
 LibusbJava.c\
 -l:libusb-0.1.4.dylib

It should create a new libusbJava.dylib. I don't have a mac to test it so you may have to tweek the flags and dependencies. The line -Wl,-rpath,@loader_path/\ tells the library to look for libusb-0.1.4.dylib in the same folder instead of the system folder. It can be ignored if necessary.

[jpangburn]

@florentbr

Thank you! These were the clues I needed. The one I was using with cmake had java code in a package /ch/ntb/inf/ but the calling Java code from OWON was looking for /ch/ntb/usb/. I had gotten the cmake to generate the jnilib eventually, but it couldn't find the right class and I realized this was why. So I had given up figuring I only use the scope once in a while, who cares if it runs on x86_64 (well me, but who else?)

There's no need to rebuild the jar. If you wish to compile the native library on Mac:

• Intall a JDK, GCC and libusb-compat
• Extract LibusbJava.c and LibusbJava.h from libusb-java_0.8+ztex20090101-9.debian.tar.xz
• Extract usb.h from libusb-compat-0.1.5
• Compile with GCC :

JAVAHOME=$(/usr/libexec/java_home)

gcc -dynamiclib -fPIC -std=c99 -Wall\
 -I "$JAVAHOME/include"\
 -I "$JAVAHOME/include/darwin"\
 -Wl,-rpath,@loader_path/\
 -o libusbJava.dylib\
 LibusbJava.c\
 -l:libusb-0.1.4.dylib

These were the big hints. I had libusb-compat from Homebrew already, because I had compiled libusb but it was a hassle then I saw the compat one on Homebrew and just used that when I tried my failed compilation yesterday (it was /ch/ntb/inf/*). If anyone else feels like it's necessary to do this, once you have libusb and libusb-compat installed from Homebrew on M1 (it uses a different Homebrew path, not /usr/local), you need to do the following:

1. Get the libusbjava source from source, and extract LibusbJava.c and LibusbJava.h as florentbr said.
2. No need to copy any other files to that directory because Homebrew already put the necessary stuff in its paths, so just run:

   gcc -dynamiclib -fPIC -std=c99 -Wall\
   -I "$JAVA_HOME/include"\
   -I "$JAVA_HOME/include/darwin"\
   -I "/opt/homebrew/include"\
   -L "/opt/homebrew/lib"\
   -o libusbJava.dylib\
   LibusbJava.c\
   -lusb-0.1.4 

3. Copy the resulting libusbJava.jnilib into the OWON app over the old one. Do the same for libusb-0.1.4.dylib and libusb-1.0.0.dylib from /opt/homebrew/lib. Then you're ready to ride.

If I now run it, it's running natively on M1:

I thought maybe you might want to distribute universal binaries, so I used lipo to combine mine with yours into fat binaries. If I modify the shell script as I mentioned above with -a x86_64 then it runs under Rosetta 2 in Intel mode (although it does burn more CPU, hovering in high 40s low 50s instead of mid 30s natively):

This way people can just run the app without having to worry about what processor they have, or if they have M1 they don't have to worry about which JVM architecture runs when the script calls /usr/libexec/java_home -exec. Whichever JVM runs will load the matching architecture from the universal binaries. I'm happy to submit a pull request with these binaries if you like.

Lastly- why do you bother maintaining a project to support this scope on Mac if you don't have a Mac? :-) (EDIT: never mind, I saw this supports linux too, that's what you're using it on I guess).

EDIT: I see that you don't need universal binaries because your install script is smart enough to check architecture. So I just added an "arm64" directory and copied the arm64 only binaries into there (except that librxtxSerial.jnilib which I don't have), and made no other changes. Ran the install-mac.sh script and it works great.

[florentbr]

@jpangburn

Congrats!

I'd rather keep away from the Universal binaries. It will complicate things if one needs to relink or swap a dependency.

Ideally, a user would just install a JRE and the installer would select the matching architecture and copy libusbJava.dylib libusb-0.1.4.dylib libusb-1.0.0.dylib from lib/mac/arm64 or lib/mac/arm64.

If I were to add Universal binaries, it would be in a separated folder like lib/mac/universal and prompt for option during installation.

You can either send me the files or make a pull request. Just make sure that the links in libusbJava.dylib libusb-0.1.4.dylib points to @loader_path : https://matthew-brett.github.io/docosx/mac_runtime_link.html

It needs to work without libusb-compat installed.

As for librxtxSerial, I removed it enterely from this project since it's not even used.

Lastly- why do you bother maintaining a project to support this scope on Mac if you don't have a Mac? :-) ...

I had a Mac at some point but it died.

[jpangburn]

@florentbr

Probably you didn't notice the edit I made to my comment at the end, I realized that your installer was checking architecture and universal binary was pointless when I was looking at the install script in order to test it. So yes, totally agree, no universal binary.

Wow, that mac runtime link is something I never had any clue about, as I write code in a variety of languages but rarely in C/C++. What a pain. And yes, mine were done wrong so when I uninstalled libusb-compat then the app doesn't start any more because it ignores the libusb-0.1.4.dylib in the same directory.

So I did % sudo install_name_tool -change /opt/homebrew/opt/libusb-compat/lib/libusb-0.1.4.dylib @loader_path/libusb-0.1.4.dylib libusbJava.dylib.

% otool -L libusbJava.dylib
libusbJava.dylib:
    libusbJava.dylib (compatibility version 0.0.0, current version 0.0.0)
    @loader_path/libusb-0.1.4.dylib (compatibility version 9.0.0, current version 9.4.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

With this done I could brew uininstall libusb-compat and the app could still start. Here's where I run into trouble- the libusb-compat file libusb-0.1.4.dylib looks like this:

% otool -L libusb-0.1.4.dylib 
libusb-0.1.4.dylib:
    /opt/homebrew/opt/libusb-compat/lib/libusb-0.1.4.dylib (compatibility version 9.0.0, current version 9.4.0)
    /opt/homebrew/opt/libusb/lib/libusb-1.0.0.dylib (compatibility version 4.0.0, current version 4.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

OK so it has a path to itself, which according to that document doesn't matter for our situation. The problem is that it has /opt/homebrew/opt/libusb/lib/libusb-1.0.0.dylib for libusb-1.0.0.dylib, so it means libusb-1.0.0.dylib has to be at that location. No big deal, right, just another name change?

% sudo install_name_tool -change /opt/homebrew/opt/libusb/lib/libusb-1.0.0.dylib @loader_path/libusb-1.0.0.dylib libusb-0.1.4.dylib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: libusb-0.1.4.dylib

So it says it invalidated the signature. When I try to start the application I get EXC_BAD_ACCESS and it crashes. I imagine that's from the jnilib calling the compat lib and the compat lib doesn't have a valid signature?

In any case, I'm not sure of the best way to fix this. I suppose I could either compile libusb-compat myself so I could fix the @loader_path without this error (potentially difficult), or maybe better yet is to fix the Homebrew formula so it doesn't get that stamped in there in the first place (no homebrew editing experience yet). Any suggestion?

[florentbr]

@jpangburn Open libusb-0.1.4.dylib with an hex editor. You'll see that it's not /opt/homebrew/opt/libusb-compat/lib but @@HOMEBREW_PREFIX@@/opt/libusb-compat/lib/. So maby with sudo install_name_tool -change @@HOMEBREW_PREFIX@@/opt/libusb-compat/lib/libusb-1.0.0.dylib @loader_path/libusb-1.0.0.dylib .

If it doesn't work, it's also possible to rewrite the path with en hex editor as long as the new path is shorter than original. I can do it if you are not confortable with this kind of hack.

[jpangburn]

When I opened it, it looks like this:

It didn't have the @@HOMEBREW_PREFIX@@ unfortunately. I edited it anyway to use @loader_path.

Checked it with otool -L but it looks like that messed up the link to libusb-1.0.0:

% otool -L libusb-0.1.4.dylib 
libusb-0.1.4.dylib:
    /opt/homebrew/opt/libusb-compat/lib/libusb-0.1.4.dylib (compatibility version 9.0.0, current version 9.4.0)
     (compatibility version 4.0.0, current version 4.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

Tried it anyway, predictably it crashed :-) I imagine with the hex editor you need to put something in front of the path but I can't tell what. I zipped and attached the libusb-compat that I get from homebrew, in case you see what I'm missing? libusb-0.1.4.dylib.zip

[florentbr]

@jpangburn

Checked it with otool -L but it looks like that messed up the link to libusb-1.0.0:

A string is loaded from an address. So you need to overwrite at the first character and then pad with 00 (null character). In your screenshot, you padded with 00 at the begining when it should have been at the end.

It didn't have the @@HOMEBREW_PREFIX@@ unfortunately.

The reason I have @@HOMEBREW_PREFIX@@ is because I downloaded libusb-compat directly without brew. The downloaded package should be in ~/Library/Caches/Homebrew.

So it says it invalidated the signature. When I try to start the application I get EXC_BAD_ACCESS and it crashes. I imagine that's from the jnilib calling the compat lib and the compat lib doesn't have a valid signature?

I've tracked down the linking with brew. When brew first downloads libusb-compat, the linking is prefixed with @@HOMEBREW_PREFIX@@. It is then renamed using ruby-macho during the installation and not with install_name_tool. It could be that ruby-macho did something to the library. Or maby it's an issue between amd64 and arm64 on your hand.

Anyway, I found an old VM with MacOS. It kept crashing, but I managed to relink and build without the issue you described :

# libusb-1.0.0.dylib
install_name_tool -id @rpath/libusb-1.0.0.dylib libusb-1.0.0.dylib
install_name_tool -add_rpath @loader_path libusb-1.0.0.dylib

# libusb-0.1.4.dylib
install_name_tool -id "@rpath/libusb-0.1.4.dylib" libusb-0.1.4.dylib
install_name_tool -change "@@HOMEBREW_PREFIX@@/opt/libusb/lib/libusb-1.0.0.dylib" "@rpath/libusb-1.0.0.dylib" libusb-0.1.4.dylib
install_name_tool -change "/opt/homebrew/opt/libusb/lib/libusb-1.0.0.dylib" "@rpath/libusb-1.0.0.dylib" libusb-0.1.4.dylib
install_name_tool -add_rpath "/opt/homebrew/opt/libusb/lib" libusb-0.1.4.dylib
install_name_tool -add_rpath "@loader_path" libusb-0.1.4.dylib

# libusbJava.dylib
gcc -dynamiclib -fPIC -std=c99 -Wall\
 -I "/System/Library/Frameworks/JavaVM.framework/Versions/A/Headers"\
 -I "$JAVA_HOME/include"\
 -I "$JAVA_HOME/include/darwin"\
 -I .\
 -Wl,-rpath,/opt/homebrew/opt/libusb-compat/lib\
 -Wl,-rpath,@loader_path\
 -install_name "@rpath/libusbJava.dylib"\
 -current_version 0.2.4\
 -compatibility_version 0.2.4\
 -o libusbJava.dylib\
 LibusbJava.c\
 libusb-0.1.4.dylib

Note that is has to be executed in this specific order and only once for the install_name_tool -add_rpath. I added 2 search paths in libusb-0.1.4.dylib and libusbJava.dylib. It will first look for libusb in /opt/homebrew/opt/libusb-compat/lib and fall back to the loading folder (@loader_path). This way, if the provided libusb-1.0.0.dylib fails, it can be fixed by installing libusb-compat from brew.

The source to build for arm64 if you want to give it a try : libusbJava-arm64-src.zip

[jpangburn]

@florentbr Ah, zero padding at the end makes a lot of sense, should have thought of that! Thanks again for all your help, super generous of you to do all this without using a mac to run your scope. I tried the zero padding at the end because I was curious:

But it didn't work. Oh well, would have been nice to be able to fix things like that.

For install_name_tool, I tried copying out of ~/Library/Caches/Homebrew but I still get the /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: libusb-0.1.4.dylib error. I get the same error when I tried changing the version in your zip file. Seems like this is a known issue with arm64?

It doesn't complain when I make changes using install_name_change to the libusbJava.dylib that I compiled before, maybe because it was compiled on my machine? So after running your changes on the two libusb* libs, I re-signed them like codesign -f -s - libusb-0.1.4.dylib and codesign -f -s - libusb-1.0.0.dylib. This seems to work! I verified by uninstalling libusb-compat and renaming my homebrew's libusb-1.0.0.dylib (can't uninstall from brew because it's a dependency on a bunch of things I use) and it still runs.

I wasn't sure if it mattered, but I recompiled libusbJava.dylib from the zip file you sent, though on my system there's no Headers directory under /System/Library/Frameworks/JavaVM.framework/Versions/A/. I assumed that was just looking for jni.h which is in my $JAVA_HOME/include, so it works anyway. This version also works.

To verify, I did otool -L on all three libs and it's just loading them via rpath:

jpangburn@Jesses-MBP-M1 MacOS % otool -L libusbJava.dylib
libusbJava.dylib:
    @rpath/libusbJava.dylib (compatibility version 0.2.4, current version 0.2.4)
    @rpath/libusb-0.1.4.dylib (compatibility version 9.0.0, current version 9.4.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
jpangburn@Jesses-MBP-M1 MacOS % otool -L libusb-0.1.4.dylib 
libusb-0.1.4.dylib:
    @rpath/libusb-0.1.4.dylib (compatibility version 9.0.0, current version 9.4.0)
    @rpath/libusb-1.0.0.dylib (compatibility version 4.0.0, current version 4.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
jpangburn@Jesses-MBP-M1 MacOS % otool -L libusb-1.0.0.dylib 
libusb-1.0.0.dylib:
    @rpath/libusb-1.0.0.dylib (compatibility version 4.0.0, current version 4.0.0)
    /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
    /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1853.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

I'll upload these to my fork and send you a pull request, unless there's anything else you think I should check?

[florentbr]

@jpangburn

I tried the zero padding at the end because I was curious: ... But it didn't work. Oh well, would have been nice to be able to fix things like that.

I used this method to change the paths in the both .dylib from the current release in x86_64. Note that there're two paths to overrite in the file.

It doesn't complain when I make changes using install_name_change to the libusbJava.dylib that I compiled before, maybe because it was compiled on my machine?

Well, either it's because it's properly signed or because it's not signed. You could also try codesign --remove-signature libusb-0.1.4.dylib before calling install_name_tool. Running and comparing with oTool -l (lower case l) may give you a hint. Maby the ones from macport are not signed?

In any case, as you discovered, the issue is related to code signing. It's stange that it's not possible to change the paths and keep the signature.

I wasn't sure if it mattered, but I recompiled libusbJava.dylib from the zip file you sent, though on my system there's no Headers directory under /System/Library/Frameworks/JavaVM.framework/Versions/A/.

It's the location of jni.h on old Mac.

The .dylib in the zip are from brew arm64_big_sur :

# https://formulae.brew.sh/api/formula/libusb-compat.json
curl -fsSL -H "Authorization: Bearer QQ==" "https://ghcr.io/v2/homebrew/core/libusb-compat/blobs/sha256:8e54f4e98a9dd9e39c8d18e053729472c23a20f35c858fefdc78c8aa6552368f" > libusb-compat.gzip

# https://formulae.brew.sh/api/formula/libusb.json
curl -fsSL -H "Authorization: Bearer QQ==" "https://ghcr.io/v2/homebrew/core/libusb/blobs/sha256:1c40f64450705461a5373c3d54257e646d39914d44bffaf9d957bbe063db2129" > libusb.gzip

If think it's preferable to use the ones from arm64_big_sur since the older the more compatible.

I'll upload these to my fork and send you a pull request, unless there's anything else you think I should check?

Yes, you can push the files.

[florentbr]

@jpangburn

I checked the source from brew and ruby-macho is likely signing the .dylib with :

codesign --sign - --force --preserve-metadata=entitlements,requirements,flags,runtime filename

[jpangburn]

@florentbr I created a pull request with those 3 files under lib/mac/arm64 so it'll pull automatically from install-mac.sh.

@BorisDess if you want to test my fix to make sure it works on a random M1 machine (and not just mine), you could test from my fork if you like (you'll need an arm64 JVM of course which should be the default when you do /usr/libexec/java_home):

git clone https://github.com/jpangburn/OWON-VDS1022.git
cd OWON-VDS1022
sudo bash install-mac.sh

You could delete your copy of the app first (or move it away if you want to save it), but I think the installer deletes any existing app of the same name.

I'll be deleting my fork shortly after @florentbr accepts this pull request though, so it won't be around at a much later date.

[florentbr]

Fixed in release 1.1.1-cf13