Notes / Experience with usage on MacOS M1 Arm machines

[tomb50]

Hello,

I thought I would document some notes for the benefit of anyone else looking to use this library on Mac M1 machines, as I encountered a couple of issues (with all of the engines in fact) and it took a while to understand what is going on in each case - some may be more relevant than others depending on your use case.

Use Case

For context - we already successfully use Graphviz-java in our project, we have it as part of an IntelliJ plugin for Jetbrains MPS IDE. We render graphs within the IDE editor pane ( onto Swing/AWT components). We also use the library separately as part of a Maven plugin to generate SVGs as part of our build process. We work on Linux and Windows machines and looking to move to Mac OS.

Graphviz Engine Recap

As described in this project's read.me Graphviz-java attempts to use a set of (potentially supported) engines until it finds one that is supported - before submitting the request to generate the graph.

These engines (ignoring Graal) are in the order of:


1. Native Graphviz - Graphviz-java will look for a 'dot' executable on the host machine and execute via CLI/ProcessBuilder
2. Javascript Graphviz + V8 Engine - Graphviz-java will use a (bundled) javascript implementation of Graphviz with a bundled version of the V8 Javascript Java binding library
3. Javascript Graphviz + Nashorn Engine - Graphviz-java will use the bundled javascript Graphviz implementation and execute it on the JDK provided Nashrorn Javascript Java binding platform.

For our use case, we do not mandate ‘dot’ executable on the host machine, our previous machines were all built with option 2, the provided v8 engine (as probably most do).

MacOS + M1 Experience

V8

Running our app on Mac M1 we immediately see that option 2 (v8) is not working, a quick google shows that v8 does not have MacOS support, see https://github.com/eclipsesource/J2V8/issues/556. Anoying but no worries we will try option 3 (Nashorn)

Nashhorn

So we then look to see if we can instead use Option 3, the Nashorn Engine, this is not ideal since it is deprecated for removal in JDK15, but lets still try to get it working.

The first thing we notice is that when attempting to invoked the Nashorn Engine, we are getting an error:

ClassNotFoundException for PromiseException

The PromiseException class is provided as a transitive dependency to Graphviz-java, originating from this repo:

https://github.com/hidekatsu-izuno/nashorn-promise



The class is definitely on the classpath, the issue here is in fact a class loading issue due to a combination of:

1. Running Graphviz-java as part of an IntelliJ Plugin - which provides a specific Classloader for the classes loaded by Graphviz (including PromiseException).
2. An (arguable) bug in Nashorn, it has an assumption that it should be looking up classes from the ContextClassLoader of the executing thread, rather than the classLoader that was provided to the NashornScriptFactory constructor. 



This scenario was actually hinted at a few years ago
, see https://github.com/nidi3/graphviz-java/issues/144

. However, this is not entirely sufficient in my use, in order to get this working a further change would be needed. This could be done either in our code or within Graphviz-java.

Full detail of the classloading issue

The classloaders in our use case are as follows:

1. BootClassLoader
2. jdk.internal.loader.ClassLoaders$PlatformClassLoader
3. com.intellij.util.lang.PathClassLoader
4. jetbrains.mps.classloading.ModuleClassLoader

PromiseException is loaded by ModuleClassLoader.

Graphviz is ultimately called by the following thread stack

1. AWT-EventQueue - MPS editor / AWT
2. _SwingWorke_r (Async AWT)
3. Thread-x (created by Graphviz Async)

The parent AWTEventQueue (and subsequent child threads) inherit contextClassLoader of PathClassLoader

Nashorn, therefore, uses this.
, and is unable to find the PromiseException Class, which was loaded by the child ModuleClassLoader

Assuming there is no "fix" to Nashorn, the only thing we can do is set the contextClassLoader to the classloader that loaded PromiseException prior to instantiating the Nashorn ScriptEngine.

I was initially hesitant about doing this in our client code (mutating the contextClassLoader of the AWT-EventDispatchThreads/SwingWorkers or manually creating new threads). So instead looked to do the switch as close to Nashorn as possible, within Graphviz

-java. This can be seen in the below commit, it doesn’t feel great but works - happy to push for PR if appropriate

https://github.com/tomb50/graphviz-java/pull/1/commits/74ecdebc1aa8d279ca9ddee0ef0e5dff086802f7



With this change, the engine subsequently works as expected.

Native

Even though the above works, we would probably prefer to not have a code change so review option 1 (native) - plus Nashorn is slow.

We finally resort to option 1, having the dot executable available

 on the host machine

Downloaded dot through home-brew, using brew install graphviz it installed fine

When running our IDE plugin, however, Graphviz-java still did not find the dot executable. After a bit of digging it is due to the relationship and differences in environment variables when launching applications through a Desktop launcher vs from a shell.

Applications on Mac do not inherit the env variables that may otherwise be defined in common profiles (.zshrc etc) - this includes the PATH variable. Graphviz-java needs to know about PATH in order to find the executable.

In this case, because dot is advised to be downloaded by brew, it is NOT in the default PATH (/bin, /usr/bin, /sbin, /usr/sbin) it is in the brew Cellar directories).

 This problem is detailed at more length online, it’s described here with some suggestions for a full solution needing to build a custom app launcher (urgh).

https://korban.net/posts/2019-07-23-launching-mac-os-application-with-custom-path/

However, it is possible to provide the correct environment variables if the application is started from the terminal, so by starting IntelliJ/MPS from the command line, our plugin that calls Graphviz-java then provides the full PATH variable (which includes brew cellar directories) and it does find dot and everything works as expected





To summarise

 (TLDR)

v8 is not supported for MacOS

Nashorn works but there are issues for certain use cases

• It will error if the ClassLoader that loaded Graphviz is a child of the ClassLoader that is the ContextClassLoader of the thread that calls Graphviz (which can happen).
• Workaround suggestion is above for certain use cases
• Also it would be possible to apply a workaround in the client code by ensuring the Thread that calls Graphviz has an appropriate CCL

  Native works, but MacOS has subtleties around the required PATH variable

• Ensure your app is launched as a process that inherits the correct PATH variable
• It would be nice in the future to be able to provide an argument to GraphViz-Java to provide a location to dot. That way it can be resolved within the Graphviz_java client application rather than enviornment-specific

Hope some of this may be helpful to anyone else who may hit similar challenges in the future

[mgroth0]

@tomb50 thanks for the incredible work.

I feel bad because I was able to get this working on my M1 Mac with fewer issues than you. But maybe something in my build environment was different, I'm not sure.

I'm trying to run https://github.com/vanniktech/gradle-dependency-graph-generator-plugin in IntelliJ. At first, it did not work. And it complained none of the engines loaded. Eventually, I stumbled upon this post of yours. I realized I did not have dot installed on my new machine.

I first tried brew install dot. This doesn't work.

Next, I tried brew install graphviz. This worked. And pretty much right away, the Gradle plugin started working without any additional steps.

I checked which dot in a terminal and got /opt/homebrew/bin/dot. As you suggest, I'd be surprised if this automatically was passed into whatever search path my java environment is using.

I launch IntelliJ through the JetBrains Toolbox. So it really baffles me how I got it to work so easily without the path issues you describe. I hope this might be helpful to you or others.

[tomb50]

Hi @mgroth0 thanks for the notes.

Looks like a typo on my part, I did brew install graphviz too - I'll correct the above post.

Interesting to hear that you were able to get it working without any issues around the PATH

I'm assuming that as your use case is a Gradle plugin when you say you are running it through IntelliJ, it is the Intellij Gradle integration that would be executing a Gradle process through a shell, therefore picking up shell environment variables, which would then be passed to the Java process spawned by Gradle - but please advise if this is not the case.

If it is not the case I don't suppose you would be able to set a breakpoint and see System.getenv() to confirm what PATH looks like prior to graphviz being called. It may invalidate some of my understandings above.

Thanks! Tom

[mgroth0]

@tomb50 , I actually adapted the gradle plugin to work inside of a regular kotlin module (not a gradle plugin, but just a regular gradle subproject built in IntelliJ).

I've taken a look atSystem.getenv() info you requested, right before I use graphviz. And here is the value for PATH:

/Users/matthewgroth/miniconda3/bin:/Users/matthewgroth/miniconda3/condabin:/Users/matthewgroth/bin:/Applications/SublimeText.app/Contents/SharedSupport/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin

I've also tried using echo $PATH in the terminal, and the output was the same.

I checked my ~/.zshrc and it doesn't look like I manually add it there. I guess homebrew automatically adds it to the PATH and my gradle/intelliJ environment somehow is passing it through.

Now here's where it gets interesting. I've only been able to get it working inside of a gradle run task and assumed it was working perfectly for me. But just to be sure, I tried executing the jar outside of gradle. And then I did in fact get the infamous None of the provided engines could be initialized exception. So, I do think your initial assumptions were correct.

[mgroth0]

One more clarification about my environment. I develop tools for myself using IntellJ/Gradle. Then I often execute them as a daemon process through Lingon.

Originally this library worked only in a gradle run task but not when executing the built jar through Lingon. But, I just discovered that Lingon does allow users to set their environmental variables. So, I set PATH to the correct value in Lingon. Now this library works perfectly for me through Lingon as well.

[caoccao]

I wonder if you have evaluated Javet (Java + V8) which supports both Mac x86_64 and arm64.