OpenWebStart doesn't provide a way of loading assistive technologies

[vlad-coteanu-uipath]

I studied for a while the possibility of loading assistive technologies inside a java application and the way OpenWebStart works doesn't allow this to happen.

Assistive technologies are loaded when the Toolkit is first instantiated. Suppose we pass the assistive_technology argument via the "property" tag, then the properties will be loaded way after the Toolkit is instantiated.

If instead, I try to use the vm-args, then the assistive_technology is identified, however, it still won't work because the .jar dependencies are also loaded after the application is started. Therefore, when loadAssistiveTechnology() method, inside java.awt.Toolkit.java will get called, then the needed classes won't be there. If I try to pass the dependencies via vm-args, using the -cp flag, then this won't work either.

I also tried using signed .jnlp, signed jars, and a lot of combinations but could not make it work.

[janakmulani]

As we understand you have a jnlp app with accessibility features and you would like to run that application using OWS. Can you provide us with a small sample? Also can you please tell us how do you start this accessibility enabled app with java (i.e without jnlp)?

[vlad-coteanu-uipath]

Resources: https://uipath-my.sharepoint.com/:u:/p/vlad_coteanu/ESN_vWurIG1Fp3WEpneUFwMB1YlinJTlaJ7ZLcT6ANN2AQ?e=NfKdLu

Yes, I have a JNLP app and I want to attach an accessibility "plugin" to it via the assistive technology mechanism. The archive contains a file called imdsloader-bkp.jnlp (public app) which is the original jnlp file. I have edited it to use a local folder as codebase and created imdsloader.jnlp. It uses the resources from the lib folder, where all the dependencies are signed using the same certificate. Also, it may contain additional UiPath dependencies. There are also 2 images with their console command attached. First I opened the app using OWS, then, I opened directly the jar file using a java -cp dependencies <mainclassname> pattern. The confirmation that the assistive technology is loaded should be observed in the logs. There are 2 things required to load it:

1. To specify the name of the assistive technology
2. To add the dependencies to the classpath of the application

Loading UiPathBridgeInterface Dll
loadUiPathBridgeDll - java.lang.UnsatisfiedLinkError: Can't load library: C:\UiPath\JNLPPlayground\UiPathJavaBridgeV8_x64.dll
UiPathJavaBridge Interface DLL loaded from: C:\UiPath\JNLPPlayground\lib\UiPathJavaBridgeV8_x64.dll
Java Version: 11.0.10
UiPath Access Bridge started
UiPathJavaServiceProvider::UiPathBridge instantiated

[janakmulani]

You are specifiying : java-vm-args="-Djavax.accessibility.assistive_technologies=UiPathJavaServiceProvider -cp C:\UiPath\JNLPPlayground\lib\*"

-Djavax.accessibility.assistive_technologies is not recognized by OWS as it is not in the list of secure jvm args.

You will have to specify it in the jvm args whitelist in deployment.properties file: deployment.jvm.arguments.whitelist=-Djavax.accessibility.assistive_technologies

-cp argument is not recognized by OWS. All classes and native libs (e.g.dlls) required by the app in jnlp file must be in signed jars that are specified under tag.

[vlad-coteanu-uipath]

Ok, I have tried that method with JVM params and I came to the conclusion that the dependent jar files are not loaded into the target java application when the loadAssistiveTechnologies method is called.

The assistive technology parameter is successfully parsed as a vm-arg, but it seems that UiPathJavaServiceProvider class, which lies inside a jar specified by tag, is not available.

2021-10-19_10_28_47.393-ows-stage1.log 2021-10-19_10_28_47.393-ows-stage2.log

[janakmulani]

Like I said earlier, please make sure that all the necessary classes and native libs like dlls are in the signed jars that are specified under the tag.

[vlad-coteanu-uipath]

Hey Janak,

All the dependencies are there. I studied the logs a little bit more and compared the cases when the assistive_technologies parameter is specified vs the one where it is not specified. It looks like in the case when it is specified, the exception java.awt.AWTError: Assistive Technology not found: UiPathJavaServiceProvider is thrown way before the step when the dependencies (UiPathJavaServiceProvider.jar) are loaded.
From what I understand, the process goes like this:

1. I call javaws imdsloader.jnlp and a process is started. It is using a default JRE it finds on the machine (in my case a 1.8 JRE).
2. Number 1 process creates another process with an embedded JRE from OWS, and there is more information about this one specified at the end of ows-stage-1.log file ("OwsJvmLauncher] About to launch external with commands:"). This process starts collecting the parameters and the dependencies for the actual Java Application and starts it.
3. This is the target Java Application.

From the failing log, it looks like the second process calls UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName()); which triggers the loading of the assistive technologies in the second process. This means that the vm-args parameter is considered for this second process, but it shouldn't because, at that moment in execution, the dependencies are not loaded into its classpath.

I know I might have errors in my logic regarding the 3 processes, but from the logs, I am pretty sure that the assistive_technologies are loaded in a process, way before the dependencies are parsed and downloaded.

2021-10-21_10_21_48.509-ows-stage2.log

2021-10-21_10_20_20.903-ows-stage2.log

[sclassen]

You are right about the three processes, and I do not see any easy way to chang the order of how things are happening.

[vlad-coteanu-uipath]

But is there any way we can consider the -Djavax.accessibility.assistive_technologies parameter a safe JVM parameter, such that we can pass it via property tag and it will only be used by the third process?

[sclassen]

There is a setting which allows you to add parameters to the whitelist. But even then, dependencies are ownly downloaded in the third process shortly before the nain method of the application is called.

[vlad-coteanu-uipath]

I am already using the "vm-args in jnlp + whitelist in deployment.properties" method. My guess is that if we use the <property> tag, either those parameters won't be loaded into the second process, or they will be loaded long after UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName()); is called. In both of the cases, the second process won't attempt to load the assistive technologies anymore.

[vlad-coteanu-uipath]

@sclassen , I see that JVMUtils belongs to IcedTeaWeb and that IcedTeaWeb is also a repo where you are active. I know that the source for the secure properties is: https://docs.oracle.com/javase/8/docs/technotes/guides/javaws/developersguide/syntax.html#, but I also think that it is outdated.

This issue could have never happened for Oracle javaws because they removed it starting from Java 9. Until Java 8 (including it), the dependencies of assistive technologies didn't have to be specified through the classpath, there was an extension mechanism that allowed placing dependencies inside lib\ext folder. We could also place the assistive technologies parameter inside the JRE and everything would have worked well. Sadly, the extension mechanism was also removed from Java 9 and the only thing left for loading the dependencies is the classpath. Considering the above, Oracle had no reason to update their implementation of JWS to allow assistive technologies, but newer implementations that aim to run with Java 9+ should do it. At least for applications started with JWS, it may be the only entry point for automation technologies.

[sclassen]

Yes, this UI look and feel has given us some problems in the past. Icedtea-web and OpenWebStart both are using Swing to display UI elements. We can try to brainstorm next week what would be possible. Would you be available for a call during cest worktime?

[vlad-coteanu-uipath]

Sure, I am based in Bucharest (CEST + 1), so almost every option that works for you, will work for me as well.

[sclassen]

Can you drop us a mail at openwebstart@karakun.com so we can contact you in a non public way. Please mention the github issue number so we can easily make the link to this topic

[janakmulani]

@vlad-coteanu-uipath Can you please tell me what are the contents of C:\UiPath\JNLPPlayground\lib\ directory? Can you provide us those contents?

[vlad-coteanu-uipath]

@janakmulani , you can access this URL for resources: https://uipath-my.sharepoint.com/:u:/p/vlad_coteanu/ESN_vWurIG1Fp3WEpneUFwMB1YlinJTlaJ7ZLcT6ANN2AQ?e=NfKdLu

[janakmulani]

This URL you had posted earlier and I have downloaded from there. I wanted to know what is there in the directory C:\UiPath\JNLPPlayground\lib\ that you specify as part of -cp jvm arg?

[vlad-coteanu-uipath]

The intention was to point to UiPathJavaServiceProvider.jar and UiPathJavaBridge.jar, but since -cp is not allowed, it was just a not successful try.

[janakmulani]

UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName());

Toolkit.initAssistiveTechnologies is called much before that when OWS tries to setup the console Windows to display logs.

java-vm-args="-Djavax.accessibility.assistive_technologies=UiPathJavaServiceProvider

Should the value of assistive_technologies be UiPath.UiPathJavaServiceProvider => fully qualified class name include the package name?

C:\UiPath\JNLPPlayground\lib\

This directory has jars and some dlls. Are these dlls included in any of the jars? The native libs that you need for your app must be placed in a jar that is specified in jnlp resources. Can you tell me what exact jars and dlls are needed for UiPath Assistive tech feature?

[vlad-coteanu-uipath]

@janakmulani ,

You are probably right, it's actually when getDefaultToolkit() method gets called first.

I have also tried with the full qualified class name. The thing is, since this is called from the second "process" and the dependencies are loaded only in the third one, even if we the "correct" name, it won't be found.

The dlls can be referenced through an environment variable and it should be enough (UIPATH_JB_PATH -> folder which contains those dlls). At least from the logs we see that it doesn't reach the point where the ServiceProvider main class is called or to the point when it loads those dlls.