Serenade detects intellij as atom in ubuntu

milosbugarinovic commented 1 year ago

System Information

Serenade Version: v2.0.2

OS and Version: Ubuntu 22.04.2

Application: IntelliJ

Issue Description

When using Serenade on Ubuntu and having IntelliJ IDE open and in focus, an issue arises where Serenade incorrectly detects that Atom editor is selected instead of IntelliJ.

How to Reproduce

Ensure that Serenade is installed and running on Ubuntu.
Open the IntelliJ IDE.
Make sure that the IntelliJ IDE is in focus (active window).
Observe that Serenade incorrectly detects that Atom editor is selected instead of IntelliJ.

Screenshots

Additional info

Expected Behavior

Serenade should accurately detect the active application and recognize that IntelliJ IDE is in focus when it is open and being used.

Actual Behavior

Serenade incorrectly identifies the Atom editor as the active application when IntelliJ IDE is open and in focus. This leads to voice commands not working correctly.

Investigation:

During my investigation of this issue, I found next:

in function determineActiveApplication()(serenade/client/src/main/execute/system.ts) I noticed that if function result includes atom (result.includes("atom")) serenade decides that the atom is in focus, and that this is being checked before checking if the application that is selected is IntelliJ

Further investigation how driver.getActiveApplication() works and what is the result of this function on my machine I discovered this by running a simple script from inside IntelliJ's terminal app (so IntelliJ would be in focuse :smile: )

const driver = require('serenade-driver')

driver.getActiveApplication().then(result => {
    console.log(result)
})

// /opt/idea/idea-iu-222.3345.118/jbr/bin/java-classpath/opt/idea/idea-iu-222.3345.118/lib/app.jar:/opt/idea/idea-iu-222.3345.118/lib/3rd-party-rt.jar:/opt/idea/idea-iu-222.3345.118/lib/util.jar:/opt/idea/idea-iu-222.3345.118/lib/util_rt.jar:/opt/idea/idea-iu-222.3345.118/lib/util-8.jar:/opt/idea/idea-iu-222.3345.118/lib/jps-model.jar:/opt/idea/idea-iu-222.3345.118/lib/stats.jar:/opt/idea/idea-iu-222.3345.118/lib/protobuf.jar:/opt/idea/idea-iu-222.3345.118/lib/external-system-rt.jar:/opt/idea/idea-iu-222.3345.118/lib/intellij-test-discovery.jar:/opt/idea/idea-iu-222.3345.118/lib/forms_rt.jar:/opt/idea/idea-iu-222.3345.118/lib/rd.jar:/opt/idea/idea-iu-222.3345.118/lib/externalprocess-rt.jar:/opt/idea/idea-iu-222.3345.118/lib/annotations-java5.jar:/opt/idea/idea-iu-222.3345.118/lib/annotations.jar:/opt/idea/idea-iu-222.3345.118/lib/async-profiler-windows.jar:/opt/idea/idea-iu-222.3345.118/lib/async-profiler.jar:/opt/idea/idea-iu-222.3345.118/lib/byte-buddy-agent.jar:/opt/idea/idea-iu-222.3345.118/lib/error-prone-annotations.jar:/opt/idea/idea-iu-222.3345.118/lib/groovy.jar:/opt/idea/idea-iu-222.3345.118/lib/idea_rt.jar:/opt/idea/idea-iu-222.3345.118/lib/intellij-coverage-agent-1.0.706.jar:/opt/idea/idea-iu-222.3345.118/lib/jsch-agent.jar:/opt/idea/idea-iu-222.3345.118/lib/junit.jar:/opt/idea/idea-iu-222.3345.118/lib/junit4.jar:/opt/idea/idea-iu-222.3345.118/lib/ant/lib/ant.jar-xx:errorfile=/home/milos/java_error_in_idea_%p.log-xx:heapdumppath=/home/milos/java_error_in_idea_.hprof-xms128m-xmx750m-xx:reservedcodecachesize=512m-xx:+useg1gc-xx:softreflrupolicymspermb=50-xx:cicompilercount=2-xx:+heapdumponoutofmemoryerror-xx:-omitstacktraceinfastthrow-xx:+ignoreunrecognizedvmoptions-xx:compilecommand=exclude,com/intellij/openapi/vfs/impl/filepartnoderoot,triedescend-ea-dsun.io.usecanoncaches=false-dsun.java2d.metal=true-djbr.catch.sigabrt=true-djdk.http.auth.tunneling.disabledschemes=""-djdk.attach.allowattachself=true-djdk.module.illegalaccess.silent=true-dkotlinx.coroutines.debug=off-dsun.tools.attach.tmp.only=true-xmx4096m-djb.vmoptionsfile=/home/milos/.config/jetbrains/intellijidea2023.1/idea64.vmoptions-djava.system.class.loader=com.intellij.util.lang.pathclassloader-didea.vendor.name=jetbrains-didea.paths.selector=intellijidea2023.1-djna.boot.library.path=/opt/idea/idea-iu-222.3345.118/lib/jna/amd64-dpty4j.preferred.native.folder=/opt/idea/idea-iu-222.3345.118/lib/pty4j-djna.nosys=true-djna.noclasspath=true-dsplash=true--add-opens=java.base/java.io=all-unnamed--add-opens=java.base/java.lang=all-unnamed--add-opens=java.base/java.lang.ref=all-unnamed--add-opens=java.base/java.lang.reflect=all-unnamed--add-opens=java.base/java.net=all-unnamed--add-opens=java.base/java.nio=all-unnamed--add-opens=java.base/java.nio.charset=all-unnamed--add-opens=java.base/java.text=all-unnamed--add-opens=java.base/java.time=all-unnamed--add-opens=java.base/java.util=all-unnamed--add-opens=java.base/java.util.concurrent=all-unnamed--add-opens=java.base/java.util.concurrent.atomic=all-unnamed--add-opens=java.base/jdk.internal.vm=all-unnamed--add-opens=java.base/sun.nio.ch=all-unnamed--add-opens=java.base/sun.nio.fs=all-unnamed--add-opens=java.base/sun.security.ssl=all-unnamed--add-opens=java.base/sun.security.util=all-unnamed--add-opens=java.base/sun.net.dns=all-unnamed--add-opens=java.desktop/com.sun.java.swing.plaf.gtk=all-unnamed--add-opens=java.desktop/java.awt=all-unnamed--add-opens=java.desktop/java.awt.dnd.peer=all-unnamed--add-opens=java.desktop/java.awt.event=all-unnamed--add-opens=java.desktop/java.awt.image=all-unnamed--add-opens=java.desktop/java.awt.peer=all-unnamed--add-opens=java.desktop/java.awt.font=all-unnamed--add-opens=java.desktop/javax.swing=all-unnamed--add-opens=java.desktop/javax.swing.plaf.basic=all-unnamed--add-opens=java.desktop/javax.swing.text.html=all-unnamed--add-opens=java.desktop/sun.awt.x11=all-unnamed--add-opens=java.desktop/sun.awt.datatransfer=all-unnamed--add-opens=java.desktop/sun.awt.image=all-unnamed--add-opens=java.desktop/sun.awt=all-unnamed--add-opens=java.desktop/sun.font=all-unnamed--add-opens=java.desktop/sun.java2d=all-unnamed--add-opens=java.desktop/sun.swing=all-unnamed--add-opens=jdk.attach/sun.tools.attach=all-unnamed--add-opens=jdk.compiler/com.sun.tools.javac.api=all-unnamed--add-opens=jdk.internal.jvmstat/sun.jvmstat.monitor=all-unnamed--add-opens=jdk.jdi/com.sun.tools.jdi=all-unnamedcom.intellij.idea.main

if you search the output of the console log you will find word atomic which contains atom :smile:

.../java.util.concurrent=all-unnamed--add-opens=java.base/java.util.concurrent.atomic=all-unnamed--add-opens=java.base/jdk.internal.vm=all-u...

`driver.getActiveApplication()` result for other applications on ubuntu

### VS Code - v1.79.1 > Commit: 4cb974a7aed77a74c7813bdccd99ee0d04901215 > Date: 2023-06-12T16:17:13.990Z > Electron: 22.5.7 > Chromium: 108.0.5359.215 > Node.js: 16.17.1 > V8: 10.8.168.25-electron.0 > OS: Linux x64 5.19.0-43-generic /usr/share/code/code--unity-launch ### Firefox Browser - 114.0 (64-bit) firefox-pmilos ### Chrome - Version 114.0.5735.106 (Official Build) (64-bit) /opt/google/chrome/chrome--profile-directory=default ### Slack - 4.32.122 64-bit /usr/lib/slack/slack ### Serenade - v2.0.2 /tmp/.mount_serenazylpws/serenade--enable-crashpad

Solution Proposal

To address the issue, we propose modifying the logic of the application detection in Serenade. One possible solution is to move the check for the Atom editor to the end of the if-else statement. Since the term atom can be found in other applications, such as "atomic" in IntelliJ, this change will allow for more accurate identification. Additionally, it is advisable to reposition any other applications with generic names that may potentially be found in different languages.

Available for Further Investigation

If you require any further assistance or if additional information is needed from my Ubuntu system to aid in the investigation of this issue, I would be more than willing to help. Please feel free to reach out to me with any specific instructions or guidance on how I can assist further in resolving this problem.

sombrafam commented 1 year ago

Thanks for the thorough investigation! Ideally, we should implement better regex/pattern matching, to avoid this from happening with other applications or other combinations. GPTs suggestion: https://gist.github.com/sombrafam/c80e659e04a87912864553f7d7348032

Kn0rk commented 1 year ago

I have been trying to get the project to build but I keep running into errors. I assume you got the project building? If its not too much work would it be possible for you to build a new AppImage. Or somehow share the adjusted linux client?

milosbugarinovic commented 1 year ago

sure @Kn0rk, I have build an AppImage with changes from this PR and I have upload it to dropbox so you can try it.

Kn0rk commented 1 year ago

Thank you that is very kind of you!

marcusoftnet commented 1 year ago

Wow! i most experiencing exactly this problem. This solved it for me thank you so much.

offscriptdev commented 1 year ago

I had the same issue in Arch Linux, and this fixed it for me too. The only thing is that the local server doesn't work in this unofficial patch. Anyways, thank you for this. It's making a huge difference for me right now as I injured my hand.

sombrafam commented 1 year ago

Yeap, it recognizes the branch, but local server does not work as it needs to download the models.

sombrafam commented 1 year ago

@milosbugarinovic how did you buid the appimage? I was able to build serenade, but didnt try the app image yet.

luizferreira commented 1 year ago

Another possible workaround that keeps the local server working is to remove from the IDE init script (e.g bin/pycharm.sh) the argument that makes serenade recognize it as Atom:

--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED

I've been using Pycharm this way for a few months now and it's working as expected.

sombrafam commented 1 year ago

Oww, that's a really nice workaround. The local server is much faster and I prefer to have the local server than having PyCharm recognized. I wonder what side effects that might cause. Click

@luizferreira, if you are not part yet, we have a discord channel: https://discord.com/channels/807359313940709436/1000094199988883647

ethantkoenig commented 9 months ago

If you are unable to modify the IDE init script for whatever reason and don't need Atom support, another workaround is to unpack, modify, and repack the AppImage file as follows:

Download Serenade-2.0.2.AppImage
Download the appropriate appimagetool-*.AppImage (from https://github.com/AppImage/appimagetool/releases)
Run path/to/Serenade-2.0.2.AppImage --appimage-extract: this should extract the app's contents into a directory named squashfs-root
Modify squashfs-root/resources/app/out/main.js, and remove the fragment :e.includes("atom")?"atom". This effectively removes the else if (result.includes("atom")) block.
Run ARCH=x86_64 path/to/appimagetool-x86_64.AppImage squashfs-root (replace x86_64 with the appropriate architecture): this creates a new Serenade AppImage file.

thebl1985 commented 9 months ago

* Run `ARCH=x86_64 path/to/appimagetool-x86_64.AppImage squashfs-root` (replace `x86_64` with the appropriate architecture): this creates a new Serenade AppImage file.

Followed the steps but this last command fails with "execv error: No such file or directory".

Did you have to make any changes to the squashfs-root directory?

There's an issue here about this but nothing worked for me.

serenadeai / serenade