"Navigate > GoToDeclaration" fails if target is in gradle project

[errael]

Apache NetBeans version

Apache NetBeans 16

What happened

Using NetBeans-16u1 (as of 2022/12/18)

If in a maven project's java file and the target declaration is in an open gradle project's file then the gradle project's sources.jar file is opened instead of the open project's source file. If in a gradle project, a source file target in maven project works. It can be reproduced with the trivial projects in attached jar file and the steps described in the next section.

Here's some information from doing GoToDeclaration. The ClasspathInfo is the same doing maven project to maven project or gradle to maven; but the contents of internal compile paths are very different between maven and gradle projects; this looks suspicious. I don't know enough about the various Classpaths and their usage or how they're set up; is this a smoking gun?

As seen at SourceUtils.getFile() cps for .COMPILE is internal compile (AFAICT):

    /src/jvi-dev/nbvi/build/testuserdir/var/cache/index/s77/java/15/classes:
    /junk/Caches/maven_repository/play/m2g/m2g-gradle-lib/1.2/m2g-gradle-lib-1.2.jar:
    /src/jvi-dev/nbvi/build/testuserdir/var/cache/index/s1/java/15/classes:
    /src/jvi-dev/nbvi/build/testuserdir/var/cache/index/s3/java/15/classes,

Referencing the following, see that "index/s2" is not part of internal compile, instead there's the gradle jar file. In .../var/cache/index

index/s3/java/15/classes/play/m2g/app/MavenApp.sig
index/s2/java/15/classes/play/m2g/g/GradleLibrary.sig
index/s1/java/15/classes/play/m2g/m/MavenLib.sig

Note that when doing GoToDeclaration maven to maven, it is resolved by CacheSourceForBinaryQueryImpl.findSourceRoots, in particular with:

URL sourceURL = JavaIndex.getSourceRootForClassFolder(binaryRoot);

where binaryRoot is file:/src/jvi-dev/nbvi/build/testuserdir/var/cache/index/s1/java/15/classes/ and sourceURL is file:/junk/play/m2g/maven-lib-project/src/main/java/.

BTW, The jVi plugin uses NbEditorKit.gotoDeclaration.

How to reproduce

m2g.zip

Attached jar file has 3 projects, 2 maven and one gradle, one java file per project. jars end up under <local_mvn_repo>/play/m2g. Open the three projects and build them. Build order maven-lib, gradle-lib (publishToMavenLocal), maven-app.

In m2g-app::MavenApp.javaon the line new GradleLibrary().someGradleLibraryMethod(); place the caret over someGradleLibraryMethod and do Navigate > GoToDeclaration. Observe: a source file from m2g-gradle-lib-1.2-sources.jar is opened. Expect: source file from the open project is opened.

One can similarly do the same over new MavenLib().mavenLibFunc(); and it works fine.

Did this work correctly in an earlier version?

No / Don't know

Operating System

Linux harmony 6.0.6-76060006-generic #202210290932~1667401208~22.04~d2df702 SMP PREEMPT_DYNAMIC Wed N x86_64 x86_64 x86_64 GNU/Linux

JDK

openjdk 17.0.2 2022-01-18 OpenJDK Runtime Environment (build 17.0.2+8-86) OpenJDK 64-Bit Server VM (build 17.0.2+8-86, mixed mode, sharing)

Apache NetBeans packaging

Own source build

Anything else

Some information encountered which may be useful for reference.

ClasspathInfo [ /// As seen at SourceUtils.getFile Leaving out: boot, module boot, internal boot

compile:
    /junk/Caches/maven_repository/play/m2g/m2g-gradle-lib/1.2/m2g-gradle-lib-1.2.jar:
    /junk/Caches/maven_repository/play/m2g/m2g-maven-lib/1.0-SNAPSHOT/m2g-maven-lib-1.0-SNAPSHOT.jar:
    /junk/play/m2g/maven-app-project/target/classes,

module compile: ,

module class:
    /junk/Caches/maven_repository/play/m2g/m2g-gradle-lib/1.2/m2g-gradle-lib-1.2.jar:
    /junk/Caches/maven_repository/play/m2g/m2g-maven-lib/1.0-SNAPSHOT/m2g-maven-lib-1.0-SNAPSHOT.jar:
    /junk/play/m2g/maven-app-project/target/classes,

src:
    /junk/play/m2g/maven-app-project/src/main/java:
    /junk/play/m2g/maven-app-project/src/main/resources,

internal compile:
    /src/jvi-dev/nbvi/build/testuserdir/var/cache/index/s77/java/15/classes:
    /junk/Caches/maven_repository/play/m2g/m2g-gradle-lib/1.2/m2g-gradle-lib-1.2.jar:
    /src/jvi-dev/nbvi/build/testuserdir/var/cache/index/s1/java/15/classes:
    /src/jvi-dev/nbvi/build/testuserdir/var/cache/index/s3/java/15/classes,

internal module class:
    /src/jvi-dev/nbvi/build/testuserdir/var/cache/index/s77/java/15/classes:
    /junk/Caches/maven_repository/play/m2g/m2g-gradle-lib/1.2/m2g-gradle-lib-1.2.jar:
    /src/jvi-dev/nbvi/build/testuserdir/var/cache/index/s1/java/15/classes:
    /src/jvi-dev/nbvi/build/testuserdir/var/cache/index/s3/java/15/classes,

internal src:
    /junk/play/m2g/maven-app-project/src/main/java,

internal out:
    /src/jvi-dev/nbvi/build/testuserdir/var/cache/index/s3/java/15/classes
]

This lookup result shows up in many places around the areas investigated.

Lookup.getDefault().lookupAll(SourceForBinaryQueryImplementation.class) Collections$UnmodifiableRandomAccessList    ObjectVariable  "size = 12" 
[ 0]    RepositoryForBinaryQueryImpl    #33669  org.netbeans.modules.maven.queries.RepositoryForBinaryQueryImpl@48368120    
[ 1]    GradleSourceForBinary   #33670  org.netbeans.modules.gradle.queries.GradleSourceForBinary@375ff7a8  
[ 2]    GlobalSourceForBinaryQuery  #33671  org.netbeans.modules.java.openjdk.project.GlobalSourceForBinaryQuery@6846b983   
[ 3]    PlatformSourceForBinaryQuery    #33672  org.netbeans.modules.java.platform.queries.PlatformSourceForBinaryQuery@343081b1    
[ 4]    ProjectSourceForBinaryQuery #33673  org.netbeans.modules.java.project.ProjectSourceForBinaryQuery@48733996  
[ 5]    OpenGradleProjectForBinary  #33674  org.netbeans.modules.gradle.java.queries.OpenGradleProjectForBinary@596587  
[ 6]    GradleSourceForRepository   #33675  org.netbeans.modules.gradle.java.queries.GradleSourceForRepository@5d0e5eae 
[ 7]    CacheSourceForBinaryQueryImpl   #33676  org.netbeans.modules.java.source.classpath.CacheSourceForBinaryQueryImpl@6d8361b7   
[ 8]    J2SELibrarySourceForBinaryQuery #33677  org.netbeans.modules.java.j2seplatform.libraries.J2SELibrarySourceForBinaryQuery@3ced4a18   
[ 9]    SourceForBinaryQueryImpl    #33678  org.netbeans.modules.java.debug.SourceForBinaryQueryImpl@77d3655f   
[10]    J2eePlatformSourceForBinaryQuery    #33679  org.netbeans.modules.j2ee.deployment.impl.query.J2eePlatformSourceForBinaryQuery@5807be25   
[11]    DefaultSourceForBinaryQuery #33680  org.netbeans.modules.java.j2seplatform.queries.DefaultSourceForBinaryQuery@2e7fe24b 

Are you willing to submit a pull request?

No

Code of Conduct

Yes

[lkishalmi]

Well, it kind of works like it should. Once the Gradle artifacts are published to the Local Maven Repository, it is no longer associated with its source Gradle project, so the source files are read from the published source jar. That could save a few hours of troubleshooting, when someone simply forgets to publish a new version from the Gradle project into MavenLocal, then would see the current source instead of the published one.

Theoretically it wouldn't be hard to add a SourceForBinaryQueryImplementation2 to associate MavenLocal artifacts with open Gradle projects, though the results might be confusing.

@sdedic @mbien @neilcsmith-net @matthiasblaesing opinions?

[mbien]

@lkishalmi i agree. I would have expected the same behavior. Once the artifact is installed in the local repo, it isn't really associated with the sources of the project anymore.

the stack trace links (e.g in the console or the stack trace analyzer window) have a somewhat similar issue. The IDE can only try to open those links in a best-effort manner (#5116). Long term this could be mitigated by e.g a key combo, for example holding down ctrl while clicking the link could open an attach source dialog (or some smarter solution).

the debugger has also a sources window where users can drag sources around if it doesn't open the right one.

[errael]

works like it should. expected the same behavior.

Doubt if anyone's surprised that I disagree with both statements.

My expectation is that the experience developing java projects with NetBeans is not dependent on project's underlying build system (as much as possible). I expect the default behavior of projects to be the same and that the different project types, for the same language, interact seamlessly.

I don't know the precise semantics of maven projects. But it seems that when interacting with the projects, the open projects are checked/used first in most (all?) situations. That feels like the right thing to do and is easy to understand.

I suspect that both maven and gradle are going to be around for the long term (ant is history). If they behave differently and don't interoperate well within NetBeans, that is unexpected and confusing.

BTW, AFAICT, there is no situtation or setup where Navigate > GoToDeclaration ever works going from maven to gradle.

I was thinking about opening issues about refactoring and breakpoint interoperability; but it seems there's a wider discussion to understand expectations. I haven't done much experimenting with refactoring, but initial results are that it doesn't work well between project types.

I'd tried removing a gradle source file from the repo. Then I could set/use breakpoints in the gradle source files. But if I local publish then the gradle project's breakpoints stop working!?

I'll suggest meditation on the "I" in IDE ;-)

[neilcsmith-net]

I don't know the precise semantics of maven projects. But it seems that when interacting with the projects, the open projects are checked/used first in most (all?) situations. That feels like the right thing to do and is easy to understand.

Why does this make me think of #4274 ! I wouldn't assume Maven support is always getting these things right either. An option to force dependency related queries to use the local repository would be welcomed there, and possibly vice-versa here.

[errael]

I wouldn't assume Maven support is always getting these things right

I don't. But so far the consensus is that gradle projectg support is operating correctly and as expected.

And the overall NetBeans team's position is that interoperability between gradle and maven projects is not important.

[mbien]

And the overall NetBeans team's position is that interoperability between gradle and maven projects is not important.

I am a bit confused where you are getting this from. But even if this would be the case, i would still be open to review contributions which help with gradle-maven interop edge cases. E.g potentially attaching arbitrary source folders to project dependencies might be a generic way to mitigate issues like this. I personally do not plan to put time into this any time soon since i have 7 maven related PRs open right now (and other things on the radar) which will cover my NB contribution time budget for a while.

[errael]

where you are getting this from

My apologies if I've misrepresented things. (I deeply dislike the use of "if" in an apology, but there it is)

My experience, limited, is that the interoperability is poor at best. This is the first issue I've seen about it, and the response has been that it's behaving correctly and as expected. The conclusion that interoperability is not important is coming from my experience.

A suggestion to use the debugger's source pane to workaround the GoToDeclaration problem proved fruitless (while bizarre, I did try it, proved fruitless, could have missed something).

open to review contributions which help with gradle-maven interop edge cases

I've also encountered difficulties with breakpoints and refactoring. I wonder if the gradle support was added more or less standalone with no attention to NetBeans project integration. I haven't reviewed things, but I'm not sure that edge cases is where it's at. I don't understand what

attaching arbitrary source folders to project dependencies might be a generic way to mitigate issues like this

is all about, but it sounds like a kludge rather than having proper integration.

[lkishalmi]

Well, sometimes there is a difference between operating correctly and as expected.

Bit of history why Maven Local artifacts is resolved to Maven projects: The only way Maven could share code between projects was through local repository, even if the project were in the same project tree in source control. That was the case until Maven 3 came out. That also created the belief that the source and the artifacts in Maven Local are the "same". Probably the most common way maven build is executed today is using mvn install (even NetBeans does that), instead of mvn package, which would be enough.

The interoperability between Maven and Gradle projects were never an important issue as people either use Maven or Gradle for one project usually not both of them. When they do that. They share released artifacts, not live code.

I can assure you that the Gradle integration was created with the attention of NetBeans Project integration, and many parts were made after how the Maven integration were done.

My expectation is that the experience developing java projects with NetBeans is not dependent on project's underlying build system (as much as possible). I expect the default behavior of projects to be the same and that the different project types, for the same language, interact seamlessly.

I would say that's a kind of utopia, probably a very few people chased / cared about that one. I'd say the build system defines the project, not the language. There could be similarities, after all, we would like to see the sources get compiled and packaged at the end, but that's it.

Just check the Ant and Maven integration. You might end up with a similar situation like with Gradle. You would need to a library produced with Ant consumable from maven, so you would add some task which publish the artifact created with Ant into Maven Local repo. The IDE would not be able to trace back the artifact in the repo into the Ant project sources. Though if you'd provide both binary and source artifacts from Ant, then the provided source jar would be opened on GoToDeclaration action.

As of breakpoints, nothing prevents you to place a breakpoint into the readonly sources opened by the GoToDeclaration action.

The beauty of NetBeans that Maven, Gradle and Ant as build tools are first class citizens. You can open those project as they are, no extra setup / preparation needed.

[errael]

I've lightly worked with Maven for a few years through some open source work and have encountered Gradle over the last couple years. Last summer I wanted to update my biggest project's build system from ant and I selected Gradle (I can't stomach xml as a programming language); I'm still a novice but I'm happy with the results (even published a Gradle plugin to assist when using NetBeans' standalone lookup library).

I decided to package some things into a library for use in various projects; the lib is used by both Maven and Gradle projects. When working with a developing library, there is mixed Gradle/Maven development; it's unavoidable. Editing, breakpoints and refactoring between projects is needed.

I can assure you that the Gradle integration was created with the attention of NetBeans Project integration, and many parts were made after how the Maven integration were done.

I didn't think carefully enough about "project", apologies. I didn't mean the user facing project view; I meant an underlying/nb-internal Java project API, presumably built around the scanner and it's results (something that provided interoperability) but I guess such an API doesn't exist. Too bad such an API wasn't formalized when Maven support was added.

My expectation is that the experience developing java projects with NetBeans is not dependent on project's underlying build system (as much as possible). I expect the default behavior of projects to be the same and that the different project types, for the same language, interact seamlessly.

I would say that's a kind of utopia, probably a very few people chased / cared about that one. I'd say the build system defines the project, not the language. There could be similarities, after all, we would like to see the sources get compiled and packaged at the end, but that's it.

By "the experience developing java projects with NetBeans" I mean editing, debugging, refactoring source code between open projects, independent of build system. I do not mean build script or other things outside of NetBeans. The expectations are about working with Java source within NetBeans; not hopes for some utopian unified view of building.

Ant is essentially deprecated; it feels like both Maven and Gradle will be around and active for a long time. When working with source code within NetBeans, the underlying build system should not have a bearing. If behavior is different, these problems will only become be more obvious.

The beauty of NetBeans that Maven, Gradle and Ant as build tools are first class citizens. You can open those project as they are, no extra setup / preparation needed.

I agree wholeheartedly. But I don't see what that had to do with interoperability problems at the source code development level.

[lkishalmi]

By "the experience developing java projects with NetBeans" I mean editing, debugging, refactoring source code between open projects, independent of build system. I do not mean build script or other things outside of NetBeans. The expectations are about working with Java source within NetBeans; not hopes for some utopian unified view of building.

It is. Projects provide classpath to the java files in the editor and source -> binary, binary -> source mappings. Your issue here is that the Gradle project does not provide a Maven Local repository -> Gradle source mapping.

Is that good or is that bad? That could be viewed from different angles. For me it seems to be correct as publishMavenLocal means a release, and from that point whatever is published is independent from it source. From your point of view, you would like to get "IDE time binding between the two". That's why @neilcsmith-net mentioned, that there could be an option/switch for that.

You could actually get the Go to declaration/refactoring/etc working if you can convince Maven to use the jar directly produced by the Gradle project (in it's build/libs dir). It could be done by providing that jar system scoped with it's path (that scope is deprecated by now, but still could work). Unfortunately it is hard to convince Maven to do anything the other way it is doing by default.

On the other hand, why would you develop in that mixed project style? It's like asking for problems. What I'd do right now is to convert the Maven project into a Gradle project and use the includeBuild functionality of Gradle. That one is just designed for your exact use case.

You can even leave the pom.xml in place for a while for those who likes Maven.

So sum this up:

• There could be a switch which would let MavenLocal artifacts associated with Gradle project
• Convert the Maven build to Gradle Upon that there could be a desirable Gradle plugin feature, which would convert Maven projects to Gradle.

[errael]

What I'd do right now is to convert the Maven project into a Gradle project and use the includeBuild functionality of Gradle.

I work with several open source projects that I do not own. I do not control the build system that is used. Which, as previously stated, is why the mixed project arises. And in most cases maintaining my own fork is counter productive. It's not a "right now" issue; this issue has become about interoperability. GoToDeclaration is far from a show stopper; suggesting converting stuff to Gradle (which is where the problems came from), while I'm sure is well intentioned, is tone deaf.

From limited experimentation, it seems like using a Maven library from a Gradle project works better than the other way around. So, at least in this case (where I control the library in question), converting from Gradle to Maven is the path of lease resistance (as much as I dislike maven).

It is clear that if NetBeans is the IDE in use, a library project which is expected to be shared, should never be built with Gradle.

mixed project style? It's like asking for problems

Welcome to the real world. Maven is the dominant build system. Choosing a model incompatible with Maven projects is difficult to fathom and is telling users that if they use Gradle, they are asking for problems.

[neilcsmith-net]

@errael I'm not sure you understood what I meant by my comment, seen as I think that the Gradle support is probably behaving more correctly than the Maven support here, and that the Maven support should perhaps be changed. It also has nothing to do with project type interoperability.

NetBeans prides itself on being a UI to the build tool. On that basis, I think dependency resolution should always reflect the build tool. @lkishalmi made an interesting comment about mvn install vs mvn package, and I think it should be considered why NetBeans defaults to the former, and the differences in dependency resolution in the two scenarios.

IMO, both Maven and Gradle support should only use the project to resolve dependency queries by default when the dependency is part of the same reactor build (or whatever Gradle calls it). Being then able to opt-in to attaching a project to a dependency would be great. I'd be happy with just being able to opt-out in some circumstances!

[matthiasblaesing]

From my experience the maven approach is sane: If the project version matches the dependency version, then "Goto Declaration" takes you into the project. If it does not match, it goes to the source associated with the dependency.

1. Clone https://github.com/mwiede/jsch.git
2. Open the jsch project
3. Create a new maven project
4. Add a dependency on com.github.mwiede:jsch:0.1.72
5. Download sources for dependency
6. Create a main method, insert System.out.println(JSch.VERSION);
7. Invoke "Goto Declaration" on VERSION

You are taken to the read-only version of the source, you downloaded in step 5.

8. Checkout the tag jsch-0.1.72
9. Go back to your main method and invoke "Goto Declaration" again

You are taken to the editable checked out file (see also the changed display of the dependency node, which now just shows the project name).

For me this is the logical approach and works beautifully. When I have this situation where I have the same version checked out in a project, I'm not interested in the generated source jar, but the live code, as that is what I'm working on.

So without a very convincing argument changing the maven behavior would be a regression in my mind.

[lkishalmi]

@matthiasblaesing Maven behavior if fine. This is about that Gradle does not behaves like that when it's artifact is published to Maven Local.

[errael]

@neilcsmith-net Concise analysis of the situation. Nice.

I guess I used "open project" as implicitly requesting "opt-in to attaching a project to a dependency"; my bad. And yes, opt-out could certainly be useful when you want a reset and/or sanity check.

IMO, both Maven and Gradle support should ...

I strongly agree that default behavior should be that same for both project types and "attach..." should be possible to get what @lkishalmi called "live code".

[neilcsmith-net]

@errael yes, resolving queries via the project based on whether project open and/or project in the same multi-project build would probably be a saner approach. That would be better than copying what Maven support currently does. Forcing the use of version changes or isolated userdirs to control is perhaps not the ideal model! 😄