GIT integration not working after first commit. #6862

Closed scaddenp closed 9 months ago

Apache NetBeans version

Apache NetBeans 19

What happened

Fresh checkout of a project from GitLab (internal version). I can make changes then commit and push ok. If I think make further changes and try to commit, it hangs on "preparing commit". I left it 12 hours and no progress made. This is not specific to this project. Problem occurs in others too. Working fine in NB 17, but issue occurred with I upgraded to 19. If I go to Git GUI (windows), with this project, I can commit and push without a problem

How to reproduce

See above

Did this work correctly in an earlier version?

Apache NetBeans 17

Operating System

windows

JDK

19

Apache NetBeans packaging

Apache NetBeans provided installer

Anything else

No response

Are you willing to submit a pull request?

No

I should note that push also hangs after 1st time it is used. It is as if some resource is locked after first use and subsequent processes cannot unlock it.

Restarting NB helps for some projects - but not all. Another possibility would be an invisible popup. No sign of a visible one though.

please try NB 20 since it updated the JGit dependencies.

The description sounds like the problem starts after the first push, not after the first commit as the title indicates? I suppose you can create as many commits you want until you push to your gitlab instance?

I'm working with NetBeans on Windows for years now and never had problems pushing changes into git. So I would rule out a general problem. You could try to get a stack trace to see where it hangs. Given that this happens on windows, I would assume, that you see a file locking problem, maybe you can find a tool to see what file is kept open.

Matthias - me too. never had a problem till I upgraded to this version. How do you get a stack trace out of NB? However I will try NB 20 and see what that does. One other thing. Sometimes get complaints from git gui. It cant unlink a pack file when compressing.

How to get a stacktrace when NetBeans seems to be stuck:

Open a command line (cmd)
Run <PATH_TO_JDK>\bin\jps.exe -v. This will list all java processes currently executing. There should be at least two: jps itself and NetBeans. In my test the NetBeans process did not show the main class, but verbose mode shows the parameters and that makes it recognizable.
Read the PID (first column) from the output and run <PATH_TO_JDK>\bin\jstack.exe PID > PATH_YOU_CAN_WRITE TO

This will get you a stack trace in the mentioned file. If you remove the output redirection from jstack you see the stacktrace in the console.

The problems you mention could be a virus scanner, that locks the file and prevents NetBeans and git gui to do their job (gut feeling)

Just getting back to this. Getting on multiple projects, even if checkout fresh. Happens in 20 too. I will see what I can do about the stack dump

Ok, got stacktrace, but not telling me a lot. I guess this bit relevent

"Git - E:\projects\git\boreservice-sb" #207 daemon prio=1 os_prio=-2 cpu=1234.38ms elapsed=234.87s tid=0x0000000047159000 nid=0x5a4c waiting for monitor entry  [0x0000000201e7e000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.netbeans.modules.git.FileStatusCache.refreshStatusesBatch(FileStatusCache.java:560)
        - waiting to lock <0x000000061a4ccdf8> (a org.netbeans.modules.git.FileStatusCache)
        at org.netbeans.modules.git.FileStatusCache.refreshAllRoots(FileStatusCache.java:251)
        - locked <0x000000062f3cadf0> (a java.io.File)
        at org.netbeans.modules.git.ui.commit.GitCommitPanel$GitCommitDialogProgressSupport.loadFiles(GitCommitPanel.java:269)
        at org.netbeans.modules.git.ui.commit.GitCommitPanel$GitCommitDialogProgressSupport.perform(GitCommitPanel.java:232)
        at org.netbeans.modules.git.client.GitProgressSupport.performIntern(GitProgressSupport.java:92)
        at org.netbeans.modules.git.client.GitProgressSupport.run(GitProgressSupport.java:85)
        at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1420)
        at org.netbeans.modules.openide.util.GlobalLookup.execute(GlobalLookup.java:45)
        at org.openide.util.lookup.Lookups.executeWith(Lookups.java:287)
        at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2035)

   Locked ownable synchronizers:
        - None

Failing to work on 15 as well after the initial commit. This is pretty maddening.

Well to have a chance to help the full threaddump/stacktrace is required. We see that there is a thread waiting for a lock, but we don't see which thread holds the lock and what might be the source for a deadlock.

ok, here is full threaddump from jstack NB_stacktrace.txt

This looks is the blocker to the trace you identified:

"GitRefresh" #98 daemon prio=1 os_prio=-2 cpu=6553171.88ms elapsed=8350.86s tid=0x0000000032d4d000 nid=0x16d8 runnable  [0x00000000341ae000]
   java.lang.Thread.State: RUNNABLE
    at java.io.WinNTFileSystem.checkAccess(java.base@11.0.13/Native Method)
    at java.io.File.canRead(java.base@11.0.13/File.java:780)
    at org.netbeans.modules.gradle.spi.GradleFiles.searchPathUp(GradleFiles.java:164)
    at org.netbeans.modules.gradle.spi.GradleFiles.searchBuildScripts(GradleFiles.java:129)
    at org.netbeans.modules.gradle.spi.GradleFiles.<init>(GradleFiles.java:104)
    at org.netbeans.modules.gradle.spi.GradleFiles.<init>(GradleFiles.java:91)
    at org.netbeans.modules.gradle.NbGradleProjectFactory.isProjectCheck(NbGradleProjectFactory.java:74)
    at org.netbeans.modules.gradle.NbGradleProjectFactory.isProject(NbGradleProjectFactory.java:56)
    at org.netbeans.modules.gradle.NbGradleProjectFactory.loadProject(NbGradleProjectFactory.java:95)
    at org.netbeans.modules.projectapi.nb.NbProjectManager.createProject(NbProjectManager.java:376)
    at org.netbeans.modules.projectapi.nb.NbProjectManager.access$300(NbProjectManager.java:69)
    at org.netbeans.modules.projectapi.nb.NbProjectManager$2.run(NbProjectManager.java:289)
    at org.netbeans.modules.projectapi.nb.NbProjectManager$2.run(NbProjectManager.java:218)
    at org.netbeans.modules.openide.util.DefaultMutexImplementation.readAccess(DefaultMutexImplementation.java:188)
    at org.openide.util.Mutex.readAccess(Mutex.java:232)

I would check a few minutes apart, if these two threads stay in the trace. If that is the case, you could try to run your IDE with debugging enabled for example on linux I can run:

~/bin/netbeans-20/bin/netbeans -J-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=1044

And then I can run a second instance of Netbeans and connect on port 1044 using JPDA. You could then inspect the stack of the GitRefresh thread and see on which file it blocks.

What is more I notice, that you seem to run on JDK 11.0.13. Looking at the tables (https://www.java.com/releases/matrix/) that is more that two years old. I would advice to test with the most recent release of JDK 17 (I use Amazon Corretto on Windows) or even 21 (though that was new when NB20 was released, so it might be rough).

I have multiple jdks, and yes using 11 was mistake. switched to 19. The two threads stays in stack. The Git - project is identical between successive attempts to do a commit. The GitRefresh differs: eg

"GitRefresh" #33 daemon prio=1 os_prio=-2 cpu=3934625.00ms elapsed=5154.73s tid=0x000000003eb53800 nid=0x3e58 runnable  [0x0000000046dee000]
   java.lang.Thread.State: RUNNABLE
    at java.io.File.getParentFile(java.base@11.0.13/File.java:506)
    at org.netbeans.modules.masterfs.filebasedfs.utils.FileInfo.getRoot(FileInfo.java:165)
    at org.netbeans.modules.masterfs.filebasedfs.fileobjects.FileObjectFactory.getInstance(FileObjectFactory.java:105)
    at org.netbeans.modules.masterfs.filebasedfs.fileobjects.FileObjectFactory.getInstance(FileObjectFactory.java:100)
    at org.netbeans.modules.masterfs.filebasedfs.fileobjects.BaseFileObj.getFactory(BaseFileObj.java:631)
    at org.netbeans.modules.masterfs.filebasedfs.fileobjects.BaseFileObj.getParent(BaseFileObj.java:591)
    at org.netbeans.modules.java.openjdk.common.BuildUtils.getFileObject(BuildUtils.java:93)
    at org.netbeans.modules.java.openjdk.project.ModuleDescription.findJDKRoot(ModuleDescription.java:135)
    at org.netbeans.modules.java.openjdk.project.ModuleDescription.getModules(ModuleDescription.java:82)
    at org.netbeans.modules.java.openjdk.project.JDKProject.isJDKProject(JDKProject.java:387)
    at org.netbeans.modules.java.openjdk.project.JDKProject$JDKProjectFactory.isProject(JDKProject.java:405)
    at org.netbeans.modules.java.openjdk.project.JDKProject$JDKProjectFactory.loadProject(JDKProject.java:410)

versus later:

"GitRefresh" #33 daemon prio=1 os_prio=-2 cpu=4255859.38ms elapsed=5647.87s tid=0x000000003eb53800 nid=0x3e58 runnable  [0x0000000046dee000]
   java.lang.Thread.State: RUNNABLE
    at java.io.WinNTFileSystem.getBooleanAttributes(java.base@11.0.13/Native Method)
    at java.io.File.exists(java.base@11.0.13/File.java:831)
    at org.netbeans.modules.masterfs.filebasedfs.fileobjects.FileObjectFactory.getFileObject(FileObjectFactory.java:205)
    at org.netbeans.modules.masterfs.filebasedfs.fileobjects.FileObjectFactory.getValidFileObject(FileObjectFactory.java:777)
    at org.netbeans.modules.masterfs.filebasedfs.fileobjects.FolderObj.getFileObject(FolderObj.java:104)
    at org.openide.filesystems.FileObject.getFileObject(FileObject.java:959)
    at org.netbeans.modules.ide.ergonomics.fod.FeatureProjectFactory$Data.hasFile(FeatureProjectFactory.java:178)
    at org.netbeans.modules.ide.ergonomics.fod.FeatureInfo.isNbProject(FeatureInfo.java:329)
    at org.netbeans.modules.ide.ergonomics.fod.FeatureInfo.isProject(FeatureInfo.java:160)
    at org.netbeans.modules.ide.ergonomics.fod.FeatureProjectFactory.loadProject(FeatureProjectFactory.java:261)

Debugging NB by stack is somewhat eyebrow raising, but I will see what I can do.

Running on 11 was an oversight - running on 19 now. No change. I notice that just opening NB and doing a Pull also locks up. It does the fetch and then the -ff merge, but the pull thread then locks indefinitely "refreshing file statuses".

I had quite a time get a second instance NB to start at all (need to set --userdir) but so far havent figured get a stacktrace. I am still in the dark as to how the stacktrace would identify the file that is being locked.

Ok, you need to tell the IDE what to do. The intention is now not to get a stack trace (you did that already), but get the info what files are modified/tried to be read.

Start your "Work" IDE like this: <PATH_TO_NETBEANS>\bin\netbeans64.exe -J-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=1044 This will launch the IDE in a "debuggable" mode. The debug agent listens now on TCP port 1044 for a debugger. The -J indicates to the NetBeans launcher that the rest of the option should be passed to the java launcher itself. That is instructed to load the debugger agent (jdwp) and configured to start in server mode, not to supend execution and listen to TCP port 1044.
Do what you need to do, to get it into its "broken" state
Launch the debugging IDE (yeah, needs a different userdir): <PATH_TO_NETBEANS>\bin\netbeans64.exe --userdir c:/temp/debugging
Ensure, that in the Plugin Manager (Tools -> Plugins -> Installed) the "Java SE" is Choose DebuActive and if it is not, activate it
Choose "Debug -> Attach Debugger", ensure "Java Debugger (JPDA)" is selected as "Debugger", as Connector choose "SocketAttach", as "Host" choose "localhost", as Port "1044". "Ok" that:
NetBeans now switches to debugging view and on the left side you will see a list of threads: And on the lower border you see Variables and Breakpoints. Variables becomes active, once a a thread is stopped.
In the debugging view you can suspend threads by clicking on the "Pause"-Symbol (at the end of the corresponding line), The view then changes and show a stack trace for that thread. You can now activate each stack frame by double clicking on it (it is then printed in bold) and in the Variables view you can see the variables in the stack frame:

In this case I clicked "Pause" on the "exec_JNA..." Thread and then double clicked the "WinNTFileSystem.delete:625" line. As the runtime JDK matches the JDK of the debugged IDE, the source code in the JDK is automatically opened. In the example we can see, that f is the passed in file and when I expand it, I can see the backing path.

This is of course not a reproduction of your situation, but the idea is, that you repeat that against your "Work IDE" and check the threads, that you can identify in the stacktrace. In these threads check the stack frames to see whether you can identify which file is passed to java.io.WinNTFileSystem.checkAccess.

Further checks can be done using the breakpoint function. For example you can create a methdo break point like this:

The red marked icon is "Create new breakpoint".

You can now choose what kind of breakpoint you want to create. This way you would get a break on each call to `checkAccess:

Be warned though: Method Breakpoints are slow.

When a break point is hit, you get this view:

You see on the left the debugging view highlights the current active stack frame, in Breakpoints view the breakpoint that was hit is highlighted. You can now switch to "Variables", change the active stack frame and inspect the variables:

In the example checkAccess is called from File#canWrite and is invoked on the path z:\\src\\jnalib\\contrib\\platform\\build.xml.

To continue execution, you can now allow execution to allow for the thread using the play button in "Debugging" tab or by choosing "Continue" in the debugging toolbar:

Breakpoints can be removed at any time, if execution is topped because a breakpoint was hit, the thread stays suspended until you allow execution again.

If you find problem, use the "stop" button in the debugging toolbar. The Debugging IDE will disconnect from the debugged IDE.

I hope this helps a bit.

Thanks for that very detailed set of instructions! I will follow up.

Phew - finally think I have resolved this - and explained why failing on all versions of NB, but not with git command line and why it is only bugging me....

The thread GitRefresh (which I guess would not run until after first commit) was sitting looking at filecache for file /A/B/A/B/... repeating A/B for nearly 1000 deep. (no actual files, only directories) Like recursion gone wrong. Window file explorer and command line tools wouldn't delete the tree so had to write custom java code to nuke it. Once gone, Git seems to be perfectly happy. The project with offending tree was not one I was using but it was still open in NB. How the file tree got created in the first place remains a complete mystery but it was one where I was experimenting with different tools to package code and JRE into a single executable.

Matthias - you are a scholar and a gentleman. I cant imagine how I would have found this without your help. Gittrefresh was first thread I tried and the weird file path caught my eye the moment I got variables up though I was looking checkAccess. Once I found the file tree actually existed, an Ah Ha moment followed. Thank you again from the bottom of my heart.

@scaddenp happy to help. Thanks for reporting back. This is an interesting case and something to keep in mind. Under linux I would not be surprised to see such structures as symlinks and bind mounts are more or less normal, under Windows this was new (yeah I know that Windows can mount, but it is less common).

I was suspicious that the problem might have been created by junctions a la linux symlinks but no. Pretty weird and I cant really get to bottom of how it was created. The immensely long path clearly created enormous trouble for some windows system routines which flowed into netbeans.

apache / netbeans