search

apache / netbeans

Apache NetBeans
https://netbeans.apache.org/
Apache License 2.0
2.63k stars 843 forks source link

NB-14: Cloning/fetching/pushing remote repositories with Git over SSH disregards cnd.tmpbase #4393

Closed zzzyxwvut closed 2 years ago

zzzyxwvut commented 2 years ago

Apache NetBeans version

Apache NetBeans 14

What happened

In Apache NetBeans 14, Git-related support for cloning, fetching from, or pushing to remote repositories is broken, as outlined below, whenever the temporary-file directory (/tmp) is mounted with the noexec option set.

In prior releases, having cnd.tmpbase system property defined, allowed to select a temporary-file directory where necessary execution would have taken place. It may, as well, work still, but its coverage seems to slip (junixsocket).

How to reproduce

[P R E L I M I N A R Y    S T E P]

Set up the ssh transport protocol for a target repository (locally generate public-private key pair, upload the public key). Create a new directory that can be read, written, searched by an Apache NetBeans process and which can be used by it for the management of its own temporary files. Enable keyring-related logging in netbeans/etc/netbeans.conf: netbeans_default_options="... -J-Dorg.netbeans.modules.keyring.level=0"

Proceed with the following steps (FAILURE and SUCCESS) in any order.


[F A I L U R E]

Manually forbid execution of binaries and other executable files from the /tmp directory: sudo mount --options remount,noexec /tmp mount | grep /tmp

Launch Apache NetBeans 14 with -J-Dcnd.tmpbase=/path/to/new/dir. (Substitute the actual path to the newly-created directory for /path/to/new/dir.)

Try either cloning or fetching from or pushing to some Git repository for which the ssh transport protocol was set.

When CLONING...

Go Team -> Git -> Clone

[Repository URL: git@...]

Check "Private/Public Key", type in the passphrase, and press the "Next" button. Expect a message written just above the "Next" button:

Cannot connect to repository at git@... .

Press "Cancel" and exit Apache NetBeans.

In /tmp (and not in /path/to/new/dir), verify the presence of libtmp*libjunixsocket*.so files (apparently, extracted from netbeans/ide/modules/ext/junixsocket-native-common-2.4.0.jar).

View ~/.netbeans/14/var/log/messages.log.

Look for the keyring provider being used:

FINE [org.netbeans.modules.keyring]: Using provider: org.netbeans.modules.keyring.gnome.libsecret.GnomeLibSecretProvider@xxxxxxxx

Look for the "remote hung up unexpectedly" entries:

INFO [org.netbeans.modules.git]: git@xxxxxxxxxxxxxxxx.git: remote hung up unexpectedly

And right below it:

java.lang.NoClassDefFoundError: Could not initialize class org.newsclub.net.unix.AFUNIXSocketAddress     at com.jcraft.jsch.JUnixSocketFactory.connect(JUnixSocketFactory.java:58)     at com.jcraft.jsch.SSHAgentConnector.open(SSHAgentConnector.java:78)     at com.jcraft.jsch.SSHAgentConnector.isAvailable(SSHAgentConnector.java:69)     at com.jcraft.jsch.AgentIdentityRepository.getStatus(AgentIdentityRepository.java:68)     at org.netbeans.libs.git.jgit.JGitSshSessionFactory.setupJSchIdentityRepository(JGitSshSessionFactory.java:182)     at org.netbeans.libs.git.jgit.JGitSshSessionFactory.setupJSch(JGitSshSessionFactory.java:206)     at org.netbeans.libs.git.jgit.JGitSshSessionFactory.getSession(JGitSshSessionFactory.java:103)     at org.eclipse.jgit.transport.SshTransport.getSession(SshTransport.java:107)     at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.(TransportGitSsh.java:247) Caused: org.eclipse.jgit.errors.TransportException: git@xxxxxxxxxxxxxxxx.git: remote hung up unexpectedly     at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.(TransportGitSsh.java:263)     at org.eclipse.jgit.transport.TransportGitSsh.openFetch(TransportGitSsh.java:137)     at org.netbeans.libs.git.jgit.commands.ListRemoteObjectsCommand.runTransportCommand(ListRemoteObjectsCommand.java:51) Caused: org.netbeans.libs.git.GitException: git@xxxxxxxxxxxxxxxx.git: remote hung up unexpectedly     at org.netbeans.libs.git.jgit.commands.TransportCommand.handleException(TransportCommand.java:238)     at org.netbeans.libs.git.jgit.commands.ListRemoteObjectsCommand.runTransportCommand(ListRemoteObjectsCommand.java:64)     at org.netbeans.libs.git.jgit.commands.TransportCommand.run(TransportCommand.java:168)     at org.netbeans.libs.git.jgit.commands.GitCommand$1.run(GitCommand.java:57)     at org.netbeans.libs.git.jgit.commands.GitCommand$1.run(GitCommand.java:54)     at java.base/java.security.AccessController.doPrivileged(AccessController.java:569)     at org.netbeans.libs.git.jgit.commands.GitCommand.execute(GitCommand.java:54)     at org.netbeans.libs.git.GitClient.listRemoteBranches(GitClient.java:893)     at org.netbeans.modules.git.client.GitClient$36.call(GitClient.java:582)     at org.netbeans.modules.git.client.GitClient$36.call(GitClient.java:578)     at org.openide.util.NetworkSettings.suppressAuthenticationDialog(NetworkSettings.java:138)     at org.netbeans.modules.git.client.GitClient$CommandInvoker$1$1.call(GitClient.java:931)     at org.netbeans.modules.git.client.GitClient$CommandInvoker$1.call(GitClient.java:956)     at org.netbeans.modules.git.client.GitClient$CommandInvoker.runMethodIntern(GitClient.java:968)     at org.netbeans.modules.git.client.GitClient$CommandInvoker.runMethod(GitClient.java:897)     at org.netbeans.modules.git.client.GitClient$CommandInvoker.runMethod(GitClient.java:875)     at org.netbeans.modules.git.client.GitClient$CommandInvoker.access$400(GitClient.java:869)     at org.netbeans.modules.git.client.GitClient.listRemoteBranches(GitClient.java:578) [catch] at org.netbeans.modules.git.ui.clone.RepositoryStep$RepositoryStepProgressSupport.perform(RepositoryStep.java:302)     at org.netbeans.modules.git.client.GitProgressSupport.performIntern(GitProgressSupport.java:92)     at org.netbeans.modules.git.client.GitProgressSupport.run(GitProgressSupport.java:85)     at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1418)     at org.netbeans.modules.openide.util.GlobalLookup.execute(GlobalLookup.java:45)     at org.openide.util.lookup.Lookups.executeWith(Lookups.java:278)     at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2033)


Affirm the availability of that not-found class: cd /path/to/netbeans/14/ jar tf netbeans/ide/modules/ext/junixsocket-common-2.4.0.jar | grep AFUNIXSocketAddress

(Clone the repository with git-clone(1): eval "$(ssh-agent -s)" ssh-add ~/.ssh/PRIVATE_KEY git clone --verbose git@...

Open the (supported) project.)

When FETCHING (or PUSHING)...

(When the project is newly opened, expect a pop-up window, entitled "Specify Git repository location", to appear. Check "Private/Public Key", type in the passphrase, and press the "OK" button. And now...) Expect a pop-up window, entitled "Git Command Failed", to appear with an error message (also saved in messages.log):

git@xxxxxxxxxxxxxxxx.git: remote hung up unexpectedly

Again, as with CLONING, libtmp*libjunixsocket*.so files should be abandoned in /tmp (and not in /path/to/new/dir).

And a similar (fetching) stack trace shall be written to ~/.netbeans/14/var/log/messages.log:

java.lang.NoClassDefFoundError: Could not initialize class org.newsclub.net.unix.AFUNIXSocketAddress     at com.jcraft.jsch.JUnixSocketFactory.connect(JUnixSocketFactory.java:58)     at com.jcraft.jsch.SSHAgentConnector.open(SSHAgentConnector.java:78)     at com.jcraft.jsch.SSHAgentConnector.isAvailable(SSHAgentConnector.java:69)     at com.jcraft.jsch.AgentIdentityRepository.getStatus(AgentIdentityRepository.java:68)     at org.netbeans.libs.git.jgit.JGitSshSessionFactory.setupJSchIdentityRepository(JGitSshSessionFactory.java:182)     at org.netbeans.libs.git.jgit.JGitSshSessionFactory.setupJSch(JGitSshSessionFactory.java:206)     at org.netbeans.libs.git.jgit.JGitSshSessionFactory.getSession(JGitSshSessionFactory.java:103)     at org.eclipse.jgit.transport.SshTransport.getSession(SshTransport.java:107)     at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.(TransportGitSsh.java:247) Caused: org.eclipse.jgit.errors.TransportException: git@xxxxxxxxxxxxxxxx.git: remote hung up unexpectedly     at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.(TransportGitSsh.java:263)     at org.eclipse.jgit.transport.TransportGitSsh.openFetch(TransportGitSsh.java:137)     at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:105)     at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:91)     at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1260)     at org.netbeans.libs.git.jgit.commands.FetchCommand.runTransportCommand(FetchCommand.java:80) Caused: org.netbeans.libs.git.GitException: git@xxxxxxxxxxxxxxxx.git: remote hung up unexpectedly     at org.netbeans.libs.git.jgit.commands.TransportCommand.handleException(TransportCommand.java:238)     at org.netbeans.libs.git.jgit.commands.FetchCommand.runTransportCommand(FetchCommand.java:98)     at org.netbeans.libs.git.jgit.commands.TransportCommand.run(TransportCommand.java:168)     at org.netbeans.libs.git.jgit.commands.GitCommand$1.run(GitCommand.java:57)     at org.netbeans.libs.git.jgit.commands.GitCommand$1.run(GitCommand.java:54)     at java.base/java.security.AccessController.doPrivileged(AccessController.java:569)     at org.netbeans.libs.git.jgit.commands.GitCommand.execute(GitCommand.java:54)     at org.netbeans.libs.git.GitClient.fetch(GitClient.java:640)     at org.netbeans.modules.git.client.GitClient$19.call(GitClient.java:403)     at org.netbeans.modules.git.client.GitClient$19.call(GitClient.java:399)     at org.openide.util.NetworkSettings.suppressAuthenticationDialog(NetworkSettings.java:138)     at org.netbeans.modules.git.client.GitClient$CommandInvoker$1$1.call(GitClient.java:931)     at org.netbeans.modules.git.client.GitClient$CommandInvoker$1$1.call(GitClient.java:937)     at org.netbeans.modules.git.client.GitClient$CommandInvoker$1.call(GitClient.java:956)     at org.netbeans.modules.git.client.GitClient$CommandInvoker.runMethodIntern(GitClient.java:968)     at org.netbeans.modules.git.client.GitClient$CommandInvoker.runMethod(GitClient.java:893)     at org.netbeans.modules.git.client.GitClient$CommandInvoker.runMethod(GitClient.java:875)     at org.netbeans.modules.git.client.GitClient$CommandInvoker.access$400(GitClient.java:869)     at org.netbeans.modules.git.client.GitClient.fetch(GitClient.java:399)     at org.netbeans.modules.git.ui.fetch.FetchAction.fetchRepeatedly(FetchAction.java:170) [catch] at org.netbeans.modules.git.ui.fetch.FetchAction$3.perform(FetchAction.java:146)     at org.netbeans.modules.git.client.GitProgressSupport.performIntern(GitProgressSupport.java:92)     at org.netbeans.modules.git.client.GitProgressSupport.run(GitProgressSupport.java:85)     at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1418)     at org.netbeans.modules.openide.util.GlobalLookup.execute(GlobalLookup.java:45)     at org.openide.util.lookup.Lookups.executeWith(Lookups.java:278)     at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2033)



[S U C C E S S    # 1]

Manually permit execution of binaries and other executable files from the /tmp directory: sudo mount --options remount,exec /tmp mount | grep /tmp

(If present, manually remove stale libtmp*libjunixsocket*.so files from /tmp.)

Launch Apache NetBeans 14.

Try fetching from (or pushing to) some Git repository for which the ssh transport protocol was set. Expect a pop-up window, entitled "Specify Git repository location", to appear. Check "Private/Public Key", type in the passphrase, and press the "OK" button.

(There should be neither a new NoClassDefFoundError logged entry nor new libtmp*libjunixsocket*.so files left over in /tmp.)


[S U C C E S S    # 2]

Manually forbid execution of binaries and other executable files from the /tmp directory: sudo mount --options remount,noexec /tmp mount | grep /tmp

Launch Apache NetBeans 14 with JUnixSocket's property set -J-Dorg.newsclub.net.unix.library.tmpdir=/path/to/new/dir, as the temporary-file directory path. (Substitute the actual path to the newly-created directory for /path/to/new/dir.)

Try fetching from (or pushing to) some Git repository for which the ssh transport protocol was set. Expect a pop-up window, entitled "Specify Git repository location", to appear. Check "Private/Public Key", type in the passphrase, and press the "OK" button.

(There should be neither a new NoClassDefFoundError logged entry nor new libtmp*libjunixsocket*.so files left over in /tmp.)

Did this work correctly in an earlier version?

Apache NetBeans 13

Operating System

GNU/Linux Debian (testing/bookworm)

JDK

OpenJDK (17.0.3+7-Debian-1)

Apache NetBeans packaging

Apache NetBeans binary zip

Anything else

It may be speculated that all that is needed to resolve this issue lies in extending cnd.tmpbase's coverage by re-using its value, if any, as the value of the property parsed by the JUnixSocket dependency of Apache NetBeans: org.newsclub.net.unix.library.tmpdir.

For example,

class UnifiedBespokeTmpDirTests
{
    private static final String PROP_J_UNIX_SOCKET_TMP_DIR =
                    "org.newsclub.net.unix.library.tmpdir";
    private static final String PROP_NET_BEANS_TMP_DIR = "cnd.tmpbase";

    // Call it some time before org.newsclub.net.unix.NativeLibraryLoader
    // from "junixsocket-common" can be loaded.
    static void unifyBespokeTmpDir(boolean dependsOnJUnixSocket)
    {
        if (!dependsOnJUnixSocket)
            return;

        final String netBeansTmpDir = System.getProperty(
                        PROP_NET_BEANS_TMP_DIR);

        if (netBeansTmpDir != null)
            System.setProperty(PROP_J_UNIX_SOCKET_TMP_DIR,
                        netBeansTmpDir);
    }

    public static void main(String[] args)
    {
        System.setProperty(PROP_NET_BEANS_TMP_DIR, "/foo");
        unifyBespokeTmpDir(true);
        assert System.getProperty(PROP_J_UNIX_SOCKET_TMP_DIR, "")
            .equals(System.getProperty(PROP_NET_BEANS_TMP_DIR));
    }
}

Are you willing to submit a pull request?

Yes

Code of Conduct

Yes

neilcsmith-net commented 2 years ago

@matthiasblaesing can you have a look at this? #3713 changed things in this area in NB14. On the other hand, I have no idea why cnd.tmpbase should be expected to affect this?

matthiasblaesing commented 2 years ago

Behavioral changes could be the result of the update of junixsocket. For the property setting the property java.io.tmpdir should be enough (changes the temp dir for the VM). It could be checked if (an updated) jgit learned to use unix domain sockets via JEP-380, then no additional native library would be necessary. The junixsocket module could be checked, that the same library extraction as is done for JNA could also be done there.

The property cnd.tmpbase sound unlikely to be used in the core IDE, as CND is not yet merged.

zzzyxwvut commented 2 years ago

The property cnd.tmpbase comes from pre-Apache releases (see this and its introduction and that); so reliance on it is admissible on *nix hosts.

In fact, its value is still tested, if defined, otherwise java.io.tmpdir is tried, while host-related information is being collected in order to create a HostInfo instance. See UnixHostInfoProvider, a member of an Apache NetBeans 14 module org.netbeans.modules.dlight.nativeexecution.

It could be corroborated as follows. Launch Apache NetBeans 14 with these defined properties: -J-Dnativeexecution.support.logger.level=0 -J-Dcnd.tmpbase=/dev/null -J-Djava.io.tmpdir=/path/to/new/dir (Substitute the actual path to the newly-created directory for /path/to/new/dir.)

Go Window -> IDE Tools -> Terminal

Expect the following Error notification:

Failed to use /dev/null as a temporary directory on localhost.  Please, run IDE with -J-Dcnd.tmpbase="other base location" flag to specify   a directory on a remote host where you have enough space and write and execute permissions.

(However, netbeans/ide/bin/nativeexecution/hostinfo.sh would still fall back to probe /tmp and try to make a few /tmp/dlight_`logname`* directories.)

Looking at diffs for NativeLibraryLoader of junixsocket, version 2.2.1 and version 2.4.0, there are no changes touching org.newsclub.net.unix.library.tmpdir.

Looking at the history of jsch in general, curiously, the project acquired an optional dependency on junixsocket near the jsch-0.1.66 tagged release, which falls within for transition of this project from jsch-0.1.55 to jsch-0.1.72.

Methods of JUnixSocketFactory fire up a chain of constructors and class initialisation for junixsocket classes: AFUNIXSocketAddress -> AFSocketAddress -> NativeUnixSocket -> NativeLibraryLoader.

Now look closely at the top frames in stack traces above.

neilcsmith-net commented 2 years ago

Yes, the cnd.tmpbase property is used in dlight.nativeexecution inside the core IDE. But where is the relationship between that and the IDE's git support?

matthiasblaesing commented 2 years ago

So I have a PR ready, that should fix this. PR is referenced, a binary build is available here:

https://doppel-helix.eu/NetBeans-dev-dev-a27179c197747f3f5a3e5031135d856374385bf5-release.zip

I tested with a readonly tmpfs on linux and was able to reproduce the problem (retargetting java.io.tmpdir there) and also could verify, that pull works after this change.

HOWEVER this needs to be tested on mac OS and of course if there are more linux users willing to test, it would also be appreciated. On windows I assume the putty agent is more common, so it is less interesting there.

The test scenario is this: With this build you should still be able to do a git pull using an SSH based connection, that is authenticated using an SSH agent. This is basicly the whole use-case the unixdomain socket library has in NetBeans (communication with the SSH agent).ssh based git pull, that is authenticated using an ssh agent (that is the whole use case of the existence of the library)

@geertjanw if I remember correctly you are a mac OS user, so it would be great if you could give it a spin.

zzzyxwvut commented 2 years ago

The CLONING and FETCHING parts of the FAILURE step (see details in the "How to reproduce" section near the top) were applied to on a GNU/Linux Debian (testing/bookworm) box with the offered build in a single run (and FETCHING alone once more in another run) and with a proviso:

I confirm success.

zzzyxwvut commented 2 years ago

@neilcsmith-net Oh, the failing remote-repository-related operations of Git are circumstantial to this issue.

It could be argued that cnd.tmpbase does its job well and it is a tall order to expect it to attend to arbitrary (in a good sense) dependencies of the IDE.

Conversely, if a claim can be made that cnd.tmpbase is the supported alternative when java.io.tmpdir proves too accommodating (e.g. some libraries may require generation of temporary files, not all of them may require execution from a temporary-file directory); it stands to reason to rely on the assumption that the definition of cnd.tmpbase should suffice and its code would negotiate whatever java.io.tmpdir alternatives current dependencies of the IDE, direct and otherwise, may offer as their own (possibly should take no action for alternatives explicitly defined by users).

Perhaps, some note on its use should be targeted to *nix users and a warning about what issues they may face whenever a newer IDE version is rolled out.

neilcsmith-net commented 2 years ago

@zzzyxwvut it's literally the title of your issue! I'm trying to understand how the setting of cnd.tmpbase was ever relevant here, if so, how, and whether this is a regression between NetBeans 13 and 14.

matthiasblaesing commented 2 years ago

@neilcsmith-net - I looked at 13 and the difference is, that the old git integration used a unix domain socket implementation based on JNA and this is already covered by the same mechanism that is now used to load the junixsocket library after the referenced PR. As we never advertised, that you could run with a noexec temp, this is IMHO not a regression.

neilcsmith-net commented 2 years ago

Thanks @matthiasblaesing So this, as expected, has nothing to do with changes in behaviour of cnd.tmpbase then. Agree not a regression, but NetBeans has tended to favour loading native libs directly without extraction for this reason. So, given comments above, happy to merge the PR for NB15 but make sure you're happy with testing across OS with rc2 - I'll add a note to the notice.

zzzyxwvut commented 2 years ago

I have now tested the same set-up with Apache NetBeans 13 as with the offered build--Git operations succeed with or without the cnd.tmpbase definition. I should have done that at the beginning--my apologies for the caused confusion.