smillidge
commented
5 years ago There's not a lot of information to go on for this ticket. Can you tell us how to reproduce the issue?
Closed TinusJ closed 5 years ago
smillidge
commented
5 years ago There's not a lot of information to go on for this ticket. Can you tell us how to reproduce the issue?
TinusJ
commented
5 years ago @smillidge Full stack trace added for instance creation as well as instance start. Added steps to reproduce. NB: This is on windows using OpenSSH
smillidge
commented
5 years ago Please try again on Payara 5.191 as there have been fices in the executor service that in the stack trace is rejecting tasks
smillidge
commented
5 years ago Closing due to inactivity. Please comment if this is fixed in later releases for you.
thirion
commented
1 year ago I've been experiencing this exact same issue. I've tried Payara 5.192, 5.2021.8, 5.2022.5, 6.2023.2 and they all have this problem. They're all able to successfully create nodes and instances from Linux to another Linux machine, but not when the remote server is Windows. I used VirtualBox with a Windows Server 2019 vhd file downloaded from Microsoft, set up OpenSSH and used keys for passwordless connections.
I checked out the source for 5.2022.5 and the cause of the problem is in InstallNodeSshCommand.java line 189 during scpClient.put().
FileUtils.quoteString(sshInstallDir) wraps the path in single quotes which breaks on Windows.
To fix it you could replace that line with:
scpClient.put(zipFile.getAbsolutePath(), "\"" + sshInstallDir.replace("\"", "\\\"") + "\"");
I couldn't get the whole Payara project to build successfully to test this, so I wrote a small sample application just to test the core of what Payara is doing and confirmed that the above worked on Windows for files with single quotes and on Linux for files with single or double quotes. Windows doesn't allow file names with double quotes.
To quickly test this, try to create a node on a remote windows machine. It should always fail due to the path being wrapped in single quotes.
To prove this is the problem, specify a file path that includes a single quote, in which case FileUtils.quoteString(sshInstallDir) will wrap it in double quotes instead, which works for the copy, but will fail for the extract command on line 200 because it directly wraps the command in single quotes. You will get an error that jar command is not found.
You can manually run the extract command on the remote server to continue testing, but it will break when creating an instance if it has a single quote in it, but you get the point.
It could be as simple as updating FileUtils.quoteString(sshInstallDir) to just wrap using double quotes as a first resort but I have no idea if that could break existing functionality. I'll keep trying to build Payara so I can just replace all scpClient.put calls with my path fix. That should be the least risk and work for my purposes.
Thanks again for a great product.
mirozitnansky
commented
1 year ago I've enabled logging on windows sftp-server.exe and this doesn't seems like problem with path.
8232 2023-04-13 17:43:55.415 close "c:/payara6/glassfish/nodes/kiosk-node-ssh/kiosk-inst-ssh-2/config/cacerts.p12" bytes read 0 written 415632
8232 2023-04-13 17:43:55.415 debug3: request 1071: sent status 0
8232 2023-04-13 17:43:55.415 sent status Success
8232 2023-04-13 17:43:55.416 debug3: request 1072: stat
8232 2023-04-13 17:43:55.416 stat name "c:/payara6/glassfish/nodes/kiosk-node-ssh/kiosk-inst-ssh-2/config/domain.xml"
8232 2023-04-13 17:43:55.416 debug1: request 1072: sent attrib have 0xf
8232 2023-04-13 17:43:55.416 debug1: request 1073: setstat name "c:/payara6/glassfish/nodes/kiosk-node-ssh/kiosk-inst-ssh-2/config/domain.xml"
8232 2023-04-13 17:43:55.416 set "c:/payara6/glassfish/nodes/kiosk-node-ssh/kiosk-inst-ssh-2/config/domain.xml" size 75585
8232 2023-04-13 17:43:55.416 set "c:/payara6/glassfish/nodes/kiosk-node-ssh/kiosk-inst-ssh-2/config/domain.xml" mode 100607
8232 2023-04-13 17:43:55.416 set "c:/payara6/glassfish/nodes/kiosk-node-ssh/kiosk-inst-ssh-2/config/domain.xml" modtime 19700101-01:00:00
8232 2023-04-13 17:43:55.416 set "c:/payara6/glassfish/nodes/kiosk-node-ssh/kiosk-inst-ssh-2/config/domain.xml" owner 0 group 0
8232 2023-04-13 17:43:55.416 debug3: request 1073: sent status 4
8232 2023-04-13 17:43:55.416 sent status FailureHi mirozitnansky. In file InstallNodeSshCommand.java they create an SFTPClient and an SCPClient. Only the SCPClient quotes the path and causes the issue.
mirozitnansky
commented
1 year ago Heelo @thirion code you are pointing to, is called with command asadmin install-node-ssh. That should copy whole payara archive to host and extract it, without domain/nodes dirs. There could be error you are describing.
But log you've pasted in your first command, points to specific line
Caused by: com.trilead.ssh2.SFTPException: Failure (SSH_FX_FAILURE: An error occurred, but no specific error code exists to describe the failure.) at com.trilead.ssh2.SFTPv3Client.expectStatusOKMessage(SFTPv3Client.java:556) at com.trilead.ssh2.SFTPv3Client.setstat(SFTPv3Client.java:584) at com.sun.enterprise.v3.admin.cluster.SecureAdminBootstrapHelper$SSHHelper.setLastModified(SecureAdminBootstrapHelper.java:405)
That is method setLastModified in SecureAdminBootstrapHelper.java:405
where com.trilead.ssh2.SFTPException ftpClient.setstat(path, attrs); returns an exception.
I have same error when using asadmin create-node-ssh (not install-node-ssh) command, or admin gui for node creation (without installation flag, payara was copied before node creation). So only node and instance folders are created.
And this setstat method of sftp client, produces error on server side on Win32 OpenSSH sftp server, in log pasted yesterday.
I was able to trace this to server function process_setstat(u_int32_t id) and it seem like problem is with setting user id 0, but that's is only my assumptions.
For now, I am understanding this as incompatibility between com.trilead.ssh2.sftp and Win32 OpenSSH sftp-server.
And that's major issue, when DCOM implementation is no longer working.
lennartkoester
commented
1 year ago It looks like I got the deployment group running, using the cygwin openssh server instead of windows' openssh server
mirozitnansky
commented
1 year ago That was supported since glassfish 3. But it's much more hassle to install cygwin than MS openssh. But yes, it's alternative to DCOM not working. On the other hand, DCOM should be removed from admin if it won't be supported.
Description
Getting the following issue when trying to create a remote instance on a remote node.
Additional failure info: com.sun.enterprise.v3.admin.cluster.SecureAdminBootstrapHelper$BootstrapException: com.trilead.ssh2.SFTPException: Failure (SSH_FX_FAILURE: An error occurred, but no specific error code exists to describe the failure.).
Expected Outcome
Successfully created instance Tinus in the DAS configuration, & succesfully install configuration and needed files on the remote
Current Outcome
SSH configuration information
Additional failure info: com.sun.enterprise.v3.admin.cluster.SecureAdminBootstrapHelper$BootstrapException: com.trilead.ssh2.SFTPException: Failure (SSH_FX_FAILURE: An error occurred, but no specific error code exists to describe the failure.). Command create-instance completed with warnings.
Steps to reproduce (Only for bug reports)
Full Stack Trace for 3.1
Full Stack trace for 4 - starting server
Samples
Environment