Hyperfoil / qDup

Lab automation and queuing scripting
Apache License 2.0
12 stars 12 forks source link

Do not hide important ssh exceptions behind non default logging levels #175

Open galderz opened 1 year ago

galderz commented 1 year ago

SSH java client library exceptions can be hidden behind debug messages which hinders qDup's usage. Example:

I was able to ssh the host in the command line without any issues but qDup didn't like it:

/opt/java-17/bin/java  -jar /opt/qDup/target/qDup-0.6.17-SNAPSHOT-uber.jar /opt/qdup-scripts/helloworld.yaml
18:59:20.280 [main] INFO  qdup._tmp_20230201_185920 - Running qDup version 0.6.17-SNAPSHOT @ 271e49b
18:59:20.412 [main] DEBUG qdup._tmp_20230201_185920.state - run-1675274360238 starting state:
 message = Hello, qDup!

18:59:20.495 [qdup-command-1] ERROR io.hyperfoil.tools.qdup.Run - failed to connect g@leo:22, verify password-less ssh works with the selected keys
18:59:20.496 [qdup-command-0] ERROR qdup._tmp_20230201_185920 - failed to connect all ssh sessions for run
18:59:20.497 [qdup-command-2] DEBUG qdup._tmp_20230201_185920.state - run-1675274360238 closing state:
 message = Hello, qDup!
 ENV = {"ABORTED":"true"}
 leo :
   tryme :
     id=7 :

Finished in 00.088 at /tmp/20230201_185920
make: *** [Makefile:31: helloworld] Error 1

Nothing in the above was hinting at the real issue. It's not clear how to enable debug/trace so I stuck the debugger and there's an actual exception coming from ssh java library hidden by:

        } catch (GeneralSecurityException | IOException e) {
            //e.printStackTrace();
            logger.debug("Exception while connecting to {}@{} using {}\n{}", host.getUserName(), host.getHostName(), identity, e.getMessage(), e);
        } finally {

With the debugger I discovered that the error is actually this:

Failed (IllegalArgumentException) to process line #19 (User galderz,infinispan): Multiple target users N/A: galderz,infinispan

The issue is this line that goes back quite a while in my ssh config but crucially is defined in a server that I have not touched in a long long time. Unless you're connecting to that server, I don't think the ssh java client should bother at all (just like command line ssh):

User galderz,infinispan

But what it's true is that qDup should be showing these things and not hiding them behind not by default logging levels.

galderz commented 1 year ago

Side note: I'm not sure the ssh java client is being used right. The ssh client does read the ~/.ssh/config so why do you need to look for a default identity file? I ask because once I fix the config file, I see this hidden exception in the same place:

java.io.FileNotFoundException: ${HOME}/.ssh/id_rsa (No such file or directory)
galderz commented 1 year ago

My .ssh/config has all it needs to connect to the machine and this proven by the fact that I can just do ssh <machine> and it works (no passwords asked no nothing).

willr3 commented 1 year ago

You're right, we need to do a better job of raising exceptions to the user. We had hoped to raise meaningful error messages rather than the underlying java exceptions but that is clearly incomplete so instead we will log the exception message as error to console.

We do not have testing to verify use of /.ssh/ config outside of what exists on our laptops. We added the ability to specify a different identity file but reading from /.ssh/config needs to be improved.

Do you mind sharing the config that caused the error (redacted of course) so I can recreate it and try to create an automated test around it?

galderz commented 1 year ago

What caused the error is in the description. One of the host entries had a line with User galderz,infinispan which failed. You can only have one user.