search

hazelcast / hazelcast-simulator

A tool for stress testing Hazelcast
Apache License 2.0
87 stars 74 forks source link

Hazelcast Simulator. Cannot connect to Agents. #690

Closed demonhunterg closed 9 years ago

demonhunterg commented 9 years ago

From @hamanhlam on June 23, 2015 3:45

Environment: Windows 7. Hazelcast simulator 0.5. Using Cygwin sshd to remote agents. Hi. I am trying to install hazelcast simulator following this article http://docs.hazelcast.org/docs/3.4/manual/html/installingsimulator.html.

At last step i ran this command to try connecting my agent instance in localhost:

$ ssh -o BatchMode=yes gitpm@10.8.15.126 "echo ok" 2>&1
output:
Permission denied (publickey,password,keyboard-interactive).

After that, I try this command: $ ssh -v gitpm@10.8.15.126 output:

OpenSSH_6.8p1, OpenSSL 1.0.2c 12 Jun 2015
debug1: Reading configuration data /etc/ssh_config
debug1: Connecting to 10.8.15.126 [10.8.15.126] port 22.
debug1: Connection established.
debug1: identity file /home/gitpm/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitpm/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitpm/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitpm/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitpm/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitpm/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitpm/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitpm/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.8
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.8
debug1: match: OpenSSH_6.8 pat OpenSSH* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr umac-64-etm@openssh.com none
debug1: kex: client->server aes128-ctr umac-64-etm@openssh.com none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:xMwO83PwDmOKGui/Q5oUQqa/ofyqSN0PNODCHcuZXnw
debug1: Host '10.8.15.126' is known and matches the ECDSA host key.
debug1: Found key in /home/gitpm/.ssh/known_hosts:1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/gitpm/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
Enter passphrase for key '/home/gitpm/.ssh/id_rsa':
debug1: Authentication succeeded (publickey).
Authenticated to 10.8.15.126 ([10.8.15.126]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
Last login: Tue Jun 23 09:23:40 2015 from 10.8.15.126

And run test:

gitpm@gitpm /cygdrive/d/development/hazelcast-simulator-0.5/my-tests $ ./run.sh

output:

D:\development\hazelcast-simulator-0.5
INFO  11:15:19 Hazelcast Simulator Coordinator
INFO  11:15:19 Version: 0.5, Commit: c6e82c5, Build Time: 14.05.2015 @ 13:46:17 CEST
INFO  11:15:19 SIMULATOR_HOME: D:\development\hazelcast-simulator-0.5
INFO  11:15:19 Loading simulator.properties: D:\development\hazelcast-simulator-0.5\my-tests\simulator.properties
INFO  11:15:19 Loading testsuite file: D:\development\hazelcast-simulator-0.5\my-tests\test.properties
INFO  11:15:19 Loading Hazelcast configuration: D:\development\hazelcast-simulator-0.5\my-tests\..\conf\hazelcast.xml
INFO  11:15:19 Loading Hazelcast client configuration: D:\development\hazelcast-simulator-0.5\my-tests\..\conf\client-hazelcast.xml
INFO  11:15:19 Loading Log4j configuration for worker: D:\development\hazelcast-simulator-0.5\conf\worker-log4j.xml
INFO  11:15:19 Loading agents file: D:\development\hazelcast-simulator-0.5\my-tests\agents.txt
INFO  11:15:19 HAZELCAST_VERSION_SPEC: outofthebox
INFO  11:15:19 --------------------------------------------------------------
INFO  11:15:19 Waiting for agents to start
INFO  11:15:19 --------------------------------------------------------------
WARN  11:15:40 Failed to connect to public address: 10.8.15.126 sleeping for 1 second and trying again

Something wrong? Help me plz. Thanks.

Copied from original issue: hazelcast/hazelcast#5560

demonhunterg commented 9 years ago

From @pveentjer on June 23, 2015 4:58

Which USER did you configure in your simulator.properties? If you have not changed anything, then it is 'simulator' but you wan to use 'gitpm'

And where are your target machines running? 10.8.15.126 looks like an ec2 ip address.

demonhunterg commented 9 years ago

Thank you. I have changed some of the information in simulator config:

GROUP_NAME=group_test
USER=gitpm

10.8.15.126 that is my ip not ec2 ip address. I want to test in local, I created SSH server on my machine, and setup the public/private key pair in order to simulate for a local machine remote another machine. In 10.8.15.126 (my machine) I have cluster with 2 members 10.8.15.126:5702 and 10.8.15.126:5703.

demonhunterg commented 9 years ago

From @mdogan on June 23, 2015 7:39

@hamanhlam, @pveentjer;

Can you move this issue to simulator repo (https://github.com/hazelcast/hazelcast-simulator)?

Donnerbart commented 9 years ago

We don't officially support Windows with Simulator, at least we don't test it yet.

For your connection issue I assume that you don't have an SSH agent running which forwards the key to SSH during the script execution. Don't get confused with the term agent here, this time it is not about the Simulator component Agent ;)

Please have a look how to setup and SSH agent for cygwin, e.g. this guide: http://blog.killtheradio.net/how-tos/ssh-agent-on-cygwin/

Donnerbart commented 9 years ago

From your log I think I'm right about the missing SSH agent:

Enter passphrase for key '/home/gitpm/.ssh/id_rsa':
debug1: Authentication succeeded (publickey).

It looks like you have to unlock your key by typing in the passphrase. This is not supported by Simulator (and not practical, since a lot of different SSH connections are used to upload all files and start the Simulator Agents).

Also from your answer it seems that you already have Hazelcast nodes running? This is not needed. This is exactly what Simulator does: Start a cluster with member and client instances and run tests on them. There is no need to start Hazelcast instances on your own. Quite the contrary: it is not supported out-of-the-box by Simulator to connect e.g. clients to an existing cluster.

demonhunterg commented 9 years ago

Yes, I have reinstalled "login ssh without passphrase".

gitpm@gitpm /usr/bin
$ ssh -v -o BatchMode=yes gitpm@10.8.15.126
OpenSSH_6.8p1, OpenSSL 1.0.2c 12 Jun 2015
debug1: Reading configuration data /etc/ssh_config
debug1: Connecting to 10.8.15.126 [10.8.15.126] port 22.
debug1: Connection established.
debug1: identity file /home/gitpm/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitpm/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitpm/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitpm/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitpm/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitpm/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitpm/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitpm/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.8
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.8
debug1: match: OpenSSH_6.8 pat OpenSSH* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr umac-64-etm@openssh.com none
debug1: kex: client->server aes128-ctr umac-64-etm@openssh.com none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:xMwO83PwDmOKGui/Q5oUQqa/ofyqSN0PNODCHcuZXnw
debug1: Host '10.8.15.126' is known and matches the ECDSA host key.
debug1: Found key in /home/gitpm/.ssh/known_hosts:1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/gitpm/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: Authentication succeeded (publickey).
Authenticated to 10.8.15.126 ([10.8.15.126]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
Last login: Wed Jun 24 17:41:00 2015 from 10.8.15.126

Run Test:

gitpm@gitpm /cygdrive/d/development/hazelcast-simulator-0.5/my-tests
$ ./run.sh
D:\development\hazelcast-simulator-0.5
INFO  17:56:02 Hazelcast Simulator Coordinator
INFO  17:56:02 Version: 0.5, Commit: c6e82c5, Build Time: 14.05.2015 @ 13:46:17 CEST
INFO  17:56:02 SIMULATOR_HOME: D:\development\hazelcast-simulator-0.5
INFO  17:56:02 Loading simulator.properties: D:\development\hazelcast-simulator-0.5\my-tests\simulator.properties
INFO  17:56:02 Loading testsuite file: D:\development\hazelcast-simulator-0.5\my-tests\test.properties
INFO  17:56:02 Loading Hazelcast configuration: D:\development\hazelcast-simulator-0.5\my-tests\..\conf\hazelcast.xml
INFO  17:56:02 Loading Hazelcast client configuration: D:\development\hazelcast-simulator-0.5\my-tests\..\conf\client-hazelcast.xml
INFO  17:56:02 Loading Log4j configuration for worker: D:\development\hazelcast-simulator-0.5\conf\worker-log4j.xml
INFO  17:56:02 Loading agents file: D:\development\hazelcast-simulator-0.5\my-tests\agents.txt
INFO  17:56:02 HAZELCAST_VERSION_SPEC: outofthebox
INFO  17:56:02 --------------------------------------------------------------
INFO  17:56:02 Waiting for agents to start
INFO  17:56:02 --------------------------------------------------------------
Failed to connect to public address: 10.8.15.126 sleeping for 1 second and trying again

Log file:

INFO  2015-06-24 17:56:02,692 [main] com.hazelcast.simulator.coordinator.Coordinator: Hazelcast Simulator Coordinator
INFO  2015-06-24 17:56:02,692 [main] com.hazelcast.simulator.coordinator.Coordinator: Version: 0.5, Commit: c6e82c5, Build Time: 14.05.2015 @ 13:46:17 CEST
INFO  2015-06-24 17:56:02,692 [main] com.hazelcast.simulator.coordinator.Coordinator: SIMULATOR_HOME: D:\development\hazelcast-simulator-0.5
DEBUG 2015-06-24 17:56:02,692 [main] com.hazelcast.simulator.common.SimulatorProperties: Loading default simulator.properties from: D:\development\hazelcast-simulator-0.5\conf\simulator.properties
INFO  2015-06-24 17:56:02,732 [main] com.hazelcast.simulator.common.SimulatorProperties: Loading simulator.properties: D:\development\hazelcast-simulator-0.5\my-tests\simulator.properties
INFO  2015-06-24 17:56:02,742 [main] com.hazelcast.simulator.coordinator.CoordinatorCli: Loading testsuite file: D:\development\hazelcast-simulator-0.5\my-tests\test.properties
INFO  2015-06-24 17:56:02,742 [main] com.hazelcast.simulator.coordinator.CoordinatorCli: Loading Hazelcast configuration: D:\development\hazelcast-simulator-0.5\my-tests\..\conf\hazelcast.xml
INFO  2015-06-24 17:56:02,742 [main] com.hazelcast.simulator.coordinator.CoordinatorCli: Loading Hazelcast client configuration: D:\development\hazelcast-simulator-0.5\my-tests\..\conf\client-hazelcast.xml
INFO  2015-06-24 17:56:02,742 [main] com.hazelcast.simulator.utils.FileUtils: Loading Log4j configuration for worker: D:\development\hazelcast-simulator-0.5\conf\worker-log4j.xml
INFO  2015-06-24 17:56:02,742 [main] com.hazelcast.simulator.coordinator.Coordinator: Loading agents file: D:\development\hazelcast-simulator-0.5\my-tests\agents.txt
INFO  2015-06-24 17:56:02,742 [main] com.hazelcast.simulator.coordinator.Coordinator: HAZELCAST_VERSION_SPEC: outofthebox
INFO  2015-06-24 17:56:02,762 [main] com.hazelcast.simulator.coordinator.remoting.AgentsClient: --------------------------------------------------------------
INFO  2015-06-24 17:56:02,762 [main] com.hazelcast.simulator.coordinator.remoting.AgentsClient: Waiting for agents to start
INFO  2015-06-24 17:56:02,762 [main] com.hazelcast.simulator.coordinator.remoting.AgentsClient: --------------------------------------------------------------
DEBUG 2015-06-24 17:56:03,773 [main] com.hazelcast.simulator.coordinator.remoting.AgentClient: Failed to connect to public address: 10.8.15.126 sleeping for 1 second and trying again
INFO  2015-06-24 17:57:02,862 [main] com.hazelcast.simulator.coordinator.remoting.AgentsClient: Connect to agent 10.8.15.126 FAILED
DEBUG 2015-06-24 17:57:02,862 [main] com.hazelcast.simulator.coordinator.remoting.AgentsClient: java.io.IOException: Couldn't connect to publicAddress: 10.8.15.126:9000

Thanks for your help!

Donnerbart commented 9 years ago

Can you login to the remote machine and see if there is a java process running under the user you specified? If so the agent is running and you have a connection issue. Couldn't connect to publicAddress: 10.8.15.126:9000 looks like this.

This may be a firewall setting (e.g. Java process is not allowed to connect to the internet in the Windows firewall OR your company firewall blocks that port). You can also try to use telnet to see if you can connect to the agent.

demonhunterg commented 9 years ago

You're right! Problems caused by firewall. Thank you very much.