scalar-labs / scalar-jepsen

Jepsen tests for ScalarDB and ScalarDL
45 stars 6 forks source link

Add docker scripts #92

Closed yito88 closed 1 year ago

yito88 commented 1 year ago

Add docker files to set up a local test environment easily

Jepsen also has scripts for docker env. However, it doesn't support arm64, and some configurations aren't required for scalar-jepsen.

That's why I made docker scripts for scalar-jepsen.

brfrn169 commented 1 year ago

@yito88 Also, I tried the steps, but I faced the following error:

ERROR [2023-01-04 11:07:05,098] main - jepsen.cli Oh jeez, I'm sorry, Jepsen broke. Here's why:
clojure.lang.ExceptionInfo: throw+: {:dir "/", :private-key-path nil, :password "root", :username "root", :type :jepsen.control/session-error, :port 22, :strict-host-key-checking false, :host nil, :sudo nil, :dummy false, :message "Error opening SSH session. Verify username, password, and node hostnames are correct.", :session nil}
    at slingshot.support$stack_trace.invoke(support.clj:201)
    at jepsen.control.SSHRemote$fn__3132.invoke(control.clj:322)
    at jepsen.control.SSHRemote.connect(control.clj:319)
    at jepsen.control$session$fn__3147.invoke(control.clj:350)
    at jepsen.reconnect$open_BANG_$fn__2921.invoke(reconnect.clj:59)
    at jepsen.reconnect$open_BANG_.invokeStatic(reconnect.clj:57)
    at jepsen.reconnect$open_BANG_.invoke(reconnect.clj:54)
    at jepsen.control$session.invokeStatic(control.clj:349)
    at jepsen.control$session.invoke(control.clj:345)
    at clojure.lang.AFn.applyToHelper(AFn.java:154)
    at clojure.lang.AFn.applyTo(AFn.java:144)
    at clojure.core$apply.invokeStatic(core.clj:665)
    at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973)
    at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973)
    at clojure.lang.RestFn.applyTo(RestFn.java:142)
    at clojure.core$apply.invokeStatic(core.clj:669)
    at clojure.core$bound_fn_STAR_$fn__5749.doInvoke(core.clj:2003)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invokeStatic(core.clj:665)
    at clojure.core$apply.invoke(core.clj:660)
    at jepsen.util$fcatch$wrapper__2167.doInvoke(util.clj:39)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at dom_top.core$real_pmap_helper$build_thread__213$fn__214.invoke(core.clj:146)
    at clojure.lang.AFn.applyToHelper(AFn.java:152)
    at clojure.lang.AFn.applyTo(AFn.java:144)
    at clojure.core$apply.invokeStatic(core.clj:665)
    at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973)
    at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973)
    at clojure.lang.RestFn.invoke(RestFn.java:425)
    at clojure.lang.AFn.applyToHelper(AFn.java:156)
    at clojure.lang.RestFn.applyTo(RestFn.java:132)
    at clojure.core$apply.invokeStatic(core.clj:669)
    at clojure.core$bound_fn_STAR_$fn__5749.doInvoke(core.clj:2003)
    at clojure.lang.RestFn.invoke(RestFn.java:397)
    at clojure.lang.AFn.run(AFn.java:22)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.jcraft.jsch.JSchException: Auth fail
    at com.jcraft.jsch.Session.connect(Session.java:512)
    at com.jcraft.jsch.Session.connect(Session.java:183)
    at clj_ssh.ssh$fn__2617.invokeStatic(ssh.clj:118)
    at clj_ssh.ssh$fn__2617.invoke(ssh.clj:115)
    at clj_ssh.ssh.protocols$fn__2575$G__2542__2584.invoke(protocols.clj:4)
    at clj_ssh.ssh$connect.invokeStatic(ssh.clj:401)
    at clj_ssh.ssh$connect.invoke(ssh.clj:397)
    at jepsen.control$clj_ssh_session.invokeStatic(control.clj:311)
    at jepsen.control$clj_ssh_session.invoke(control.clj:298)
    at jepsen.control.SSHRemote$fn__3132.invoke(control.clj:320)
    ... 34 common frames omitted

It looks like ssh failed. Do I need any additional configurations other than the steps?

yito88 commented 1 year ago

Thank you for trying this! Ah, we need to specify an ssh private key on the ubuntu env. I'll fix the doc.

lein run test --test lwt --ssh-private-key ~/.ssh/id_rsa
brfrn169 commented 1 year ago

@yito88 I tried it with --ssh-private-key, but I faced the following error:

ERROR [2023-01-07 07:11:06,396] main - jepsen.cli Oh jeez, I'm sorry, Jepsen broke. Here's why:
clojure.lang.ExceptionInfo: throw+: {:dir "/", :private-key-path "/root/.ssh/id_rsa", :password "", :username "root", :type :jepsen.control/session-error, :port 22, :strict-host-key-checking false, :host nil, :sudo nil, :dummy false, :message "Error opening SSH session. Verify username, password, and node hostnames are correct.", :session nil}
    at slingshot.support$stack_trace.invoke(support.clj:201)
    at jepsen.control.SSHRemote$fn__3132.invoke(control.clj:322)
    at jepsen.control.SSHRemote.connect(control.clj:319)
    at jepsen.control$session$fn__3147.invoke(control.clj:350)
    at jepsen.reconnect$open_BANG_$fn__2921.invoke(reconnect.clj:59)
    at jepsen.reconnect$open_BANG_.invokeStatic(reconnect.clj:57)
    at jepsen.reconnect$open_BANG_.invoke(reconnect.clj:54)
    at jepsen.control$session.invokeStatic(control.clj:349)
    at jepsen.control$session.invoke(control.clj:345)
    at clojure.lang.AFn.applyToHelper(AFn.java:154)
    at clojure.lang.AFn.applyTo(AFn.java:144)
    at clojure.core$apply.invokeStatic(core.clj:665)
    at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973)
    at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973)
    at clojure.lang.RestFn.applyTo(RestFn.java:142)
    at clojure.core$apply.invokeStatic(core.clj:669)
    at clojure.core$bound_fn_STAR_$fn__5749.doInvoke(core.clj:2003)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invokeStatic(core.clj:665)
    at clojure.core$apply.invoke(core.clj:660)
    at jepsen.util$fcatch$wrapper__2167.doInvoke(util.clj:39)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at dom_top.core$real_pmap_helper$build_thread__213$fn__214.invoke(core.clj:146)
    at clojure.lang.AFn.applyToHelper(AFn.java:152)
    at clojure.lang.AFn.applyTo(AFn.java:144)
    at clojure.core$apply.invokeStatic(core.clj:665)
    at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973)
    at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973)
    at clojure.lang.RestFn.invoke(RestFn.java:425)
    at clojure.lang.AFn.applyToHelper(AFn.java:156)
    at clojure.lang.RestFn.applyTo(RestFn.java:132)
    at clojure.core$apply.invokeStatic(core.clj:669)
    at clojure.core$bound_fn_STAR_$fn__5749.doInvoke(core.clj:2003)
    at clojure.lang.RestFn.invoke(RestFn.java:397)
    at clojure.lang.AFn.run(AFn.java:22)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.jcraft.jsch.JSchException: invalid privatekey: [B@886cf34
    at com.jcraft.jsch.KeyPair.load(KeyPair.java:664)
    at com.jcraft.jsch.KeyPair.load(KeyPair.java:561)
    at com.jcraft.jsch.KeyPair.load(KeyPair.java:532)
    at clj_ssh.ssh$keypair.invokeStatic(ssh.clj:208)
    at clj_ssh.ssh$keypair.invoke(ssh.clj:169)
    at clj_ssh.ssh$add_identity.invokeStatic(ssh.clj:268)
    at clj_ssh.ssh$add_identity.invoke(ssh.clj:245)
    at jepsen.control$clj_ssh_session.invokeStatic(control.clj:303)
    at jepsen.control$clj_ssh_session.invoke(control.clj:298)
    at jepsen.control.SSHRemote$fn__3132.invoke(control.clj:320)
    ... 34 common frames omitted

I could ssh to n1, n2, n3, n4, and n5 without password from the control node, so it looks like the private key is valid. Umm...

yito88 commented 1 year ago

Thank you! It sounds like this issue: https://stackoverflow.com/questions/53134212/invalid-privatekey-when-using-jsch I'm not sure why it didn't happen on my machine before. It happens now. (OpenSSH was updated??) Anyway, I've added -m PEM when generating an ssh key.