search

jepsen-io / jepsen

A framework for distributed systems verification, with fault injection
6.81k stars 719 forks source link

[docker] rabbitmq test fails with "rabbitmq-server: unrecognized service" #160

Open mbsimonovic opened 7 years ago

mbsimonovic commented 7 years ago

from jepsen-control docker container:

$ cd rabbitmq && lein test :only jepsen.rabbitmq-test


lein test jepsen.rabbitmq-test
INFO  jepsen.os.debian - :n5 setting up debian
INFO  jepsen.os.debian - :n2 setting up debian
INFO  jepsen.os.debian - :n1 setting up debian
INFO  jepsen.os.debian - :n4 setting up debian
INFO  jepsen.os.debian - :n3 setting up debian
INFO  jepsen.os.debian - Installing #{man}
INFO  jepsen.os.debian - Installing #{man}
INFO  jepsen.os.debian - Installing #{man}
INFO  jepsen.os.debian - Installing #{man}
INFO  jepsen.os.debian - Installing #{man}
INFO  jepsen.rabbitmq - :n2 Nuking rabbit
INFO  jepsen.rabbitmq - :n4 Nuking rabbit
INFO  jepsen.rabbitmq - :n5 Nuking rabbit
INFO  jepsen.rabbitmq - :n3 Nuking rabbit
INFO  jepsen.rabbitmq - :n1 Nuking rabbit
INFO  jepsen.rabbitmq - Fetching deb package
INFO  jepsen.rabbitmq - Fetching deb package
INFO  jepsen.rabbitmq - Fetching deb package
INFO  jepsen.rabbitmq - Installing rabbitmq
INFO  jepsen.rabbitmq - Installing rabbitmq
INFO  jepsen.rabbitmq - Installing rabbitmq
INFO  jepsen.rabbitmq - :n2 Nuking rabbit
INFO  jepsen.rabbitmq - :n1 Nuking rabbit
INFO  jepsen.rabbitmq - :n3 Nuking rabbit
INFO  jepsen.rabbitmq - :n4 Nuking rabbit
INFO  jepsen.rabbitmq - :n5 Nuking rabbit
INFO  jepsen.rabbitmq - :n3 Rabbit dead
INFO  jepsen.rabbitmq - :n1 Rabbit dead

lein test :only jepsen.rabbitmq-test/rabbit-test

ERROR in (rabbit-test) (FutureTask.java:122)
Uncaught exception, not in assertion.
expected: nil
  actual: java.util.concurrent.ExecutionException: java.lang.RuntimeException: rabbitmq-server: unrecognized service

...

Caused by: java.lang.RuntimeException: rabbitmq-server: unrecognized service
 at jepsen.control$throw_on_nonzero_exit.invoke (control.clj:105)
    jepsen.control$exec_STAR_.doInvoke (control.clj:121)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invoke (core.clj:624)
    jepsen.control$exec.doInvoke (control.clj:135)
    clojure.lang.RestFn.invoke (RestFn.java:436)
    jepsen.rabbitmq$reify__8631.teardown_BANG_ (rabbitmq.clj:97)
    jepsen.db$eval5744$fn__5745$G__5734__5749.invoke (db.clj:4)
    jepsen.db$eval5744$fn__5745$G__5733__5754.invoke (db.clj:4)
    clojure.lang.AFn.applyToHelper (AFn.java:160)
    clojure.lang.AFn.applyTo (AFn.java:144)
    clojure.core$apply.invoke (core.clj:626)
    clojure.core$partial$fn__4228.doInvoke (core.clj:2468)
    clojure.lang.RestFn.invoke (RestFn.java:421)
    jepsen.core$on_nodes$fn__7547.invoke (core.clj:85)
    clojure.core$pmap$fn__6328$fn__6329.invoke (core.clj:6466)
    clojure.core$binding_conveyor_fn$fn__4145.invoke (core.clj:1910)
    clojure.lang.AFn.call (AFn.java:18)
    java.util.concurrent.FutureTask.run (FutureTask.java:266)
    java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1142)
    java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617)
    java.lang.Thread.run (Thread.java:745)

hm, rabbitmq is installed only on n1,n2 and n3, but isn't it supposed to be install on all 5 nodes?

mbsimonovic commented 7 years ago

adding :nodes [:n1 :n2 :n3 :n4 :n5] to rabbitmq-test makes it try to start rabbitmq on 4 nodes?! btw how can I enable DEBUG logging level?

vutruongqa commented 5 years ago

Same issue above, anyone help please?

tvhphuong commented 5 years ago

I have the same issue, but when run again. It stuck at another step root@control:/jepsen/rabbitmq# lein test jepsen.rabbitmq-test

lein test jepsen.rabbitmq-test

INFO jepsen.os.debian - :n5 setting up debian INFO jepsen.os.debian - :n4 setting up debian INFO jepsen.os.debian - :n3 setting up debian INFO jepsen.os.debian - :n2 setting up debian INFO jepsen.os.debian - :n1 setting up debian INFO jepsen.os.debian - Installing #{man} INFO jepsen.os.debian - Installing #{man} INFO jepsen.os.debian - Installing #{man} INFO jepsen.os.debian - Installing #{man} INFO jepsen.os.debian - Installing #{man} INFO jepsen.rabbitmq - :n4 Nuking rabbit INFO jepsen.rabbitmq - :n1 Nuking rabbit INFO jepsen.rabbitmq - :n3 Nuking rabbit INFO jepsen.rabbitmq - :n2 Nuking rabbit INFO jepsen.rabbitmq - :n5 Nuking rabbit INFO jepsen.rabbitmq - Fetching deb package INFO jepsen.rabbitmq - Installing rabbitmq INFO jepsen.rabbitmq - :n4 Rabbit dead INFO jepsen.rabbitmq - :n2 Rabbit dead INFO jepsen.rabbitmq - :n1 Rabbit dead INFO jepsen.rabbitmq - Setting cookie INFO jepsen.rabbitmq - Setting cookie INFO jepsen.rabbitmq - uploading config INFO jepsen.rabbitmq - uploading config INFO jepsen.rabbitmq - Starting rabbitmq INFO jepsen.rabbitmq - Starting rabbitmq

Do you know why it has different issues while starting the test ?

aphyr commented 5 years ago

I haven't worked on the rabbit tests in 3 years, but my guess is that whatever version of Debian (and possibly Jepsen) you're using no longer runs the rabbitmq service in the same way that it did circa 2015.

vutruongqa commented 5 years ago

@aphyr, Can you suggest any tool to test rabbitmq that is easier to read than Jepsen(clojure)? I have spent a week to read on that but still can't understand how it works. If you have a document that guide step by step, please help to share. Thanks much for your answer

vjuranek commented 5 years ago

If you have a document that guide step by step, please help to share.

have you checked tutorial?

aphyr commented 5 years ago

Sorry @vutruongqa, I don't really know any other tool that does what Jepsen does. At least, not in this degree of depth. There are a few Jepsen clones out there but they're usually oriented towards testing one particular system, or just inducing faults without correctness testing.

aphyr commented 5 years ago

Oh, and as for support, well, there's a lot of documentation in Jepsen. The README links to the API docs, architecture overview, there are extensive code comments and docstrings, a comprehensive tutorial (as @vjuranek noted), and there are several free resources for learning Clojure; Clojure for the Brave and True is quite nice.

As far as bitrot goes, I'd love to spend some time bringing the RabbitMQ tests back up to speed with modern Jepsen, but my backlog right now is measured in years, not days. :-(

If you'd like commercial support, there are also engineers who work on Jepsen on a contract basis, and they might have some availability to help in your testing. I understand if you're budget constrained, though.