search

jepsen-io / jepsen

A framework for distributed systems verification, with fault injection
6.77k stars 713 forks source link

jepsen nemesis crashes while running test for etcd #239

Open malhar1995 opened 6 years ago

malhar1995 commented 6 years ago

Hello everyone,

I tried running lein run test --concurrency 10 for etcd. Initially, it ran without any problems. But currently, every time I run it, it says, WARN [2018-03-13 15:40:02,787] jepsen nemesis - jepsen.core Process :nemesis crashed

java.lang.IllegalArgumentException: No implementation of method: :heal! of protocol: #'jepsen.net/Net found for class: nil

I'm not sure how to fix this.

Moreover, is there a mailing list for jepsen? If there is, I think posting queries on the mailing list would be better rather than posting them here.

P.S. I tried removing jepsen and setting it up again only to find that the first time I run the aforementioned test on etcd, nemesis still crashes but it says, Everything looks good! but on subsequent runs, it says, Analysis invalid!.

Thank you.

aphyr commented 6 years ago

As of this morning, there is a mailing list: https://groups.google.com/a/jepsen.io/forum/#!forum/talk

The error you're seeing sounds to me like you changed something in the source--possibly setting the test's :net key to nil.

malhar1995 commented 6 years ago

Initially, even I thought that I might have mistakenly changed some source code and so, I downloaded the jepsen repository again and the same problem persists.

Also, on a different note, I am currently studying your analyses on different versions of MongoDB and I wish to run the tests for each version and see it for myself. However, I am unable to correctly point out those tests in the repository. More specifically, I want to run tests for MongoDB 2.4.3, 2.6.7 and 3.4.0-rc3. What do the directories mongodb-rocks and mongodb-smartos stand for? Also, there is one repository named mongodb inside jepsen-io. Does it contain tests for MongoDB 3.4.0-rc3?

I could not find steps to reproduce the results for MongoDB in the documentation or the blog posts.

aphyr commented 6 years ago

However, I am unable to correctly point out those tests in the repository.

I'm guessing you're reading the blog posts, since you're trying to reproduce them. The 2.4.3 code is ancient, but you can dig it up in the git history if you like. 2.6.7 has direct links throughout the blog post to the source at the specific Git SHA's. Both of these predate having nice command line runners, so you'll be editing the source and running lein test by hand.

What do the directories mongodb-rocks and mongodb-smartos stand for?

Mongo with RocksDB, and Mongo on SmartOS. You can google for these terms, by the way.

I could not find steps to reproduce the results for MongoDB in the documentation or the blog posts.

It's the first thing in the README for the current MongoDB test repo. That repo (and the appropriate Git SHA) are linked throughout the 3.4 analysis.

Does it contain tests for MongoDB 3.4.0-rc3?

To answer your question, I suggest looking at the links in the 3.4.0-rc3 analysis.

aphyr commented 6 years ago

Ah, right, you've also got a mysterious stacktrace still: "java.lang.IllegalArgumentException: No implementation of method: :heal! of protocol: #'jepsen.net/Net found for class: nil". You're going to want to inspect the state of your test's :net field. The etcd test doesn't change it, IIRC, so it should be the default iptables network; there's probably something else strange at play here, but since I can't reproduce this problem myself you're gonna have to debug it on your own. :(

mkcp commented 6 years ago

I've bumped into that error once or twice in a docker dev env but never been able to reproduce it -- it usually goes away on the next run. I'll check in if I run into it again

kaibimouez commented 4 years ago

Hello @aphyr , i changing the ip adress of my ec2 aws instance and i have this problem: No implementation of method: :heal! of protocol: #'jepsen.net/Net found for class: nil I think because of the changing of the IP Address. How can i change the new Address in Iptables? Thx a lot Mouez Kaibi

aphyr commented 4 years ago

Sorry, @kaibimouez, I don't know; I've never seen this before. As I noted previously, you'll want to look at the stacktrace and see where Jepsen is trying to call (heal! nil ...), and add some logging: it might be that the test's net object is nil somehow.

kaibimouez commented 4 years ago

grafik I think you have talked about : https://github.com/jepsen-io/jepsen/issues/239#issuecomment-373439203