jetoile / hadoop-unit

Hadoop-Unit is a project which allow testing projects which need hadoop ecosysteme like kafka, solr, hdfs, hive, hbase, ...
https://blog.jetoile.fr/hadoop-unit/
Apache License 2.0
52 stars 15 forks source link

Cassandra EmbeddedServer starup timeout hardcoded to 10 secs #38

Closed scalaster closed 8 years ago

scalaster commented 8 years ago

My core i7 laptop with 8GB RAM never manages to bootstrap Cassandra within 10 secs when I run a full stack integration test. Is there a workaround? If not, I can change it to get the value from default.properties

jetoile commented 8 years ago

Hello,

I just released a newer version of hadoop-unit which use achille instead of cassandra-unit.

Which version did you use?

scalaster commented 8 years ago

Hi

Thanks for coming back so quickly I was trying with 1.3 Now refactored for 1.4 because its not backward compatible. But with that I have other problems

Embedded Cassandra now fails to start and all components shutdown as a result of that exception But Achilles keeps polling every 4, 8, 16, etc seconds and does not let the test die.

Is there another version I can try?

jetoile commented 8 years ago

Hello,

Can you tell me more on your test/environment please? On wich OS are you running and what is the exception?

Thx

scalaster commented 8 years ago

Hi

Thanks for coming back to me

I am running on -- dont judge me -- Windows 10 at the moment

I get the exception after it has executed the test method

java.io.IOException: Unable to delete file: embedded_cassandra\embeddedCassandra\system_schema\views-9786ac1cdd583201a7cdad556410c985\ma-1-big-Index.db

Achilles doesn't die at the end of the test. It keeps printing this message on the console [Achilles Embedded Cassandra Cluster-reconnection-1] ERROR com.datastax.driver.core.ControlConnection - [Control connection] Cannot connect to any host, scheduling retry in 4000 milliseconds

It does this in 4, 8, 16, 32 seconds.

Thanks, -- S

On Wed, Aug 3, 2016 at 7:19 PM, jetoile notifications@github.com wrote:

Hello,

Can you tell me more on your test/environment please? On wich OS are you running and what is the exception?

Thx

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jetoile/hadoop-unit/issues/38#issuecomment-237323499, or mute the thread https://github.com/notifications/unsubscribe-auth/AAQaUgiSneHUCvm4dz3JIREpbwnsC5Ukks5qcNu_gaJpZM4JblEX .

jetoile commented 8 years ago

Hi,

Sorry for the delay. Is it possible to share your code? I will try to test it on a windows computer.

Thanks,

Khanh

scalaster commented 8 years ago

Hi

Sorry, just seeing your message

Thank you for offering to have a look Attached is my simple integration test. I did the producer in a thread because I couldn't get producer to start after spark streaming context

Thanks, -- S

On Sat, Aug 6, 2016 at 4:05 PM, jetoile notifications@github.com wrote:

Hi,

Sorry for the delay. Is it possible to share your code? I will try to test it on a windows computer.

Thanks,

Khanh

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jetoile/hadoop-unit/issues/38#issuecomment-238027690, or mute the thread https://github.com/notifications/unsubscribe-auth/AAQaUit5l89-2oPnYa0OU9ewoTkCu2Rxks5qdKK5gaJpZM4JblEX .

jetoile commented 8 years ago

Hello,

I did not receive your code. Could you make a gist or open a git repo? Thx.

Khanh

scalaster commented 8 years ago

Hi

Just got back from holiday. I wrote an end-to-end test but its still a work in progress. Here are the 2 files

https://gist.github.com/scalaster/3d8fbe16cf71811a29558e2587cce71e https://gist.github.com/scalaster/ec9f5151987f36f67f84df017b165f59

Thanks, S

jetoile commented 8 years ago

Hi,

I just push your code on the branch cassandra-sample (module : sample/spark-streaming-cassandra). I did not use embedded hadoop-unit because of jar conflict but it works well with a standalone hadoop-unit. (I tested on linux).

Tell me if it is ok for you. If you agree I can integrate your code into the master branch.

Khanh

jetoile commented 8 years ago

the code has been merged into trunk. Close this issue