search

eligosource / eventsourced

A library for building reliable, scalable and distributed event-sourced applications in Scala
Apache License 2.0
828 stars 98 forks source link

dynamodb-journal #78

Closed sclasen closed 11 years ago

sclasen commented 11 years ago

clean version of dynamodb journal on top of current master

krasserm commented 11 years ago

Hi Scott,

thanks for the pull requests and your awesome work on the dynamodb journal. Before merging it to master we still need to deal with the Travis CI failure. I propose to run the dynamodb tests with the sbt integration test configuration in the same way as I did for hbase (see this commit). This will then run the dynamodb and hbase tests with sbt it:test but not with the default sbt test which is executed on Travis and my own build server. Once we fixed the hbase and dynamodb issues on Travis we can also include it:test to run there.

krasserm commented 11 years ago
sclasen commented 11 years ago

Hey Martin

sclasen commented 11 years ago

@krasserm all set afaik

ironfish commented 11 years ago

Hi Martin, Hi Scott,

Just wanted to throw my 2 cents into the mix. I'm a big test freak and I think moving some of the more complicated test to IT makes a lot of sense. I do think though (as Martin pointed out) that eventually we should get Travis to include the IT test as part of the build. Maybe not for experimental Journal's but definitely for Production ones.

WDYT?

krasserm commented 11 years ago

@ddevore +1 on that. If there are no automated integration tests running, a journal shouldn't be tagged as production ready. The situation for DynamoDB is a bit more involved though, because running DynamoDB tests introduces actual AWS costs, because there's no DynamoDB development version for local deployment available. @sclasen what is your estimate for the costs of a single integration test run i.e. launching the AWS images with DynamoDB, running the DynamoDBJournal spec as well as the eventsourced-core-tests without the stress tests (once running with all journals)? Maybe we could run integration tests once a day and the 'normal' tests with every commit ... and we'll need to answer the question who is willing to sponsor that.

Thoughts?

stigkj commented 11 years ago

Hi,

Not sure how good it is as I have not tried it myself, but here is a link to a fake DynamoDB store implementation: https://github.com/ananthakumaran/fake_dynamo.

krasserm commented 11 years ago

@stigkj many thanks for the pointer, looks useful. Will take a closer looks soon.

Cheers, Martin

sclasen commented 11 years ago

We dont necessarily have to run the integration tests from an aws instance (other than stress testing), so running from travis would be fine. We would just need a dynamo table with sufficient throughput to not throttle during tests (not sure what that # is), and to have travis have access to a set of encrypted IAM credentials that has access to only that table.

The actual cost for time to run tests against a table like that would be less than $0.01. But creating and tearing down a table would take a bit of time and require more powerful IAM credentials. The easier way would be to leave the table there which would be about $0.07/hr,

AWS does have a free tier for dynamodb too, so if the tests could run within those limits (5 write 10 read throughput) then the dynamo cost would be zero.

+1 I'm big on integration tests.

krasserm commented 11 years ago

@sclasen have you tried running DynamodbJournalSpec against the free tier? Sounds like a good starting point. Maybe this timeout needs to be increased for that.

sclasen commented 11 years ago

I'll give it a shot, it's probably not timeout related but more of will running the tests cause throttling or errors on a table with a free level of throughput.

Sent from my iPhone

On Mar 23, 2013, at 3:04 AM, Martin Krasser notifications@github.com wrote:

@sclasen have you tried running DynamodbJournalSpec against the free tier? Sounds like a good starting point. Maybe this timeout needs to be increased for that.

— Reply to this email directly or view it on GitHub.

sclasen commented 11 years ago

:+1: The DynamoDBJournalSpec seems to run fine on a free level table.

krasserm commented 11 years ago

Good news, thanks for trying it out. I created #79 to track that.