search

taoensso / faraday

Amazon DynamoDB client for Clojure
https://www.taoensso.com/faraday
Eclipse Public License 1.0
238 stars 84 forks source link

Terrible Write Performance with DynamoDB Local #51

Closed skbach closed 9 years ago

skbach commented 9 years ago

It's taking about 150 ms to put a single item into a table using the samples from the intro.md. I'm testing simply inputting a single string value.

The docs from amazon say that throughput is ignored for DynamoDB Local.

Is there anything I can do to speed this up? At this point, MongoDB is about 100x faster to insert one item. My data is rather large and will take months or years to import at this rate :).

ptaoussanis commented 9 years ago

Hi Scott, have you confirmed that this is a problem with Faraday specifically and not DDB / your DDB configuration?

Could you possibly try check your write performance against DDB local directly? Faraday is a very thin layer over the DDB API; barring bugs - it shouldn't significantly impact the perf you'd see against the Java API.

Disclaimer: I've never used DDB local myself.

ptaoussanis commented 9 years ago

Also, could you please clarify what intro.md is / what the samples look like?

tcoupland commented 9 years ago

Another idea might be to checkout faraday and run the tests in local mode. Seeing how fast they run might help narrow down the problem!

skbach commented 9 years ago

Hi,

I should've been more clear. I'm pretty certain the problem is NOT with Faraday, and maybe I shouldn't have polluted your issue tracker with Dynamo issues :). However, Amazon has been unresponsive and several other people online are reporting similar write speeds. I figured you had probably tested the code using Local, and was wondering if there was some configuration (either in the Client API or Dynamo Service) that could improve it.

By intro.md I mean the intro on this github page. I tested using the sample (far/put-item client-opts :my-table...etc.

I've since given up on dynamo and moved to mongo, and now my write performance is nearly 500 x faster (from 120 ms per item to 250 microseconds per item) on the same box. (even with the most conservative mongo writeconcern settings). That's why I figured there must be something trivially wrong with my dynamo setup.

Thanks.

ptaoussanis commented 9 years ago

I would guess that this is a config issue, but I'm not sure. In any case, I'd note that DDB local is (afaik) intended for development and testing purposes; not for use as a production database itself. It's conceivable to me that they'd intentionally even simulate some network latency by default, but you or I would need to look at the docs to check.

I'd add that even if it worked as a production local DB, you'd be nullifying most of the point of using DDB in the first place. If you're not looking for the scaling / services aspect of DDB, then better choices exist for a local DB. Not sure what your actual requirements are, but you may also want to investigate Riak and PostgreSQL. I'm not a fan of MongoDB, but that's certainly also an option if you're satisfied with it and it fits your particular requirements.

Best of luck, cheers! :-)