Support DynamoDB Local

[jmglov]

From the DynamoDB Local page:

DynamoDB Local is a small client-side database and server that mimics the DynamoDB service. DynamoDB Local enables you to write applications that use the DynamoDB API, without actually manipulating any tables or data in DynamoDB.

It would be great if Amazonica could support using DynamoDB Local, probably by doing something like this:

(require '[amazonica.aws.dynamodbv2 :as dynamo])
(dynamo/use-local! true)

@mcohen01, if you agree in principle with this feature request, I'll try to send a pull request in the next week or so that implements it.

[caryfitzhugh]

I believe I had gotten it working recently just by setting a different endpoint in the creds I used for each call. And then spinning up the DDB-local service. I think there is a docker image with ddb-local to make it even easier.

But there was something that stunk about it - I don't remember. Maybe ddb-local was so far behind or outdated, or something. I wish I could remember, but there was something that was so stinky, we decided to just hold off on incorporating DDB for now (since we didn't need it and didn't want to pay for a real DDB for the next few months while we are building...)

On Thu, Feb 5, 2015 at 7:26 AM, Josh Glover notifications@github.com wrote:

From the DynamoDB Local page http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Tools.DynamoDBLocal.html :

DynamoDB Local is a small client-side database and server that mimics the DynamoDB service. DynamoDB Local enables you to write applications that use the DynamoDB API, without actually manipulating any tables or data in DynamoDB.

It would be great if Amazonica could support using DynamoDB Local, probably by doing something like this:

(require '[amazonica.aws.dynamodbv2 :as dynamo]) (dynamo/use-local! true)

@mcohen01 https://github.com/mcohen01, if you agree in principle with this feature request, I'll try to send a pull request in the next week or so that implements it.

— Reply to this email directly or view it on GitHub https://github.com/mcohen01/amazonica/issues/118.

[jmglov]

@caryfitzhugh Interesting. Thanks for the info. I'll have a quick look at the credentials stuff to see if I can figure that out.

I'm not too fussed about paying for test DynamoDB instances, since the first 25 units of read and write capacity are free, but it would be nice to develop on the train. :)

[caryfitzhugh]

I will look for more details when i get to my office and can see my code.

On Thu, Feb 5, 2015 at 8:10 AM, Josh Glover notifications@github.com wrote:

@caryfitzhugh https://github.com/caryfitzhugh Interesting. Thanks for the info. I'll have a quick look at the credentials stuff to see if I can figure that out.

I'm not too fussed about paying for test DynamoDB instances, since the first 25 units of read and write capacity are free, but it would be nice to develop on the train. :)

— Reply to this email directly or view it on GitHub https://github.com/mcohen01/amazonica/issues/118#issuecomment-73043297.

[mcohen01]

I've never used the Local db, and haven't used Dynamo in a while, so you guys would be the best ones to make whatever changes are needed.

[caryfitzhugh]

I looked, and couldn't find much. My git commit rate was lower then when I was trying so many different things.

Looks like I successfully got it to create tables and things of that sort, so - that's encouraging. I know that I added :endpoint into the creds, and the endpoint was an obvious url , like http://127.0.0.1:3801

Hope that helps.

On Fri, Feb 6, 2015 at 12:07 PM, Michael Cohen notifications@github.com wrote:

I've never used the Local db, and haven't used Dynamo in a while, so you guys would be the best ones to make whatever changes are needed.

— Reply to this email directly or view it on GitHub https://github.com/mcohen01/amazonica/issues/118#issuecomment-73273173.

[Integralist]

I'm using DynamoDB Local via Spurious and have issues with the Amazonica library, in the sense that I want to set the region to be eu-west-1 rather than use the default value used by DynamoDB Local which is (I think?) us-east-1

The problem is that (as far as I know) Amazonica doesn't allow you to set both the region and an endpoint within the credentials map passed into a function. Is that definitely the case? It sounds like that when reading the following text within the README...

the credentials map may contain an :endpoint entry. If the value of the :endpoint key is a lower case, hyphenated translation of one of the Regions enums, .setRegion will be called on the Client, otherwise .setEndpoint will be called

...because it suggests that the :endpoint key either calls setRegion or setEndpoint.

So I'm currently using...

(def credentials {:access-key "development_access"
                  :secret-key "development_secret"
                  :endpoint "dynamodb.spurious.localhost:49155"
                  :client-config {:protocol "http"}})

(create-table credentials
              :table-name "testing"
              :key-schema [{:attribute-name "component_key" :key-type "HASH"}
                           {:attribute-name "batch_version" :key-type "RANGE"}]
              :attribute-definitions [{:attribute-name "component_key" :attribute-type "S"}
                                      {:attribute-name "batch_version" :attribute-type "N"}]
              :provisioned-throughput {:read-capacity-units 10
                                       :write-capacity-units 10})

...which means the region stays as the default and the table is created in an account which is in the wrong region; as I want the table created in the eu-west-1 region.

But then, I can see in a few places the use of a :region key (here and in the README (ec2/create-image {:region "us-east-1"} :instance-id "i-1b9a9f71")).

I tried using the :region key and it made no difference. So I looked through the Amazonica source code and found AWS_DEFAULT_REGION which I tried to set to eu-west-1 in both my terminal shell AND in a .env file using the lein-dotenv plugin. But neither worked to get DynamoDB Local to switch to that region.

For example, I tried...

(def credentials {:access-key "development_access"
                  :secret-key "development_secret"
                  :endpoint "dynamodb.spurious.localhost:49155"
                  :region "eu-west-1"
                  :client-config {:protocol "http"}})

...notice the addition of the :region key. But like I say, that didn't work and it looks like it's still creating tables within the default region instead.

[Integralist]

Ps, I also tried adding :region as a separate key from the credentials map...

(create-table credentials
              :region "eu-west-1"
              :table-name "test5"
              :key-schema [{:attribute-name "component_key" :key-type "HASH"}
                           {:attribute-name "batch_version" :key-type "RANGE"}]
              :attribute-definitions [{:attribute-name "component_key" :attribute-type "S"}
                                      {:attribute-name "batch_version" :attribute-type "N"}]
              :provisioned-throughput {:read-capacity-units 10
                                       :write-capacity-units 10})

[mcohen01]

@Integralist sorry, the docs you pointed to with a :region key were incorrect (fixed now). There is no :region key, only an :endpoint. And yes, when you pass an :endpoint key it ultimately results in either setRegion or setEndpoint being called on the client. But this basically makes no difference, the Java calls essentially do the same thing. I don't know anything about DynamoDB Local, but if you were using the full blown service specifying :endpoint "dynamodb.us-west-1.amazonaws.com" would have the exact same effect as specifying :endpoint "us-west-1", which is to say, you'd be using DynamoDB in the US_WEST_1 region.

I don't know anything about Spurious, but I would try setting the :endpoint to something like "dynamodb.eu-west-1.spurious.localhost:49155".

[jmglov]

This thread and the commentary in #121 have shown me that Amazonica supports DynamoDB Local already, but (possibly) only through the credentials map. I don't use an explicit map, as I need the config set by Elastic Beanstalk, and I don't want my keys in the code anyway. :)

I'll see if I can make this work through environment variables and JDK system properties, and either implement support for that or close this issue once I can verify that it already works.

[gws]

@Integralist Take a look at this thread, specifically this response - could setSignerRegionOverride be what you're looking for?

[Integralist]

@gws that looks good. But how would I call that from Clojure? as when I run (keys (ns-publics 'amazonica.aws.dynamodbv2)) from in a REPL I don't see anything resembling setSignerRegionOverride?

All I get back is:

(query batch-write-item set-endpoint create-table batch-get-item update-table get-item update-item put-item delete-item delete-table describe-table adjust-client-configuration list-tables scan)

[Integralist]

BTW if I try (set-signer-region-override "eu-west-1") then (as expected) it can't resolve the symbol as it looks like it's not being intern'ed by Amazonica.

Should Amazonica not intern all functions it reflects from the Java SDK? As demonstrated here and here

I assume the issue is that the version of the AWS Java SDK used by Amazonica is lower than the version that introduced the setSignerRegionOverride method?

[mcohen01]

You can't, you have no way to get a handle on the client. Did you try setting the endpoint to something like "dynamodb.eu-west-1.spurious.localhost:49155?"

These methods aren't interned because they're on the superclass. I can push a build out that does expose them but you should note that the jdk docs say those methods are for interval use only and not meant to be called by client code.

[mcohen01]

You can try the client-superclass-methods-exposed branch and see if that works. set-signer-region-override is available.

[Integralist]

@mcohen01 just to confirm that the suggestion of dynamodb.eu-west-1.spurious.localhost:49155 didn't work for me.

What's the best way to load a github branch of your library? I'm going to clone the repo, change branch and lein install it so that should mean it ends up in my ~/.m2 cache directory. But I'll ask just in case there is a better way

[Integralist]

Just an F.Y.I there was a warning message installing your branch (don't know if it's a problem or not)...

M. ~/Code/amazonica on client-superclass-methods-exposed [8fb2211]
± lein install
Compiling 1 source files to /Users/markmcdonnell/Code/amazonica/target/classes
warning: [options] bootstrap class path not set in conjunction with -source 1.6
1 warning
Created /Users/markmcdonnell/Code/amazonica/target/amazonica-0.3.15.jar
Wrote /Users/markmcdonnell/Code/amazonica/pom.xml
Installed jar and pom into local repo.

[Integralist]

@mcohen01 As per comment(s) above, I cloned the repo, checked out the branch and ran a lein install. From my library I jump into the repl and ran (keys (ns-publics 'amazonica.aws.dynamodbv2)) but that still only showed the following methods as being available...

(query batch-write-item set-endpoint create-table batch-get-item update-table get-item update-item put-item delete-item delete-table describe-table adjust-client-configuration list-tables scan) 

[mcohen01]

If you just run lein repl in the cloned dir what do you see output from ns-publics? On Feb 21, 2015 8:37 AM, "Mark McDonnell" notifications@github.com wrote:

@mcohen01 https://github.com/mcohen01 As per comment(s) above, I cloned the repo, checked out the branch and ran a lein install. From my library I jump into the repl and ran (keys (ns-publics 'amazonica.aws.dynamodbv2)) but that still only showed the following methods as being available...

(query batch-write-item set-endpoint create-table batch-get-item update-table get-item update-item put-item delete-item delete-table describe-table adjust-client-configuration list-tables scan)

Reply to this email directly or view it on GitHub https://github.com/mcohen01/amazonica/issues/118#issuecomment-75379513.

[Integralist]

@mcohen01 if I run (keys (ns-publics 'amazonica.aws.dynamodbv2)) then I get Exception No namespace: amazonica.aws.dynamodbv2 found clojure.core/the-ns (core.clj:3830) and if I run ns-publics I get #<core$ns_publics clojure.core$ns_publics@a8c880e>.

Only once (on my very first time of running lein repl and after the above errors) did I manage to get the list of methods exposed - even if I follow the precise same steps in a fresh lein repl I can't replicate that successful output?? - but when I did get it displayed on screen it included the relevant function set-signer-region-override...

(query notify with-endpoint batch-write-item with-region get-class set-region hash-code shutdown set-configuration create-table batch-get-item update-table get-signer-by-uri get-time-offset get-item to-string get-request-metrics-collector update-item put-item notify-all delete-item add-request-handler delete-table equals describe-table remove-request-handler list-tables set-time-offset with-time-offset set-signer-region-override get-signer-region-override wait set-service-name-intern scan get-service-name)

[Integralist]

I'm assuming that if I can't replicate that successful output more than once then that might be why it didn't show for me when I tried from within my libs repl originally

[Integralist]

@mcohen01 OK, figured out what was wrong. I noticed I was using version 0.3.13 and your latest version is 0.3.15. So I changed my project.clj to use the same version as your branch. I then was able to access the function required.

The following allowed me to see the correct set of tables...

(let [credentials (cred (resource :app))]
  (set-signer-region-override credentials "eu-west-1")
  (list-tables credentials))

...I didn't bother trying to create a table, because I'm assuming if I can correctly access the tables then I can trust the creating of tables will work fine as well.

Are you comfortable exposing these functions?

[Integralist]

@mcohen01 hmm, so although I can list the tables from the eu-west-1 table correctly (as seen in previous comment) I still can't create a table?

Here's what I tried to execute...

(create-table (cred (resource :app)) "table-name: foobar\nkey-schema:\n- attribute-name: component_key\n  key-type: HASH\n- attribute-name: batch_version\n  key-type: RANGE\nattribute-definitions:\n- attribute-name: component_key\n  attribute-type: S\n- attribute-name: batch_version\n  attribute-type: N\nprovisioned-throughput:\n  read-capacity-units: 10\n  write-capacity-units: 10\n\n")

Here's the error...

IllegalArgumentException Could not determine best method to invoke for create-table using arguments ({:client-config {:protocol "http"}, :access-key "development_access", :secret-key "development_secret", :endpoint "dynamodb.spurious.localhost:49155"} "table-name: foobar\nkey-schema:\n- attribute-name: component_key\n  key-type: HASH\n- attribute-name: batch_version\n  key-type: RANGE\nattribute-definitions:\n- attribute-name: component_key\n  attribute-type: S\n- attribute-name: batch_version\n  attribute-type: N\nprovisioned-throughput:\n  read-capacity-units: 10\n  write-capacity-units: 10\n\n")  amazonica.core/intern-function/fn--1532 (core.clj:806)

[Integralist]

@mcohen01 So I decided to go back to the 0.3.13 version to see if I was getting the same error as was showing in 0.3.15 (see previous comment). Oddly the same error was occurring for both versions of Amazonica. I thought this was strange, as I knew the code worked.

Once I changed the dependency back to 0.3.13 I decided to lein install (so it would be placed in my local cache ~/.m2) and to test it within my example application. I found my code still worked. Which was good.

So although the above error was occurring, it didn't affect the overall application working. So I decided maybe it was the same with the 0.3.15 version of Amazonica. Sadly, when running the 0.3.15 version from within my example application I then got the following error...

Caused by: java.lang.RuntimeException: Unable to resolve symbol: set-signer-region-override in this context

Now with the dependency back at 0.3.15 and this error occuring where it couldn't resolve the symbol set-signer-region-override; I decided to remove my code that referenced that symbol and instead changed it to print out the public keys for the dynamodbv2 namespace:

(print (keys (ns-publics 'amazonica.aws.dynamodbv2)))

Which printed the following within my application...

(query batch-write-item set-endpoint create-table batch-get-item update-table get-item update-item put-item delete-item delete-table describe-table adjust-client-configuration list-tables scan)

...this shows that indeed the method you used for exposes all methods (inc. non-public methods) doesn't seem to be getting picked up here; and correlates to my earlier tests in the repl where 99.9% of the time the new methods aren't exposed (and yet there was that .1% I was able to find the method whilst in the repl).

[Integralist]

It seems so strange that within a REPL in my helper library I can run the following with success...

(let [credentials (cred (resource :app))] 
  (set-signer-region-override credentials "eu-west-1") 
  (list-tables credentials))

But add in the create-table call...

(let [credentials (cred (resource :app))] 
  (set-signer-region-override credentials "eu-west-1") 
  (create-table credentials "table-name: abcdeftest-table3\nkey-schema:\n- attribute-name: component_key\n  key-type: HASH\n- attribute-name: batch_version\n  key-type: RANGE\nattribute-definitions:\n- attribute-name: component_key\n  attribute-type: S\n- attribute-name: batch_version\n  attribute-type: N\nprovisioned-throughput:\n  read-capacity-units: 10\n  write-capacity-units: 10\n\n") 
  (list-tables credentials))

...and then it fails...

IllegalArgumentException Could not determine best method to invoke for create-table using arguments ({:client-config {:protocol "http"}, :access-key "development_access", :secret-key "development_secret", :endpoint "dynamodb.spurious.localhost:49155"} "table-name: abcdeftest-table3\nkey-schema:\n- attribute-name: component_key\n  key-type: HASH\n- attribute-name: batch_version\n  key-type: RANGE\nattribute-definitions:\n- attribute-name: component_key\n  attribute-type: S\n- attribute-name: batch_version\n  attribute-type: N\nprovisioned-throughput:\n  read-capacity-units: 10\n  write-capacity-units: 10\n\n")  amazonica.core/intern-function/fn--1532 (core.clj:806)

And yet, if I just go ahead and lein install the helper and run my application with lein ring server I get...

Caused by: java.lang.RuntimeException: Unable to resolve symbol: set-signer-region-override in this context

[gws]

@Integralist A few things:

1. It'll be quite a bit easier to use Leiningen's checkout dependencies to test @mcohen01's branch. Just check out that branch somewhere and symlink it into checkouts in your project and you're good to go (see link for details). You should probably run lein clean in your helper library before any of this just to be safe. You should leave the version of amazonica in your library's project.clj as 0.3.15 (or whatever you had been using), to ensure the right dependencies are pulled in, because checkout dependencies aren't transitive.
2. I'm not sure about that create-table syntax, but I could be mistaken, I'm fairly new to the library as well. Try something more like this:

(dynamodb/create-table credentials
                       :table-name "abcdeftest"
                       :key-schema [{:attribute-name "id" :key-type "HASH"}]
                       :attribute-definitions [{:attribute-name "id" :attribute-type "S"}]
                       :provisioned-throughput {:read-capacity-units 5 :write-capacity-units 5}))

I hope some of that helps!

[mcohen01]

@integralist v0.3.16 exposes set-signer-region-override, and @gws is correct about syntax. Looks like you're passing one big string instead of key-value pairs.

[Integralist]

@mcohen01 hi, I'm not sure what's changed since your previous branch 0.3.15, but the code I was able to execute before (i.e. (let [credentials (cred (resource :app))] (set-signer-region-override credentials "eu-west-1") (list-tables credentials))) via the REPL now causes the following error...

UnknownHostException eu-west-1: unknown error  java.net.Inet6AddressImpl.lookupAllHostAddr (Inet6AddressImpl.java:-2)  

Along with the following error in stdout...

INFO: Unable to execute HTTP request: eu-west-1: unknown error
java.net.UnknownHostException: eu-west-1: unknown error

...where before, it actually connected fine and I was able to list the tables in my DynamoDB Local.

The problem before was just the create-table command. I'm going to go back and try the 0.3.15 branch but with the inline schema as demonstrated by @gws above (although I've been using the inline string schema loaded from a yaml file no problem up until this point)

[Integralist]

re: "I'm going to go back and try the 0.3.15 branch" -> scrap that. I removed the test branch @mcohen01 created from my laptop when trying to use 0.3.16 and I've just realised that since he published 0.3.16 he's removed that previous test branch. Hopefully @mcohen01 will know what was different between that test branch and what is now 0.3.16

[Integralist]

Looks like it's the (list-tables credentials) call that triggers the error.

The call (set-signer-region-override credentials "eu-west-1") simply returns nil

[mcohen01]

@Integralist sorry, can you try with 0.3.18?

[Integralist]

@mcohen01 Ladies and Gentlemen... we have a winner! :-D

Thanks everyone for your patience; we got there in the end.

[mcohen01]

glad to hear