Question: accessing aws-sdk

[danielytics]

This is probably a dumb question, but I've been searching for a solution for a few hours now without luck.

I'm trying to access the S3 API, so that I can manipulate S3 objects from my lambda function and I can't figure out how to do this.

I've tried (def aws (nodejs/require "aws-sdk")) and (def s3 (nodejs/require "aws-sdk/clients/s3")) and while they seem to give me something back, I'm unsure what to do with this.

I've tried:

(def S3 (nodejs/require "aws-sdk/clients/s3"))
(def s3-client (new S3))

and this works in the sense that s3-client is now a non-nil object, but I've been unable to call functions on it.

eg

(.copyObject s3-client #js {:Bucket "bucket" :CopySource "bucket/source-key" :Key "destination-key"})

but it complains that s3-client does not have a copyObject field.

I then tried to use the cljsjs package for aws-sdk, but still no luck.

I assume this is standard nodejs interop from clojurescript, but this is my first time using nodejs (or lambda) from clojurescript and I can't seem to figure it out. What am I missing?

Any and all help is greatly appreciated!!

[moea]

@danielytics No problem, this stuff is awkward at first. If you're using a version of the AWS SDK which supports the /clients/...-style require (which you will be if it's not throwing an exception), then your code should work - this works for me in a Clojurescript REPL against the most recent published aws-sdk in npm:

(def S3 (nodejs/require "aws-sdk/clients/s3"))
(def s3-client (new S3))

Is this this failing on Lambda only? What optimizations level is your project using? If it was an issue with advanced mode compilation renaming copyObject at the call site, I would expect it to work with the cljsjs guy, which includes externs (I'm assuming, I haven't used it). Trying it with none optimisations (which I'd recommend generally, unless you have a specific reason not to, or aren't depending on any straight-JS code) would be worthwhile, if applicable.

[danielytics]

Thanks very much. I was testing it only with lein cljs-lambda deploy, so I'll try it in local tests/repl and with optimisations none. I would also expect that if it was a renaming issue that the cljsjs library should fix it.

I've used plenty of js from cljs in the past without much issue, just not with nodejs or lambda, so I thought perhaps there was some detail I'm missing, but it seems there isn't. Thanks again for the quick reply; I'll report back once I've tried the above in case its useful to others (although it will be tomorrow morning utc)

[moea]

@danielytics FWIW I have a bunch of code deployed with cljs-lambda, using S3 via aws-sdk w/ optimizations none, as, I think, do others - so I'm more than happy to help in whatever way I can. If optimizations was advanced when you tested, it definitely wouldn't work locally or on Lambda - and the issue with cljsjs may just be some unrelated thing.

[danielytics]

Great, thanks! I imagine that's the case then: I need to use optimisations none and the cljsjs issue was unrelated. I'll confirm in the morning.

[danielytics]

Ok, that took a lot longer to get working than it should have... I have it working now though! Thanks again for the help, @moea.

Here's all of the mistakes I made:

• Compiled with default settings (which includes advanced optimisation). Much easier to diagnose errors with optimisations turned off. So, at least while learning, I would recommend that if anyone has trouble, remember to turn this off.
• Didn't actually have aws-sdk installed locally! npm install aws-sdk fixed this and suddenly the functions were no longer undefined. I assume this is why the cljsjs package didn't work. I still have no idea why the AWS and S3 objects seemed to exist anyway. Presumably its a nodejs thing? "empty" module maybe? Who knows.
• Now the functions existed and the calls went through without error, but I wasn't seeing anything actually happen. The problem was quite simple: my lambda function was returning before the calls completed! Wrapping a call inside p/promise and resolving it in the AWS API callback fixed this:

(p/promise
  (fn [resolve]
    (.listObjects s3-client params
      (fn [e d] (resolve {:msg "Done" :data d :error e})))))

• This worked in simple tests, but my original code still failed. It turns out that it was because I set my lambda function up to trigger on S3 CreateObject calls and then iterated through the list of records using doseq... which meant that I wasn't returning the promise I was creating and my lambda function still exited too soon. I replaced doseq with the following so that processing each record can independently make calls and the lambda waits for all of them (I swapped promises for core.async to make this easier):

(async/map vector
  (for [record records]
    (process-record! record))

Each call to process-record! would return a channel. The AWS API callbacks simply put true onto their channel. I also successfully made multiple calls by giving each call its own channel and returning a vector of these channels and then calling flatten on the result of the for. There may be a cleaner way (for example, creating a promise for each call and then awaiting them maybe?)

Hopefully this is useful to somebody else!

[moea]

For my part, I'll update the template to use none optimizations.

[danielytics]

Or just make add a note in the README.