search

adobe-apiplatform / api-gateway-aws

AWS SDK for NGINX with Lua
Apache License 2.0
171 stars 44 forks source link

How to use this with S3 or ES? #13

Open simonvanderveldt opened 8 years ago

simonvanderveldt commented 8 years ago

I have a question regarding how to use this with S3 (or other services that don't really have an action). I've been using the code as proposed in #12 with some changes and I've been able to requests plain text files from S3, but it doesn't really feel right, mainly because AwsService basically expects one to use performAction and because to make it work with other files I'd have to add all captured headers to the response, effectively creating something that somewhat resembles a proxy.

Wouldn't it make more sense to use this library only for the signing of the request and just proxy_pass the actual request? That would make the code simpler and fix the header issue as well.

Does anyone have any experience with this?

ddragosd commented 8 years ago

@simonvanderveldt you're bringing a good point to use S3 with proxy_pass. Did you manage to get this working ? It would be great to document it once we integrate #12.

When NGINX is used only to proxy directly to an AWS service this is not a bad idea.

For other cases( i.e. sending logs to Kinesis on log_by_lua or calling KMS on access_by_lua ) performAction can be used instead of proxy_pass as the backend would be different.

simonvanderveldt commented 8 years ago

@ddragosd sorry for the delay, needed some time to checkout the options and see how to implement it, was my first time using openresty :)

Anyway, we currently use AWS ElasticSearch service as well as S3 and we needed a proxy to sign our requests to ElasticSearch using v4 auth based on IAM roles/instance profiles. So I worked on that, though the actual implementation is pretty much the same as it would be for S3. It's basically:

I've commited what I have here https://github.com/simonvanderveldt/api-gateway-aws/tree/payjp/extend-get-authz-header based on the code that's in #12. I do think it would be relatively easy to implement this based on master as well, not sure the changes in #12 are actually needed.

I'd be interested to know what you think about it. I don't mind to make a PR to include it here, though I expect some things should be changed/enhanced. There were a couple of issues I ran into, the main one within the current code is that the URI path has to be encoded but the nginx included urlencode functions also encode the slashes which is not what AWS expects. So I added a simple/ugly string replace :x

hectcastro commented 7 years ago

I put together a working example of how to use the changes in #12 to proxy_pass inbound requests to Amazon S3 using the api-gateway.aws.AwsV4Signature module here:

https://github.com/azavea/docker-s3-proxy-cache

Docker is used to combine the Lua module dependencies with OpenResty into one package. From there, the majority of the heavy lifting is contained within the access_by_lua_block of the default Nginx virtual host configuration:

https://github.com/azavea/docker-s3-proxy-cache/blob/develop/usr/local/openresty/nginx/conf/conf.d/default.conf#L28-L62