Open DiscreteTom opened 1 month ago
Looks like the generated files are based on the aws-js-sdk
v2.751.0.
However the Lambda.invokeWithResponseStream
is added in v2.1353.0.
I don't think this SDK is auto-generated; it is written by hand. What APIs are you interested in?
cc @Tieske @windmgc
I think the SDK can be re-generated to add the missing REST shapes for request phase, however I don't think that response streaming will work immediately because it currently tries to "single shot" all requests using resty REST client which buffers the whole response to memory.
We would need to:
ngx.say
-ing each chunk back to the API caller/client. So two different issues for us now:
lua-resty-aws
does not support streaming response.@DiscreteTom Could you elaborate your use case with invokeWithResponseStream
?
Thanks for the update. I'm implementing streaming response for kong's lambda plugin.
I've modified the lib locally and tested it with kong, and the streaming response is indeed buffered (if I test it correctly).
So I'm planning to rewrite some parts of the lambda plugin to make it support response streaming. This lib will stil be used for sigv4 signing or other stuff.
SDK is updated and I see the new shape, I'm just waiting for push permission so I can open the PR. I will test and prove the theory in a minute - if correct (it buffers the whole response), I think I can easily make it return the res
object to the caller, optionally.
Problem obviously is then that the AWS Lambda plugin still needs updating to stream the chunked response to the Kong client.
-===-
New shape:
"InvokeWithResponseStream": {
"name": "InvokeWithResponseStream",
"http": {
"method": "POST",
"requestUri": "/2021-11-15/functions/{FunctionName}/response-streaming-invocations"
},
"input": {
"shape": "InvokeWithResponseStreamRequest"
},
"output": {
"shape": "InvokeWithResponseStreamResponse"
},
So yeah @DiscreteTom the best way is if I modify the request executor to return you the whole res
table, instead of just the buffered string.
That way, you can use the body_reader to stream directly back to the client in the Lambda Plugin. You basically just connect the "aws response" and "client request handle" streams together and let them run.
I've done this before on other things, if you need help.
@DiscreteTom Sorry I haven't refreshed the SDK yet, problems with GitHub permissions.
I should be able to do today.
In the meantime, here is the AWS Stream parser, in Lua format, that you can use to parse each Lambda chunk:
https://github.com/Kong/kong/blob/d875bacf80d3ff2657b33d7cb8863142ca1083a2/kong/tools/aws_stream.lua
You would use it like normal Lua package in the plugins:
local dump = function(...)
local t = { n = select("#", ...), ...}
if t.n == 1 and type(t[1]) == "table" then t = t[1] end
print(require("pl.pretty").write(t))
end
local chunk_in_hex = "000000760000005296d5fade0b3a6576656e742d7479706507000c6d65737361676553746172740d3a636f6e74656e742d747970650700106170706c69636174696f6e2f6a736f6e0d3a6d6573736167652d747970650700056576656e747b22726f6c65223a22617373697374616e74227d6fa8c599000001110000005706f176a90b3a6576656e742d74797065070011636f6e74656e74426c6f636b44656c74610d3a636f6e74656e742d747970650700106170706c69636174696f6e2f6a736f6e0d3a6d6573736167652d747970650700056576656e747b22636f6e74656e74426c6f636b496e646578223a302c2264656c7461223a7b2274657874223a2250692028cf80292069732061206d617468656d61746963616c20636f6e7374616e74207468617420726570726573656e7473207468652063697263756d666572656e63652d746f2d6469616d6574657220726174696f206f66206120636972636c652e204974277320617070726f78696d6174656c7920332e31343135392e227d7d7d4e31940000007d00000056e6680fd60b3a6576656e742d74797065070010636f6e74656e74426c6f636b53746f700d3a636f6e74656e742d747970650700106170706c69636174696f6e2f6a736f6e0d3a6d6573736167652d747970650700056576656e747b22636f6e74656e74426c6f636b496e646578223a307db32e4d340000007a00000051ca2c46650b3a6576656e742d7479706507000b6d65737361676553746f700d3a636f6e74656e742d747970650700106170706c69636174696f6e2f6a736f6e0d3a6d6573736167652d747970650700056576656e747b2273746f70526561736f6e223a22656e645f7475726e227d5fbf09fc000000b90000004ee991d99b0b3a6576656e742d747970650700086d657461646174610d3a636f6e74656e742d747970650700106170706c69636174696f6e2f6a736f6e0d3a6d6573736167652d747970650700056576656e747b226d657472696373223a7b226c6174656e63794d73223a313530367d2c227573616765223a7b22696e707574546f6b656e73223a382c226f7574707574546f6b656e73223a33392c22746f74616c546f6b656e73223a34377d7d5ad2dd7b"
local _STREAM = require("kong.tools.aws_stream")
local parser, err = _STREAM:new(chunk_in_hex, true)
if err then
print("ERROR: ", err)
return
end
while true do
local msg = parser:next_message()
if not msg then
break
end
print(dump(msg))
end
I will be adding this to this SDK so that it can be called directly, and each chunk can be parsed into an array of messages for the client response.
@ttyS0e I made some local changes to bypass the dependency issue so I can already test it locally for now. Thanks for the stream parser, it is very helpful!
I created a draft PR on kong: https://github.com/Kong/kong/pull/13171
The response streaming is working now. I use two files (aws_stream.lua and execute.lua) as a workaround to support the feature for now. You can reference these for the next release of resty-aws lib.
When the resty-aws library is ready I will remove these two files, modify the handler.lua to adopt this lib and finish the PR.
Nice, this is excellent! Thanks.
I will get this into the SDK and produce a new tag for you, I just need gateway team approval and such.
Even in the latest release (1.5.0-1) the
Lambda.invokeWithResponseStream
is missing. It is provided in theaws-js-sdk
, see https://github.com/aws/aws-sdk-js/blob/d5f58a06ed7e551658293f7fb555543945ed783b/apis/lambda-2015-03-31.normal.json#L1133