search

bakdata / aws-lambda-r-runtime

Serverless execution of R code on AWS Lambda
https://medium.com/bakdata/running-r-on-aws-lambda-9d40643551a6
MIT License
143 stars 52 forks source link

How should complex json calls (i.e. from apigateway) be handled? #34

Closed DaveParr closed 5 years ago

DaveParr commented 5 years ago

In practice the call to the service is more complex than {'x':1}.

For instance sam has an example as follows:

Davids-MBP:~ davidparr$ sam local generate-event apigateway aws-proxy
{
  "body": "eyJ0ZXN0IjoiYm9keSJ9",
  "resource": "/{proxy+}",
  "path": "/path/to/resource",
  "httpMethod": "POST",
  "isBase64Encoded": true,
  "queryStringParameters": {
    "foo": "bar"
  },
  "pathParameters": {
    "proxy": "/path/to/resource"
  },
  "stageVariables": {
    "baz": "qux"
  },
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
    "Accept-Encoding": "gzip, deflate, sdch",
    "Accept-Language": "en-US,en;q=0.8",
    "Cache-Control": "max-age=0",
    "CloudFront-Forwarded-Proto": "https",
    "CloudFront-Is-Desktop-Viewer": "true",
    "CloudFront-Is-Mobile-Viewer": "false",
    "CloudFront-Is-SmartTV-Viewer": "false",
    "CloudFront-Is-Tablet-Viewer": "false",
    "CloudFront-Viewer-Country": "US",
    "Host": "1234567890.execute-api.us-east-1.amazonaws.com",
    "Upgrade-Insecure-Requests": "1",
    "User-Agent": "Custom User Agent String",
    "Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)",
    "X-Amz-Cf-Id": "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==",
    "X-Forwarded-For": "127.0.0.1, 127.0.0.2",
    "X-Forwarded-Port": "443",
    "X-Forwarded-Proto": "https"
  },
  "requestContext": {
    "accountId": "123456789012",
    "resourceId": "123456",
    "stage": "prod",
    "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef",
    "requestTime": "09/Apr/2015:12:34:56 +0000",
    "requestTimeEpoch": 1428582896000,
    "identity": {
      "cognitoIdentityPoolId": null,
      "accountId": null,
      "cognitoIdentityId": null,
      "caller": null,
      "accessKey": null,
      "sourceIp": "127.0.0.1",
      "cognitoAuthenticationType": null,
      "cognitoAuthenticationProvider": null,
      "userArn": null,
      "userAgent": "Custom User Agent String",
      "user": null
    },
    "path": "/prod/path/to/resource",
    "resourcePath": "/{proxy+}",
    "httpMethod": "POST",
    "apiId": "1234567890",
    "protocol": "HTTP/1.1"
  }
}

When this is held as a 'test' even in the console (and potentially therefore also when used in practice?) the script.R fails with:

Error in increment(body = myData  : 
  unused arguments (resource = "/{proxy+}", path = "/path/to/resource", httpMethod = "POST", isBase64Encoded = TRUE, queryStringParameters = list("bar"), pathParameters = list("/path/to/resource"), stageVariables = list("qux"), headers = list("text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", "gzip, deflate, sdch", "en-US,en;q=0.8", "max-age=0", "https", "true", "false", "false", "false", "US", "1234567890.execute-api.eu-west-2.amazonaws.com", "1", "Custom User Agent String", "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)", 
    "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==", "127.0.0.1, 127.0.0.2", "443", "https"), requestContext = list("123456789012", "123456", "prod", "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", "09/Apr/2015:12:34:56 +0000", 1428582896000, list(NULL, NULL, NULL, NULL, NULL, "127.0.0.1", NULL, NULL, NULL, "Custom User Agent String", NULL), "/prod/path/to/resource", "/{proxy+}", "POST", "1234567890", "HTTP/1
Calls: do.call
Execution halted

In some cases the structure can be predetermined, so in a toy case the increment function from the example becomes:

increment <- function(body, resource, path, httpMethod, isBase64Encoded, queryStringParameters, pathParameters, stageVariables, headers, requestContext) {
  ...
}

However there is no guarantee in lambda of the source of the json, or really it's wider structure. It could need to accept multiple trigger patterns. Is it correct to fail if the json isn't exactly what is expected, even if all the cruft (resource, path, httpMethod, isBase64Encoded, queryStringParameters, pathParameters, stageVariables, headers, requestContext) has no bearing on the calculation? Is it possible to absorb these unimportant arguments in the event currently?

philipp94831 commented 5 years ago

Hi @DaveParr, I already build an example on how to handle API Gateway requests: https://github.com/bakdata/aws-lambda-r-runtime/blob/master/tests/R/api.R The key is the .... You also need to update the layer because the old response format with result was unsuitable.

DaveParr commented 5 years ago

Perfect, I had assumed and tried ... but this was failing in arn:aws:lambda:eu-west-2:131329294410:layer:r-runtime-3_6_0:4, updated to arn:aws:lambda:eu-west-2:131329294410:layer:r-runtime-3_6_0:6 and it works like a charm.