search

FieryCod / holy-lambda

The extraordinary simple, performant, and extensible custom AWS Lambda runtime for Clojure.
https://fierycod.github.io/holy-lambda
MIT License
341 stars 20 forks source link

[FEATURE] Integration with AWS Step Functions #83

Open mcuervoe opened 2 years ago

mcuervoe commented 2 years ago

Is your feature request related to a problem? Please describe. When AWS Lambda is used in AWS Step Functions, AWS Step Functions uses the field "errorType" to implement catch and retry logic. When the Lambda is implemented in Java, Javascript, or Python, this field is generated when the code raises an exception.

In the case of Java, the "errorType" takes the name of the exception class.

See these documents for details:

Unfortunately, when the lambda function is implemented with holy-lambda, I have not seen a way to make this mechanism works:

(try
    (validate-request mode request)
    {:statusCode 200
     :headers    {"content-type" "application/json"}
     :body       {"message": "ok"}}
    (catch Exception e
        {:statusCode   500
         :errorType    (get (ex-data e) :error-type (type e))
         :errorMessage (.getMessage e)
         :headers      {}
         :body         (json/generate-string (ex-data e))})

In this case, the output gets wrapped in a field "output". AWS Step Functions receives something like:

{
  "output": "{\"statusCode\":500,\"errorType\":\"invalid-file-name\",\"errorMessage\":\"key is not valid file name, key=120290614_769f878d-696a-419a-b66c-91e642f7d0e3.zip\",\"headers\":{},\"body\":\"{\\\"error-type\\\":\\\"invalid-file-name\\\",\\\"key\\\":\\\"120290614_769f878d-696a-419a-b66c-91e642f7d0e3.zip\\\",\\\"mode\\\":\\\"archive\\\",\\\"env\\\":\\\"test\\\",\\\"id\\\":\\\"arn:aws:states:eu-west-1:593702161747:execution:EvidenceArchivingStepsTest:cabf4fe4-5b25-5df5-9502-4cc2d95a210c_b79f44d7-a610-ec98-f47c-3c79eddc9c0b\\\",\\\"bucket\\\":\\\"eid-evidence-test-input\\\",\\\"message\\\":\\\"key is not valid file name, key=120290614_769f878d-696a-419a-b66c-91e642f7d0e3.zip\\\"}\"}",
  "outputDetails": {
    "truncated": false
  }
}

In this case, AWS Steps Functions does not even consider this as an error (and therefore, the catch and retry clauses of the workflow can't be used)

Describe the solution you'd like I think it is needed a way to provide a errorType to AWS Step Functions when some error happens processing the lambda

Describe alternatives you've considered I see there are two possible alternatives:

  1. When the code throws an exception, ensure that "errorType" gets populated.
    • Pros:
    • Consistency with Java, Javascript, Python implementations
    • Cons:
    • We may break code that is based on previous behavior.
    • Creating custom java exception classes is necessary for a fine-tune workflow behavior mapping.
  2. Provide a mechanism to return "erroType" from a map
    • Pros:
    • No need to create custom java exception classes, so this is a much more idiomatic solution

Additional context Add any other context or screenshots about the feature request here.

FieryCod commented 2 years ago

Do you maybe have a simple repro with step functions? I'm happy to tune HL to make it work. What I can see is you're getting the error in the form of:

(defn- send-runtime-error
  [runtime iid ^Exception err disable-analytics?]
  (u/println-err! (u/->str "[holy-lambda] Runtime error:\n" (pr-str (Throwable->map err))))
  (let [response (u/http "POST" (url runtime iid "/error")
                         {:statusCode 500
                          :headers    {"content-type" "application/json"}
                          :body       (Throwable->map err)}
                         disable-analytics?)]
    (when-not (response :success?)
      (u/println-err! (u/->str "[holy-lambda] Runtime error sent failed.\n" (str (response :body))))
      (System/exit 1))))

I have to check the custom runtime design documents.

mcuervoe commented 2 years ago

I do not have this simple repo, but I will create one. I will post it here when I have it ready.

FieryCod commented 2 years ago

Thank you! Just simple "hello world" on AWS SAM + Step Functions exposing this issue would be enough 😀 Nothing too fancy please.

mcuervoe commented 2 years ago

Hi, I have created this repo:

https://github.com/mcuervoe/holy-lambda-steps

It is an implementation in Clojure/HL of the example in python that you can find in https://aws.amazon.com/getting-started/hands-on/handle-serverless-application-errors-step-functions-lambda/

You can follow the instruction in this article to invoke the step function with different inputs to generate different exceptions (Step 4, Test your Error Handling Workflow).

If you have any issues with the deployment of the code, let me know

Miguel

FieryCod commented 2 years ago

@mcuervoe please check the latest HL release (0.6.7). This should fix your issue :)

FYI: HL now supports both Clojure ExceptionInfo and Java classes. (https://github.com/FieryCod/holy-lambda/commit/842b5cf513ab98dfede405dfdc6054a09e7e5ecc)

{:errorMessage (.getMessage err)
 :errorType    (or
                 ;; You can throw an `(ex-info "SomeMessage {:type "CustomErrorType"})` to receive "error" "CustomErrorType"
                 (:type (ex-data err))
                 ;; As a fallback the ErrorName is inferred from the class
                 (.getName (.getClass ^Class err)))
 :stackTrace   (mapv str (.getStackTrace err))}

Please let me know if this solves your issue. Btw, thank you for the repro. Really well done and easy to follow!

mcuervoe commented 2 years ago

Hi @FieryCod ,

I tested the change, and it works beautifully. I updated the repo https://github.com/mcuervoe/holy-lambda-steps so that it accepts new status codes in the input to try the throw of ExceptionInfo with the field :type to determine the error type:

I think it can be used as an example of the integration of HL and Step Functions

I really like the new functionality of allowing ex-info to drive the workflow when errors occur.

Thanks, Miguel