Issues creating an Amazon Lambda function

chriscoomber commented 3 years ago

Hi there. I'm learning KMM and I have written a test project Kotlin library. I wanted to make a quick website which showed the documentation and also had a window for people to try out the library live. For this, I want to use something like the kotlin playground, with this as the backend. For the backend, I want to:

Add my library as a dependency this server (I think I did this well enough - I added mavenLocal() which I've installed my library to and I added my library as a dependency - 2 lines of build script total)
Create an AWS Lambda function so that my website (which doesn't exist yet) can make HTTP requests to this AWS function (via an AWS API Gateway).

I'm stuck on the latter. As an aside, it would be useful to have more documentation here for how I should configure my AWS function. For example, I think I needed to create a new AWS Lambda function (I did everything via the AWS website as the aws CLI wasn't working properly on my machine), using Java 8 runtime. Then I needed to upload the ZIP, which needs to be done by first creating an S3 bucket and uploading it to that. And then I need to add a trigger using AWS API Gateway, with HTTP rather than REST (right?), and then I wasn't sure about stages so I left that default. Also I wasn't sure about CORS so I checked that box because it sounded necessary. Maybe I'm a noob but not much of this was obvious.

Anyway, I've now created my Lambda function but when I run curl -X GET https://dthvag45o1.execute-api.eu-west-2.amazonaws.com/versions I get a 500 error:

< HTTP/2 500
< date: Sat, 27 Feb 2021 16:08:28 GMT
< content-type: application/json
< content-length: 35
< apigw-requestid: baYzbg_SLPEEMxA=
<
{"message":"Internal Server Error"}

I have no way to debug this. I've no idea what I did wrong. The Lambda function's dashboard is registering that it saw an error. Although, I get the same error when putting any string instead of versions.

Do you have any advice?

chriscoomber commented 3 years ago

Also, what's the difference between the server here and the one given by https://github.com/JetBrains/kotlin-web-demo? Edit: nevermind, I see now that kotlin-playground calls that the "old" server.

chriscoomber commented 3 years ago

Also, is Add Kotless and remove aws-serverless-container =) a very cryptic instruction or a TODO message?

AlexanderPrendota commented 3 years ago

Hey, thanks for the issue. Did u try the SAM to deploy the lambda?

Could you please provide cloudwatch logs?

AlexanderPrendota commented 3 years ago

For Kotless, yes, I need to write an instruction. Right now the project works with kotless. You just need to remove aws-serverless-container and rewrite the string boot main class. Should be like:

import io.kotless.dsl.spring.Kotless
import org.springframework.boot.autoconfigure.SpringBootApplication
import org.springframework.boot.runApplication
import kotlin.reflect.KClass

@SpringBootApplication
class CompilerApplication : Kotless() {
  override val bootKlass: KClass<*> = this::class
}

fun main(args: Array<String>) {
  runApplication<CompilerApplication>(*args)
}

chriscoomber commented 3 years ago

I don't think I used SAM. I just did:

gradlew buildLambda
Upload to S3 the zip file in build/distributions/ via https://s3.console.aws.amazon.com/s3/
Create function from https://eu-west-2.console.aws.amazon.com/lambda/home?region=eu-west-2#/functions
- Author from scratch
- Runtime: Java8
- security group - allow 8080 and 80 in
- create
- Click on the layer I created then
- Function Code -> actions -> Upload a file from Amazon S3 - choose the zip I uploaded earlier
- Runtime settings: set the handler to com.compiler.server.lambdas.StreamLambdaHandler::handleRequest
- Environment variables: add ACCESS_CONTROL_ALLOW_ORIGIN_VALUE = and ACCESS_CONTROL_ALLOW_HEADER_VALUE =
- Add a trigger
- For testing I just added an API gateway (HTTP type, not REST) and made sure to tick the CORS checkbox.

That API gateway had an invoke URL of https://ah41tetfai.execute-api.eu-west-2.amazonaws.com, so I tried calling https://ah41tetfai.execute-api.eu-west-2.amazonaws.com/versions and that's when I see the 500 error.

{"message":"Internal Server Error"}

I don't see any logs at all in Cloudwatch when I call that URL, so it's possible I'm not even invoking the function? When I run a test event in I see some logs:

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|                                                                                                                        message                                                                                                                        |
|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 14:55:10.653 [main] INFO com.amazonaws.serverless.proxy.internal.LambdaContainerHandler - Starting Lambda Container Handler                                                                                                                           |
| .   ____          _            __ _ _                                                                                                                                                                                                                 |
| /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \                                                                                                                                                                                                               |
| ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \                                                                                                                                                                                                             |
| \\/  ___)| |_)| | | | | || (_| |  ) ) ) )                                                                                                                                                                                                             |
| '  |____| .__|_| |_|_| |_\__, | / / / /                                                                                                                                                                                                               |
| =========|_|==============|___/=/_/_/_/                                                                                                                                                                                                               |
| :: Spring Boot ::                (v2.4.3)                                                                                                                                                                                                             |
| 2021-03-01 14:55:12.131  INFO 1 --- [           main] lambdainternal.LambdaRTEntry             : Starting LambdaRTEntry using Java 1.8.0_201 on 169.254.148.117 with PID 1 (/var/runtime/lib/LambdaJavaRTEntry-1.0.jar started by sbx_user1051 in /)  |
| 2021-03-01 14:55:12.133  INFO 1 --- [           main] lambdainternal.LambdaRTEntry             : No active profile set, falling back to default profiles: default                                                                                     |
| 2021-03-01 14:55:14.762  INFO 1 --- [           main] c.a.s.p.i.servlet.AwsServletContext      : Initializing Spring embedded WebApplicationContext                                                                                                   |
| 2021-03-01 14:55:14.763  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2408 ms                                                                                      |
| START RequestId: 3e67403e-c98d-44f2-ab81-dc201fafbed2 Version: $LATEST                                                                                                                                                                                |
| 14:55:22.551 [main] INFO com.amazonaws.serverless.proxy.internal.LambdaContainerHandler - Starting Lambda Container Handler                                                                                                                           |
| .   ____          _            __ _ _                                                                                                                                                                                                                 |
| /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \                                                                                                                                                                                                               |
| ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \                                                                                                                                                                                                             |
| \\/  ___)| |_)| | | | | || (_| |  ) ) ) )                                                                                                                                                                                                             |
| '  |____| .__|_| |_|_| |_\__, | / / / /                                                                                                                                                                                                               |
| =========|_|==============|___/=/_/_/_/                                                                                                                                                                                                               |
| :: Spring Boot ::                (v2.4.3)                                                                                                                                                                                                             |
| 2021-03-01 14:55:28.109  INFO 1 --- [           main] lambdainternal.LambdaRTEntry             : Starting LambdaRTEntry using Java 1.8.0_201 on 169.254.148.117 with PID 1 (/var/runtime/lib/LambdaJavaRTEntry-1.0.jar started by sbx_user1051 in /)  |
| 2021-03-01 14:55:28.148  INFO 1 --- [           main] lambdainternal.LambdaRTEntry             : No active profile set, falling back to default profiles: default                                                                                     |
| END RequestId: 3e67403e-c98d-44f2-ab81-dc201fafbed2                                                                                                                                                                                                   |
| REPORT RequestId: 3e67403e-c98d-44f2-ab81-dc201fafbed2 Duration: 15007.24 ms Billed Duration: 15000 ms Memory Size: 512 MB Max Memory Used: 121 MB                                                                                                    |
| 2021-03-01T14:55:35.259Z 3e67403e-c98d-44f2-ab81-dc201fafbed2 Task timed out after 15.01 seconds                                                                                                                                                      |
| 14:55:36.988 [main] INFO com.amazonaws.serverless.proxy.internal.LambdaContainerHandler - Starting Lambda Container Handler                                                                                                                           |
| .   ____          _            __ _ _                                                                                                                                                                                                                 |
| /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \                                                                                                                                                                                                               |
| ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \                                                                                                                                                                                                             |
| \\/  ___)| |_)| | | | | || (_| |  ) ) ) )                                                                                                                                                                                                             |
| '  |____| .__|_| |_|_| |_\__, | / / / /                                                                                                                                                                                                               |
| =========|_|==============|___/=/_/_/_/                                                                                                                                                                                                               |
| :: Spring Boot ::                (v2.4.3)                                                                                                                                                                                                             |
| 2021-03-01 14:55:38.489  INFO 1 --- [           main] lambdainternal.LambdaRTEntry             : Starting LambdaRTEntry using Java 1.8.0_201 on 169.254.148.117 with PID 1 (/var/runtime/lib/LambdaJavaRTEntry-1.0.jar started by sbx_user1051 in /)  |
| 2021-03-01 14:55:38.492  INFO 1 --- [           main] lambdainternal.LambdaRTEntry             : No active profile set, falling back to default profiles: default                                                                                     |
| 2021-03-01 14:55:40.962  INFO 1 --- [           main] c.a.s.p.i.servlet.AwsServletContext      : Initializing Spring embedded WebApplicationContext                                                                                                   |
| 2021-03-01 14:55:40.962  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2271 ms                                                                                      |
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

I'm happy to use SAM, but I haven't ever used that before so I don't know how to do it.

chriscoomber commented 3 years ago

OK so I tried using an Elastic Load Balancer instead of an API gateway as the trigger, and it started working. (I used the load balancer I created before to be a HTTPS reverse proxy in #257).

Edit: Not it's not working. I just confused myself. Oh well I'll leave what I wrote before realizing this after the fold.

I guess you can close this issue, however I propose expanding the lambda instructions to include a "quick-setup" guide for people who aren't experienced with this. Or maybe you could argue that's out of scope of this repository. I'm not sure. Here's what I would write (I need to test this and edit it):

Quick start

Prerequisites

You have an AWS account

Build the lambda function

./gradlew buildLambda (for windows, replace / with \) The .zip file is create in build/distributions

Upload the function code to s3

Go to https://s3.console.aws.amazon.com/s3.

Create a new bucket
Upload the .zip file into this bucket.
Make a note of the s3 URI (e.g. s3://myBucket/kotlin-compiler-server-1.4.30-SNAPSHOT.zip)

Create the lambda

Go to https://console.aws.amazon.com/lambda/ and create function. Use the following settings:

Author from scratch
Function name: your choice
Runtime: Java 8
Leave everything else default

Install the function code

You will now see this:

Click on your new lambda function.

Under Function Code -> Actions, click Upload a file from Amazon S3 and us the S3 URL from earlier
Under Runtime Settings -> Edit, change the Handler to com.compiler.server.lambdas.StreamLambdaHandler::handleRequest
Under Environment variables -> Edit add ACCESS_CONTROL_ALLOW_ORIGIN_VALUE = * and ACCESS_CONTROL_ALLOW_HEADER_VALUE = *

Add a trigger

Click + Add Trigger, and choose Load Balancer. Where it prompts you to choose a load balancer, create a new one:

name: up to you
scheme: internet-facing
IP address type: ipv4 is fine
Listeners: http/port 80
Availability Zones: add all subnets

Click next, skip warning about security (we will worry about SSL later)

Step 3: Choose default security group

Step 4: Create a new target group, Target type = Lambda function. Leave health checks unchecked for now.

Step 5: Register targets: Select your lambda function.

Now find the load balancer you just created and find its DNS name, and call http:///version and you should get a response from the lambda function!

Edit: as mentioned above, this doesn't actually work. I can't get it working.

AlexanderPrendota commented 3 years ago

Hey, I will try to help you. Did u deploy the last master? First of all, try SAM. Here is an example with spring boot + sam. It is really easy to deploy.

Did u try to use Kotless? It is a nice tool. Try to use one. You can find docs here. Also, I have just tested the last master with Kotless and it works on my side.

Also, I'm going to deploy the lambda like you this week.

chriscoomber commented 3 years ago

OK thanks for the help. I've managed to get it working with Docker on aws ECS instead of aws Lambda, so this isn't high priority for me (though Lambda feels more efficient somehow). I feel a bit embarrassed that maybe this isn't a real issue. It's not really in scope of kotlin-compiler-server to explain how to set up aws Lambda. So maybe just close this issue.

On the other hand I think it is nice for repositories like this to have a quick-start guide for noobs. But I can see how it might be more efficient to just link a general aws Lambda tutorial.

FWIW I think my problem is with wiring - i.e. making sure that I could get the request from my computer, to the EC2 Load Balancer, to the Lambda function. Something in the middle there was getting lost.

chriscoomber commented 3 years ago

By the way thanks for the help, and for creating this server. I now have some nice documentation going on in github pages. (Still a WIP: https://chriscoomber.github.io/manydice/)

JetBrains / kotlin-compiler-server