search

micronaut-projects / micronaut-aws

Projects specific to integrating Micronaut and Amazon Web Services (AWS)
Apache License 2.0
85 stars 79 forks source link

GraalVM native generated app is not working in AWS #6

Closed huksley closed 5 years ago

huksley commented 5 years ago

I am using mn version 1.1.0.M1 Created app using mn create-app my-app --features aws-api-gateway-graal

Trying to deploy it and it not working, gives error in CloudWatch:

START RequestId: 6ed53036-dc55-4c60-ae7a-ba91ad8f988c Version: $LATEST
io.micronaut.http.client.exceptions.ReadTimeoutException: Read Timeout
Request loop failed with: Read Timeout
END RequestId: 6ed53036-dc55-4c60-ae7a-ba91ad8f988c
REPORT RequestId: 6ed53036-dc55-4c60-ae7a-ba91ad8f988c  Duration: 15015.13 ms   Billed Duration: 15000 ms Memory Size: 128 MB   Max Memory Used: 95 MB  
2019-02-13T09:29:19.172Z 6ed53036-dc55-4c60-ae7a-ba91ad8f988c Task timed out after 15.02 seconds

Example project here: https://github.com/huksley/micronaut-aws-api-gateway-graal

graemerocher commented 5 years ago

Hmm.. I see that occasionally as well ... not for every request. Seems intermittent. Think we need to remove the read timeout handling from the client. Thanks for the feedback

huksley commented 5 years ago

@graemerocher thanks! btw I have a slightly more complicated project (derived from Micronaut Guide one) deploying to AWS and calling API results in Segfaults, is there any way to debug that? https://github.com/huksley/serverless-java-micronaut/issues/1

graemerocher commented 5 years ago

Best advice is to report the segfault to https://github.com/oracle/graal/issues

graemerocher commented 5 years ago

The read timeout issues is resolved by https://github.com/micronaut-projects/micronaut-aws/commit/ee5d792597951c2852d12935f606508b55d88a26

Any Graal issues will need to be resolved by reports to Graal. Thanks for the feedback!

huksley commented 5 years ago

That was fast! :bullettrain_side: Also fixed https://github.com/huksley/serverless-java-micronaut/issues/1 by adding some fixes mentioned in graal issues.

Btw, how I can try with master version of Micronaut, i.e. not 1.1.0-1M?

graemerocher commented 5 years ago

You can add the snapshot repo to build.gradle:

repositories {
    ...
    maven { url "https://oss.sonatype.org/content/repositories/snapshots/" }
}

And specifying the snapshot version as your BOM:

dependencyManagement {
    imports {
        mavenBom 'io.micronaut:micronaut-bom:1.1.0.BUILD-SNAPSHOT'
    }
}
huksley commented 5 years ago

@graemerocher Thanks, compiling https://github.com/huksley/micronaut-aws-api-gateway-graal with snapshot dependencies, the problem still the same (running in sam local-api):

2019-02-13 20:27:26 Found credentials in shared credentials file: ~/.aws/credentials
2019-02-13 20:27:26 Mounting MyServiceFunction at http://127.0.0.1:3000/{proxy+} [GET, DELETE, PUT, POST, HEAD, OPTIONS, PATCH]
2019-02-13 20:27:26 You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2019-02-13 20:27:26  * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)
2019-02-13 20:27:33 Invoking example.StreamLambdaHandler::handleRequest (provided)
2019-02-13 20:27:33 Decompressing /home/rgai/src/my-app/build/function.zip
2019-02-13 20:27:34 Requested to skip pulling images ...
2019-02-13 20:27:34 Mounting /tmp/tmpIc6iwR as /var/task:ro inside runtime container
18:27:35.933 [main] INFO  i.m.f.a.p.AbstractLambdaContainerHandler - Starting Lambda Container Handler
18:27:35.939 [main] INFO  i.m.context.env.DefaultEnvironment - Established active environments: [function]
START RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72 Version: $LATEST
2019-02-13 20:27:50 Function 'MyServiceFunction' timed out after 15 seconds
2019-02-13 20:27:52 Function returned an invalid response (must include one of: body, headers or statusCode in the response object). Response received: 
2019-02-13 20:27:52 127.0.0.1 - - [13/Feb/2019 20:27:52] "GET /ping HTTP/1.1" 502 -

I've checked my dependencies via ./gradlew dependencies

runtimeClasspath - Runtime classpath of source set 'main'.
+--- io.micronaut:micronaut-http-client -> 1.1.0.BUILD-SNAPSHOT
|    +--- org.slf4j:slf4j-api:1.7.25
|    +--- io.reactivex.rxjava2:rxjava:2.2.6
|    |    \--- org.reactivestreams:reactive-streams:1.0.2
|    +--- io.micronaut:micronaut-http-netty:1.1.0.BUILD-SNAPSHOT
|    |    +--- org.slf4j:slf4j-api:1.7.25
|    |    +--- io.micronaut:micronaut-buffer-netty:1.1.0.BUILD-SNAPSHOT
|    |    |    +--- org.slf4j:slf4j-api:1.7.25
|    |    |    +--- io.micronaut:micronaut-core:1.1.0.BUILD-SNAPSHOT
|    |    |    |    +--- org.slf4j:slf4j-api:1.7.25
|    |    |    |    +--- org.reactivestreams:reactive-streams:1.0.2
|    |    |    |    \--- com.google.code.findbugs:jsr305:3.0.2
|    |    |    \--- io.netty:netty-buffer:4.1.30.Final
|    |    |         \--- io.netty:netty-common:4.1.30.Final
|    |    +--- io.micronaut:micronaut-http:1.1.0.BUILD-SNAPSHOT
|    |    |    \--- io.micronaut:micronaut-inject:1.1.0.BUILD-SNAPSHOT
|    |    |         +--- javax.inject:javax.inject:1
|    |    |         +--- io.micronaut:micronaut-core:1.1.0.BUILD-SNAPSHOT (*)
|    |    |         +--- org.yaml:snakeyaml:1.23
|    |    |         \--- javax.annotation:javax.annotation-api:1.3.2
|    |    +--- io.micronaut:micronaut-websocket:1.1.0.BUILD-SNAPSHOT
|    |    |    +--- org.slf4j:slf4j-api:1.7.25
|    |    |    +--- io.micronaut:micronaut-http:1.1.0.BUILD-SNAPSHOT (*)
|    |    |    +--- io.micronaut:micronaut-inject:1.1.0.BUILD-SNAPSHOT (*)
|    |    |    +--- io.micronaut:micronaut-aop:1.1.0.BUILD-SNAPSHOT
|    |    |    |    \--- io.micronaut:micronaut-inject:1.1.0.BUILD-SNAPSHOT (*)
|    |    |    \--- io.reactivex.rxjava2:rxjava:2.2.6 (*)
|    |    +--- io.netty:netty-codec-http:4.1.30.Final
|    |    |    \--- io.netty:netty-codec:4.1.30.Final
|    |    |         \--- io.netty:netty-transport:4.1.30.Final
|    |    |              +--- io.netty:netty-buffer:4.1.30.Final (*)
|    |    |              \--- io.netty:netty-resolver:4.1.30.Final
|    |    |                   \--- io.netty:netty-common:4.1.30.Final
|    |    +--- io.reactivex.rxjava2:rxjava:2.2.6 (*)
|    |    \--- io.netty:netty-handler:4.1.30.Final
|    |         +--- io.netty:netty-buffer:4.1.30.Final (*)
|    |         +--- io.netty:netty-transport:4.1.30.Final (*)
|    |         \--- io.netty:netty-codec:4.1.30.Final (*)
|    +--- io.micronaut:micronaut-runtime:1.1.0.BUILD-SNAPSHOT
|    |    +--- io.micronaut:micronaut-inject:1.1.0.BUILD-SNAPSHOT (*)
|    |    +--- io.micronaut:micronaut-http:1.1.0.BUILD-SNAPSHOT (*)
|    |    +--- io.micronaut:micronaut-aop:1.1.0.BUILD-SNAPSHOT (*)
|    |    +--- javax.validation:validation-api:2.0.1.Final
|    |    +--- javax.cache:cache-api:1.1.0
|    |    +--- com.fasterxml.jackson.core:jackson-databind:2.9.8
|    |    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.9.0 -> 2.9.8
|    |    |    \--- com.fasterxml.jackson.core:jackson-core:2.9.8
|    |    +--- io.reactivex.rxjava2:rxjava:2.2.6 (*)
|    |    +--- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.9.8
|    |    |    +--- com.fasterxml.jackson.core:jackson-core:2.9.8
|    |    |    \--- com.fasterxml.jackson.core:jackson-databind:2.9.8 (*)
|    |    \--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.9.8
|    |         +--- com.fasterxml.jackson.core:jackson-annotations:2.9.0 -> 2.9.8
|    |         +--- com.fasterxml.jackson.core:jackson-core:2.9.8
|    |         \--- com.fasterxml.jackson.core:jackson-databind:2.9.8 (*)
|    +--- io.netty:netty-handler-proxy:4.1.30.Final
|    |    +--- io.netty:netty-transport:4.1.30.Final (*)
|    |    +--- io.netty:netty-codec-socks:4.1.30.Final
|    |    |    \--- io.netty:netty-codec:4.1.30.Final (*)
|    |    \--- io.netty:netty-codec-http:4.1.30.Final (*)
|    \--- io.micronaut:micronaut-websocket:1.1.0.BUILD-SNAPSHOT (*)
+--- io.micronaut.aws:micronaut-function-aws-custom-runtime -> 1.1.0.M2
|    +--- io.micronaut:micronaut-http-client:1.0.4 -> 1.1.0.BUILD-SNAPSHOT (*)
|    \--- io.micronaut.aws:micronaut-function-aws-api-proxy:1.1.0.M2
|         +--- io.micronaut:micronaut-http-netty:1.0.4 -> 1.1.0.BUILD-SNAPSHOT (*)
|         +--- io.micronaut:micronaut-http-server:1.0.4 -> 1.1.0.BUILD-SNAPSHOT
|         |    +--- io.micronaut:micronaut-runtime:1.1.0.BUILD-SNAPSHOT (*)
|         |    +--- io.micronaut:micronaut-router:1.1.0.BUILD-SNAPSHOT
|         |    |    +--- org.slf4j:slf4j-api:1.7.25
|         |    |    +--- io.micronaut:micronaut-inject:1.1.0.BUILD-SNAPSHOT (*)
|         |    |    \--- io.micronaut:micronaut-http:1.1.0.BUILD-SNAPSHOT (*)
|         |    \--- io.micronaut:micronaut-websocket:1.1.0.BUILD-SNAPSHOT (*)
|         +--- com.amazonaws.serverless:aws-serverless-java-container-core:1.3
|         |    +--- com.amazonaws:aws-lambda-java-core:1.2.0
|         |    +--- javax.ws.rs:javax.ws.rs-api:2.1
|         |    +--- com.fasterxml.jackson.core:jackson-databind:2.9.7 -> 2.9.8 (*)
|         |    +--- commons-fileupload:commons-fileupload:1.3.3
|         |    |    \--- commons-io:commons-io:2.2
|         |    +--- org.apache.httpcomponents:httpmime:4.5.6
|         |    |    \--- org.apache.httpcomponents:httpclient:4.5.6
|         |    |         +--- org.apache.httpcomponents:httpcore:4.4.10
|         |    |         +--- commons-logging:commons-logging:1.2
|         |    |         \--- commons-codec:commons-codec:1.10
|         |    +--- org.apache.httpcomponents:httpcore:4.4.10
|         |    \--- org.slf4j:slf4j-api:1.8.0-beta2 -> 1.7.25
|         \--- io.micronaut:micronaut-function-aws:1.1.0.M2
|              +--- io.micronaut:micronaut-function:1.0.4 -> 1.1.0.BUILD-SNAPSHOT
|              |    +--- org.slf4j:slf4j-api:1.7.25
|              |    +--- io.micronaut:micronaut-runtime:1.1.0.BUILD-SNAPSHOT (*)
|              |    \--- io.micronaut:micronaut-http:1.1.0.BUILD-SNAPSHOT (*)
|              \--- com.amazonaws:aws-lambda-java-core:1.2.0
+--- io.micronaut:micronaut-http-server-netty -> 1.1.0.BUILD-SNAPSHOT
|    +--- org.slf4j:slf4j-api:1.7.25
|    +--- io.micronaut:micronaut-http-server:1.1.0.BUILD-SNAPSHOT (*)
|    \--- io.micronaut:micronaut-http-netty:1.1.0.BUILD-SNAPSHOT (*)
+--- io.micronaut:micronaut-inject -> 1.1.0.BUILD-SNAPSHOT (*)
+--- io.micronaut:micronaut-validation -> 1.1.0.BUILD-SNAPSHOT
|    +--- org.slf4j:slf4j-api:1.7.25
|    +--- io.micronaut:micronaut-inject:1.1.0.BUILD-SNAPSHOT (*)
|    +--- io.micronaut:micronaut-http:1.1.0.BUILD-SNAPSHOT (*)
|    \--- javax.validation:validation-api:2.0.1.Final
+--- io.micronaut:micronaut-runtime -> 1.1.0.BUILD-SNAPSHOT (*)
+--- ch.qos.logback:logback-classic:1.2.3
|    +--- ch.qos.logback:logback-core:1.2.3
|    \--- org.slf4j:slf4j-api:1.7.25
\--- io.micronaut:micronaut-graal -> 1.1.0.BUILD-SNAPSHOT
     +--- org.slf4j:slf4j-api:1.7.25
     \--- io.micronaut:micronaut-runtime:1.1.0.BUILD-SNAPSHOT (*)
huksley commented 5 years ago

Hmm, enabling trace level on logback results in errors like this:

18:42:36.168 [main] TRACE io.netty.channel.nio.NioEventLoop - failed to instrument a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@7fa51ed5c890
java.lang.NoSuchFieldException: selectedKeys
    at java.lang.Class.getDeclaredField(DynamicHub.java:2070)

Full dump here: https://gist.github.com/huksley/c94a3f60b0fa8199ebebfbb32d91f588 I have substitutions in place for Netty, maybe something else needed?

Actually, NoSuchFieldException are warnings only: NioEventLoop

Might be relevant:

graemerocher commented 5 years ago

You need the snapshot of micronaut-function-aws-custom-runtime

huksley commented 5 years ago

Finally got mn create example working, here are a summary of results and findings

All differences below vanilla mn create-app my-app --features aws-api-gateway-graal and final project: https://github.com/huksley/micronaut-aws-api-gateway-graal/compare/f6ab1cc30a0935121d8dcb9f190b1a2f5efc3395...master

P.S. This is just so a /ping request starts working. My more sophisticated project works in SAM but intermittent read timeout problems at AWS Lambda.

UPDATE: After sticking to the same SNAPSHOT dependency my VAT validation project works ok, event quite fast to query SOAP European Union endpoints https://gist.github.com/huksley/5bf190143d3175c672ee97f41941ffbe

graemerocher commented 5 years ago

@huksley Thanks for the update and sticking with it. The project template for aws-api-gateway-graal can be found here https://github.com/micronaut-projects/micronaut-profiles/tree/master/base/features/aws-api-gateway-graal/skeleton

It would be great if you could send a PR with the changes to make all of this work better. Could you do that?

huksley commented 5 years ago

It would be great if you could send a PR with the changes to make all of this work better. Could you do that?

Yes, I will do it! Can you advise how I can try out my changes in this template before making a PR for it?

graemerocher commented 5 years ago

You will need to build master from source https://docs.micronaut.io/latest/guide/index.html#buildSource

Then checkout https://github.com/micronaut-projects/micronaut-profiles and to

./gradlew pTML on that repo too

Then you should be able to do mn create-app my-app --features aws-api-gateway-graal to create an app. Every time you make a change to micronaut-profiles you need to do ./gradlew clean pTML again

huksley commented 5 years ago

@graemerocher

It would be great if you could send a PR with the changes to make all of this work better. Could you do that?

https://github.com/micronaut-projects/micronaut-profiles/pull/107