vertx caching folder cannot be created with Amazon Lambda extension

[oztimpower]

Amazon Lambda extension requires an immutable file system.

Upon execution on AWS, vertx attempts to create the vertx cache folder. This causes the AWS Lambda execution to fail with the stack trace below. When run locally using sam-local (docker) it works ok.

The vertx configuration in io.quarkus.vertx.core.runtime.config.VertxConfiguration.caching is set to true by default. For the Amazon Lambda this should be set to false.

Execution on AWS fails:

java.lang.IllegalStateException: Failed to create cache dir
 at io.vertx.core.file.impl.FileResolver.setupCacheDir(FileResolver.java:332)
 at io.vertx.core.file.impl.FileResolver.<init>(FileResolver.java:87)
 at io.vertx.core.impl.VertxImpl.<init>(VertxImpl.java:165)
 at io.vertx.core.impl.VertxImpl.vertx(VertxImpl.java:92)
 at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:40)
 at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:32)
 at io.vertx.core.Vertx.vertx(Vertx.java:85)
 at io.quarkus.vertx.core.runtime.VertxCoreRecorder.initializeWeb(VertxCoreRecorder.java:100)
 at io.quarkus.vertx.core.runtime.VertxCoreRecorder.initializeWeb(VertxCoreRecorder.java:82)
 at io.quarkus.deployment.steps.VertxCoreProcessor$buildWeb104.deploy_0(VertxCoreProcessor$buildWeb104.zig:71)
 at io.quarkus.deployment.steps.VertxCoreProcessor$buildWeb104.deploy(VertxCoreProcessor$buildWeb104.zig:96)
 at io.quarkus.runner.ApplicationImpl5.doStart(ApplicationImpl5.zig:226)
 at io.quarkus.runtime.Application.start(Application.java:93)
 at io.quarkus.runtime.Application.run(Application.java:213)
 at io.quarkus.runner.GeneratedMain.main(GeneratedMain.zig:34)

Code (Kotlin):

package org.tjpower

import com.amazonaws.services.lambda.runtime.Context
import com.amazonaws.services.lambda.runtime.RequestHandler
import javax.enterprise.context.ApplicationScoped
import javax.inject.Inject

class HelloRequest() {
    var firstName: String? = null
    var lastName: String? = null
}

@ApplicationScoped
open class HelloGreeter() {
    open fun greet(firstName: String?, lastName: String?): String {
        return "$firstName, $lastName"
    }
}

class HelloLambda : RequestHandler<HelloRequest, String> {

//    @Inject
//    lateinit var greeter: HelloGreeter

    val greeter = HelloGreeter()

    override fun handleRequest(request: HelloRequest, context: Context): String {
        return greeter.greet(request.firstName, request.lastName)
    }
}

[geoand]

@cescoffier what are the implications of disabling caching?

[cescoffier]

@geoand the main impact would be when serving static files, as they are unpacked into the cache dir and read from there. In a lambda environment, serving static file is probably not used (@patriot1burke can you confirm?), so we could disable that.

Vert.x use the system tmp directory to create the cache directory, do it means that on AWS there is no tmp?

[oztimpower]

I did lots of digging today to drill into the problem. With a pure java implementation, it works. However with the native-image version it doesn't. The difference is that the native image uses "/var/tmp" as the temp folder.

I tested it manually, modifying the bootstrap shell script, creating the directories in both /tmp and /var/tmp. I can create /tmp/vertx-cache, but I cannot create /var/tmp/vertx-cache, which is what native-image sees.

mkdir: cannot create directory ‘/var/tmp/vertx-cache’: Read-only file system

package io.quarkus.vertx.core.runtime;
@Recorder
public class VertxCoreRecorder {

...
136:

        String fileCacheDir = System.getProperty(CACHE_DIR_BASE_PROP_NAME,
                System.getProperty("java.io.tmpdir", ".") + File.separator + "vertx-cache");

[oztimpower]

I have traced the issue. Yes /tmp is writeable for AWS Lambda, for both files and directories.

However native-image sees the java.io.tmpdir as “/var/tmp”, which is read-only. Is this determined at compile time in native-image by the docker linux container used?

The pure-java version works fine (which means it uses /tmp for it’s java.io http://java.io/.tmpdir on AWS lambda).

java.io http://java.io/.tmpdir appear to be different for native-image vs JVM.

I’ve updated the issue accordingly.

On 30 Sep 2019, at 1:52 PM, Clement Escoffier notifications@github.com wrote:

@geoand https://github.com/geoand the main impact would be when serving static files, as they are unpacked into the cache dir and read from there. In a lambda environment, serving static file is probably not used (@patriot1burke https://github.com/patriot1burke can you confirm?), so we could disable that.

Vert.x use the system tmp directory to create the cache directory, do it means that on AWS there is no tmp?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/quarkusio/quarkus/issues/4254?email_source=notifications&email_token=AIPGZO5XA6W2G6NS6RNNYNDQMGHZHA5CNFSM4I3N4VTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD74P4MQ#issuecomment-536411698, or mute the thread https://github.com/notifications/unsubscribe-auth/AIPGZOYG4PX6ZBFG6INR45TQMGHZHANCNFSM4I3N4VTA.

[cescoffier]

Can you try to pass -Djava.io.tmpdir=/tmp to the native executable?

On 30 Sep 2019, at 16:51, oztimpower notifications@github.com wrote:

I have traced the issue. Yes /tmp is writeable for AWS Lambda, for both files and directories.

However native-image sees the java.io.tmpdir as “/var/tmp”, which is read-only. Is this determined at compile time in native-image by the docker linux container used?

The pure-java version works fine (which means it uses /tmp for it’s java.io http://java.io/.tmpdir on AWS lambda).

java.io http://java.io/.tmpdir appear to be different for native-image vs JVM.

I’ve updated the issue accordingly.

On 30 Sep 2019, at 1:52 PM, Clement Escoffier notifications@github.com wrote:

@geoand https://github.com/geoand the main impact would be when serving static files, as they are unpacked into the cache dir and read from there. In a lambda environment, serving static file is probably not used (@patriot1burke https://github.com/patriot1burke can you confirm?), so we could disable that.

Vert.x use the system tmp directory to create the cache directory, do it means that on AWS there is no tmp?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/quarkusio/quarkus/issues/4254?email_source=notifications&email_token=AIPGZO5XA6W2G6NS6RNNYNDQMGHZHA5CNFSM4I3N4VTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD74P4MQ#issuecomment-536411698, or mute the thread https://github.com/notifications/unsubscribe-auth/AIPGZOYG4PX6ZBFG6INR45TQMGHZHANCNFSM4I3N4VTA.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/quarkusio/quarkus/issues/4254?email_source=notifications&email_token=AADCG7JXD7ZDRW33NYL6LLLQMIHATA5CNFSM4I3N4VTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD755P6A#issuecomment-536598520, or mute the thread https://github.com/notifications/unsubscribe-auth/AADCG7MHFUI5OJRZGTCI7MDQMIHATANCNFSM4I3N4VTA.

[oztimpower]

That worked: *-runner -Djava.io.tmpdir=/tmp

However next problem, -H:EnableURLProtocols=http needs to be passed to the native-image command. For some reason that’s not appearing in the native-image command, captured below. When I manually added the http protocol to the native-image compile command, along with the above modification to java.io.tmpdir, everything worked ok, all fixed.

START RequestId: 211f3508-4af3-46ef-ab8a-73ce01946479 Version: $LATEST Exception in thread "Lambda Thread" 16:24:21 INFO [io.quarkus]] (main) Quarkus 999-SNAPSHOT started in 0.219s. Listening on: http://0.0.0.0:8080 com.oracle.svm.core.jdk.UnsupportedFeatureError: Accessing an URL protocol that was not enabled. The URL protocol http is supported but not enabled by default. It must be enabled by adding the -H:EnableURLProtocols=http option to the native-image command. 16:24:21 INFO [io.quarkus]] (main) Profile prod activated. at com.oracle.svm.core.util.VMError.unsupportedFeature(VMError.java:102) at com.oracle.svm.core.jdk.JavaNetSubstitutions.unsupported(JavaNetSubstitutions.java:196) 16:24:21 INFO [io.quarkus]] (main) Installed features: [cdi, kotlin, resteasy, resteasy-jsonb] at com.oracle.svm.core.jdk.JavaNetSubstitutions.getURLStreamHandler(JavaNetSubstitutions.java:157) at java.net.URL.getURLStreamHandler(URL.java:62) at java.net.URL.(URL.java:599) at java.net.URL.(URL.java:490) at java.net.URL.(URL.java:439) at io.quarkus.amazon.lambda.runtime.AmazonLambdaApi.invocationNext(AmazonLambdaApi.java:31) at io.quarkus.amazon.lambda.runtime.AmazonLambdaRecorder$2.run(AmazonLambdaRecorder.java:59) at java.lang.Thread.run(Thread.java:748) at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:460) at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193) END RequestId: 211f3508-4af3-46ef-ab8a-73ce01946479 REPORT RequestId: 211f3508-4af3-46ef-ab8a-73ce01946479 Duration: 10010.17 ms Billed Duration: 10000 ms Memory Size: 128 MB Max Memory Used: 15 MB
2019-09-30T16:24:31.513Z 211f3508-4af3-46ef-ab8a-73ce01946479 Task timed out after 10.01 seconds

docker run -v /Users/timothypower/workspace/IdeaProjects/aws-samples/services/rest-kotlin/build:/project:z --rm quay.io/quarkus/ubi-quarkus-native-image:19.2.0.1 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dio.netty.noUnsafe=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dvertx.disableDnsResolver=true --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -jar rest-kotlin-0.0.1-SNAPSHOT-runner.jar -J-Djava.util.concurrent.ForkJoinPool.common.parallelism=1 -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:+PrintAnalysisCallTree -H:-AddAllCharsets -H:-SpawnIsolates -H:-JNI --no-server -H:-UseServiceLoaderFeature -H:+StackTrace

On 30 Sep 2019, at 11:33 PM, Clement Escoffier notifications@github.com wrote:

Can you try to pass -Djava.io.tmpdir=/tmp to the native executable?

On 30 Sep 2019, at 16:51, oztimpower notifications@github.com wrote:

I have traced the issue. Yes /tmp is writeable for AWS Lambda, for both files and directories.

However native-image sees the java.io.tmpdir as “/var/tmp”, which is read-only. Is this determined at compile time in native-image by the docker linux container used?

The pure-java version works fine (which means it uses /tmp for it’s java.io http://java.io/.tmpdir on AWS lambda).

java.io http://java.io/.tmpdir appear to be different for native-image vs JVM.

I’ve updated the issue accordingly.

On 30 Sep 2019, at 1:52 PM, Clement Escoffier notifications@github.com wrote:

@geoand https://github.com/geoand the main impact would be when serving static files, as they are unpacked into the cache dir and read from there. In a lambda environment, serving static file is probably not used (@patriot1burke https://github.com/patriot1burke can you confirm?), so we could disable that.

Vert.x use the system tmp directory to create the cache directory, do it means that on AWS there is no tmp?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/quarkusio/quarkus/issues/4254?email_source=notifications&email_token=AIPGZO5XA6W2G6NS6RNNYNDQMGHZHA5CNFSM4I3N4VTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD74P4MQ#issuecomment-536411698, or mute the thread https://github.com/notifications/unsubscribe-auth/AIPGZOYG4PX6ZBFG6INR45TQMGHZHANCNFSM4I3N4VTA.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/quarkusio/quarkus/issues/4254?email_source=notifications&email_token=AADCG7JXD7ZDRW33NYL6LLLQMIHATA5CNFSM4I3N4VTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD755P6A#issuecomment-536598520, or mute the thread https://github.com/notifications/unsubscribe-auth/AADCG7MHFUI5OJRZGTCI7MDQMIHATANCNFSM4I3N4VTA.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/quarkusio/quarkus/issues/4254?email_source=notifications&email_token=AIPGZO342Y25CCV533WP233QMIL6HA5CNFSM4I3N4VTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD76CJRQ#issuecomment-536618182, or mute the thread https://github.com/notifications/unsubscribe-auth/AIPGZO3VG2MTOEC2Z64ZFGTQMIL6HANCNFSM4I3N4VTA.

[gsmet]

However next problem, -H:EnableURLProtocols=http needs to be passed to the native-image command.

@oztimpower , I'm curious about this part: how did you create your project? IIRC we enable the handler when in the Maven project template.

[oztimpower]

I used Gradle, hand crafted from the documentation.

Is there a way to force this in the code? The Amazon lambda extension has a lot of http calls in the Quarkus wrapper runtime code. The bootstrap of native-image does not appear to module aware.

On 12 Oct 2019, at 9:38 PM, Guillaume Smet notifications@github.com wrote:

 However next problem, -H:EnableURLProtocols=http needs to be passed to the native-image command.

@oztimpower , I'm curious about this part: how did you create your project? IIRC we enable the handler when in the Maven project template.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[gsmet]

Yeah, it's probably something we should push via the extension mechanism.

Could you create a specific issue for that?

I'm leaning towards closing this one as you fixed the initial issue. It will be clearer if we have a specific one.

Thanks!

[johnoliver]

for future reference for anyone who may stumble across this issue when deploying to openshift, I had to set -Dvertx.cacheDirBase=/tmp/vertx by creating an environment variable called JAVA_OPTS set to that value. Setting -Djava.io.tmpdir=/tmp alone did not seem to be sufficient.

[machi1990]

for future reference for anyone who may stumble across this issue when deploying to openshift, I had to set -Dvertx.cacheDirBase=/tmp/vertx by creating an environment variable called JAVA_OPTS set to that value. Setting -Djava.io.tmpdir=/tmp alone did not seem to be sufficient.

Thanks for the tip. Maybe we can add it in the doc, WDYT? @gsmet @cescoffier

[geoand]

That would seem like a valuable addition to the vertx guide

[machi1990]

@johnoliver would you like to contribute the tip here https://github.com/quarkusio/quarkus/blob/9eb5164f24a2c3bc6b4015bc4e399a7ef955704c/docs/src/main/asciidoc/vertx.adoc ?

[johnoliver]

@machi1990 https://github.com/quarkusio/quarkus/pull/5287

[sherl0cks]

@oztimpower was there a resolution to this on lambda? The suggestions for OpenShift do not work on lambda for me.

[oztimpower]

Hi @sherl0cks yes the issue was resolved for Amazon Lambda. I recall seeing a similar question about OpenShift that forcing the /tmp folder did not work. Recommend you search the issues or create a new issue to track it.

[sherl0cks]

@oztimpower @cescoffier @gsmet can you share the resolution for lambda? I see #4327, but that file is no longer present in master (moved in 0.27) and it is unclear to me how I can set this command line switch in the lambda custom runtime. I've tried all sorts of ways to disable vertx caching to no effect.

To be clear, I'm not interest in OpenShift, only aws lambda native. I'm using 1.0.1.final of quarkus.

[cescoffier]

@sherl0cks setting -Dvertx.cacheDirBase=/somewher/in/the/cloud/that/is/writable should still work.

[oztimpower]

@sherl0cks have you tried using the SAM CLI as per the latest Quarkus documentation?

Amazon lambda works for me, the Maven deployment automated the bundling and the /tmp folder issue was migrated to code, and thus dropped from the script.

Possible to post your error message, particularly with SAM local?

I use Quarkus with native lambda in Prod, works well.

[sherl0cks]

Just got this working. Below are the steps I took. @cescoffier would you be interested in a PR to the maven archetype for step 7 and info in the docs to explain how this works for future users?

1) unzip function.zip 2) rename bootstrap to something like runner 3) create a new shell script called bootstrap with something like ./runner -Djava.io.tmpdir=/tmp. @oztimpower I was unable to achieve the same effect via setting a system property in code. Do you have example of what you did? 4) make sure bootstrap is executable 5) zip bootstrap & runner into a new function.zip 6) run update-native.sh (assuming you came from quarkus-aws-hello-world example) and then test with invoke-native.sh 7) automate steps 1-6 by adding the files to src and updating the zip.xml config for the maven assembly plugin that creates the .zip file

I'm not using SAM cli as I generally find it to be more trouble than its worth. Reading the maven archetype and the docs, I do not see anything in that guide that would address this concern. If you can link to something specific that I missed, please do.

For any future readers, my code calls an https endpoint and thus I need to update the zip.xml to package in libsunec.so and cacerts per the native ssl guide.

[patriot1burke]

Did you increase your cold start latency?

[sherl0cks]

I can’t say until I find a way to change vertx temp without this extra script. That said, I’m getting ~250ms init on a cold start right now, which is basically flat from 128mb to to max 3008mb. Duration starts at about ~250ms and then levels out at ~25ms at 512mb.

[ctron-te]

Strangely when I try the sample project with Java everything works fine. Then I converted it to Kotlin, which works fine with java runtime but when I move to provided runtime I get java.lang.IllegalStateException: Failed to create cache dir.

This is using Quarkus 1.2.0.Final

[patriot1burke]

Any reason to have quarkus set the java.io.tmpdir and vertx cache dir automatically? Pretty sure /tmp is guaranteed to be available for lambda.

[patriot1burke]

My issue is I can't reproduce the problem.

[patriot1burke]

BTW all, we want to avoid requiring a bootstrap shell script around the runner so as to reduce cold start latency even more.

[oztimpower]

I'm experiencing the same problem as @ctron-te with head (999-SNAPSHOT).

Kotlin AWS Lambda project that can reproduce it. Project is a Kotlin version of the Java AWS Lambda template, i.e. InputObject, OutputObject, ProcessingService.

1. git clone https://github.com/oztimpower/kotlin-lambda.git
2. mvn package -Dnative -Dquarkus.native.container-build=true
3. ./manage.sh native create
4. ./manage.sh native invoke

START RequestId: 555e0118-5147-4830-86ec-d73469fd046e Version: $LATEST 2020-03-04 14:09:01,667 ERROR [io.qua.application] (main) Failed to start application: java.lang.IllegalStateException: Failed to create cache dir: /var/tmp//vertx-cache/file-cache-14c1dacd-5b2b-4f42-bd36-1e706fa769b2 at io.vertx.core.file.impl.FileResolver.setupCacheDir(FileResolver.java:332) at io.vertx.core.file.impl.FileResolver.(FileResolver.java:87) at io.vertx.core.impl.VertxImpl.(VertxImpl.java:167) at io.vertx.core.impl.VertxImpl.vertx(VertxImpl.java:92) at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:40) at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:32) at io.vertx.core.Vertx.vertx(Vertx.java:85)

[oztimpower]

I should add it works fine with Java source Native executable. Issue is with Kotlin.

[patriot1burke]

This PR might help. https://github.com/quarkusio/quarkus/pull/7619

[oztimpower]

I confirm that #7619, resolves the issue with my Kotlin Amazon Lambda project. i.e., forcing the tmp dirs in AmazonLambdaProcessor.java works.