Deleting static resources in DEV mode causes cache consistency bugs

[FroMage]

Describe the bug

Static resources are served with StaticHandler from Vert.x, which sometimes need to be loaded from the QuarkusClassLoader or from the classpath. This is accomplished via FileResolverImpl which uses a FileCache to extract things from zips/jars, but also copy directories and their contents to a cache folder under /tmp/vertx-cache.

Under some circumstances, it is possible to trigger caching a folder such as META-INF/resources containing a.txt and b.txt, then using QuarkusDevModeTest.deleteResourceFile to delete a.txt and the cache doesn't see it, so it will keep serving it.

Expected behavior

No response

Actual behavior

No response

How to Reproduce?

package io.quarkiverse.web.assets.sass.test;

import java.net.URL;

import org.hamcrest.Matchers;
import org.jboss.shrinkwrap.api.ShrinkWrap;
import org.jboss.shrinkwrap.api.asset.StringAsset;
import org.jboss.shrinkwrap.api.spec.JavaArchive;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;

import io.quarkus.test.QuarkusDevModeTest;
import io.quarkus.test.common.http.TestHTTPResource;
import io.restassured.RestAssured;

public class SassDevModeTest {

    @RegisterExtension
    static final QuarkusDevModeTest config = new QuarkusDevModeTest()
            .setArchiveProducer(() -> ShrinkWrap.create(JavaArchive.class)
                    .addAsManifestResource(new StringAsset("foo"),
                            "resources/a.txt"));

    @TestHTTPResource
    URL url;

    @Test
    public void testStaticDeletion() {
        config.addResourceFile("META-INF/resources/b.txt", "foo");
        RestAssured
        .when()
        .get("/b.txt").then()
        .statusCode(200);
        config.deleteResourceFile("META-INF/resources/a.txt");
        RestAssured
                .when()
                .get("/a.txt").then()
                .statusCode(404);
    }
}

This causes:

2023-02-06 17:04:05,431 ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (vert.x-eventloop-thread-1) HTTP Request to /a.txt failed, error id: 9afbaf14-7249-40ea-a8f2-f3c8de90b141-1: io.vertx.core.VertxException: Invalid resource: META-INF/resources/b.txt
    at io.vertx.core.file.impl.FileResolverImpl.unpackFromFileURL(FileResolverImpl.java:269)
    at io.vertx.core.file.impl.FileResolverImpl.unpackUrlResource(FileResolverImpl.java:237)
    at io.vertx.core.file.impl.FileResolverImpl.resolveFile(FileResolverImpl.java:162)
    at io.vertx.core.impl.VertxImpl.resolveFile(VertxImpl.java:803)
    at io.vertx.core.file.impl.FileSystemImpl$20.perform(FileSystemImpl.java:1135)
    at io.vertx.core.file.impl.FileSystemImpl$20.perform(FileSystemImpl.java:1133)
    at io.vertx.core.file.impl.FileSystemImpl$BlockingAction.handle(FileSystemImpl.java:1174)
    at io.vertx.core.file.impl.FileSystemImpl$BlockingAction.handle(FileSystemImpl.java:1156)
    at io.vertx.core.impl.ContextBase.lambda$null$0(ContextBase.java:137)
    at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264)
    at io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:135)
    at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
    at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
    at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
    at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
    at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:833)

Which is a bit weird, but caused by adding b.txt and deleting a.txt. I've seen other symptoms of this bug, this is not the most common. The most common is keeping on serving a file after we've deleted it. But this is all due to the FileCache not being reset in DEV/TEST modes after the FS has changed, when no full reload has happened.

Output of uname -a or ver

No response

Output of java -version

No response

GraalVM version (if different from Java)

No response

Quarkus version or git rev

No response

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

[quarkus-bot[bot]]

/cc @gwenneg (cache)

[FroMage]

@cescoffier @tsegismont any idea how to assign this to?

[pmlopes]

@FroMage does quarkus dev mode, also activate vertx web dev mode?

[gsmet]

@pmlopes a pointer on how it's supposed to be done? It will be easier to check. Thanks!

[tsegismont]

@gsmet https://vertx.io/docs/vertx-web/java/#_development_mode

[tsegismont]

@gsmet @FroMage You might also be interested in https://vertx.io/docs/vertx-web/java/#_disabling_file_caching_on_disk

[gsmet]

@FroMage could you check if setting quarkus.vertx.caching=false solves your issue?

If so we need to decide if we want to make it the default in dev mode. From what I can see, it defaults to true and we don't force it to false in dev mode.

/cc @cescoffier

[FroMage]

So, I confirm that quarkus.vertx.caching=false solves my original issue, which I haven't been able to properly write a test for. But it doesn't solve the issue I described here with the reproducer.