FileAlreadyExists for .embedmongo/fileSets/hashes/xxxx

[K0nstantine]

We use embed.mongo.spring for tests and they fail randomly in Tekton-Tasks with the following Exception:

Caused by: java.nio.file.FileAlreadyExistsException: /root/.embedmongo/fileSets/hashes/ebb0bb5b11aeca90f1437d5ae0db2dc4508488c9c63ac5f11072ee82923f3162 at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:94) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:261) at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:482) at java.base/java.nio.file.Files.newOutputStream(Files.java:227) at java.base/java.nio.file.Files.write(Files.java:3492) at de.flapdoodle.embed.process.store.ContentHashExtractedFileSetStore.lambda$readOrCreateArchiveContentAndFileSetDescriptionHash$9(ContentHashExtractedFileSetStore.java:162) at de.flapdoodle.types.ThrowingRunnable.lambda$mapException$0(ThrowingRunnable.java:28) ... 134 more

We have 2 TestsClasses that are using embed.mongo. They look like this:

@SpringBootTest
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
@DirtiesContext
class EventsConsumptionTest extends AbstractKafkaTest {
    @Autowired
    private Repository repo;
    @Autowired
    private ObjectMapper objectMapper;

    @BeforeAll
    public void setUp() {
        setUpProducer();
    }

    @AfterEach
    public void reset() {
        repo.deleteAll();
    }

    @Test
    @SneakyThrows
    void testCreatedEvent() {
        Created event = this.objectMapper.readValue(new File(EVENTS_PATH + "/created.json"), Created.class);
        sendEvent(inboundTopic, event);
        Thread.sleep(2000);
        Optional<Entity> saved = repo.findById(event.getId());
        assertFalse(saved.isEmpty());
    }

We also have defined MongoConfiguration for the tests:

@Configuration
public class MongoTestConfig {

    @Bean
    MongodArguments mongodArguments() {
        return MongodArguments.builder()
            .replication(Storage.of("test", 10))
            .build();
    }
}

Tests work locally with no problems. In the Build-Pipeline fails nearly every 5th build. If you could help on this one, it would be great!

[michaelmosmann]

@K0nstantine .. hmm.. i have to look into this..

[K0nstantine]

Hier noch der Exception-Stack dazu: exception_stack.txt

[michaelmosmann]

@K0nstantine .. ah.. danke:) ich hab so eine Idee...

[michaelmosmann]

@K0nstantine .. habs gefunden. release folgt.

[michaelmosmann]

@K0nstantine .. release 4.12.3, sollte in den nächsten Stunden in maven central sein .. ich schließe das mal, einfach wieder aufmachen, wenn nötig.

[K0nstantine]

Ich wechsle mal auf Englisch, damit die anderen evtl. auch verstehen 😄 .

Thanks for the fast reply and update. Unfortunately the error still happens. It is however not that often now. Somehow it helped but not for 100%. With the 4.12.3 nearly every second build fails.

May be some more details: Tekton caches .m2 dir of the previous builds and reuses cache for the following builds. But is shouldn't have any influence onto this issue, should it?

P.S. I can't reopen the issue

[michaelmosmann]

@K0nstantine ... are there more than one jvm (ci build agent?) running on this machine at the same time?

[K0nstantine]

One Container is started per build. Which means it has to be only one JVM for this particular build.

[michaelmosmann]

@K0nstantine .. hmm. Do you have a recent stack trace for that?

[K0nstantine]

exception_stack.txt

One more update: error happens only with multiple test files, which load Spring Context (@SpringBootTest). With a single test - no error happens.

[michaelmosmann]

@K0nstantine did a new release (4.12.6)

[K0nstantine]

Works like a charm! Could you explain what the error was or did you just update some third-party dependencies? Vielen lieben Dank auf jedem Fall!

[michaelmosmann]

@K0nstantine to avoid unnecessary downloads and extractions i try to cache these artifacts.. to ensure that you get the right thing, i use an sha-hash of the archive, but to avoid to read the archive every time to create this hash, i cache this hash in a file where the filename is a sha-hash of the archive path, the fileset (exec + lib files) and lastModified of the archive file ..

So there is a file with hash(archivePath, fileset, lastModified) as filename and the content is hash(archive) and an directory with the name hash(archive) with the files described in fileset.

If two threads are running, the second maybe faster than the first .. so there are two points where io conflicts may happen.. and i check both cases for race conditions (extracted files will use atomic move, which can fail, write hash file is a single write which can fail).

I think this change will make everything much more stable..

Thanks a lot for this problem:)