search

Aiven-Open / tiered-storage-for-apache-kafka

RemoteStorageManager for Apache Kafka® Tiered Storage
Apache License 2.0
91 stars 19 forks source link

Build Docker image before E2E tests #413

Open ivanyu opened 12 months ago

ivanyu commented 12 months ago

Currently, we build the image as a part of E2E test stage. Since E2E run in a type matrix, we build the image multiple times.

jeqo commented 6 months ago

There are some mismatches that make this challenging: we build the library against different JDK versions, but package it in the same docker base image -- so effectively it runs with the same JDK version. Then the JDK version matching between build and e2e doesn't matter much as the e2e is in a standalone classpath -- so there's no dependency between JDK versions to care here.

Another mismatch is that we need to build the image once, and pass it to a 2nd set of parallel jobs. To pass docker image between jobs we are constraint to upload/download: https://docs.docker.com/build/ci/github-actions/share-image-jobs/ which means ~1min operation with a ~1.2GB image size (mostly Confluent base image)

Leaving this for context if someone would like to tackle it.