S3 configuration: request for help

[Antiarchitect]

Hi, @schabdo seems like I need your help again. I'm trying to test the S3 extension first against true AWS S3 and MinIO later. Currently, I have application-custom.properties:

org.eclipse.hawkbit.artifact.repository.s3.enabled=true
org.eclipse.hawkbit.artifact.repository.s3.bucketName=my-bucket-name
aws.region=eu-central-1
aws.s3.endpoint=s3.eu-central-1.amazonaws.com
aws.accessKeyId=xxxxxxxxxxx
aws.secretAccessKey=xxxxxxxxxxxxxxxxx

I'm testing via GUI - Uploads section. Seems like artifacts are not using S3 storage at all. Is my configuration correct?

[Antiarchitect]

As I'm using the edge of hawkbit the dependency configuration in hawkbit-runtime/hawkbit-update-server/pom.xml

      <dependency>
          <groupId>org.eclipse.hawkbit</groupId>
          <artifactId>hawkbit-extension-artifact-repository-s3</artifactId>
          <version>${project.version}</version>
          <scope>system</scope>
          <systemPath>${project.basedir}/src/main/resources/hawkbit-extension-artifact-repository-s3.jar</systemPath>
      </dependency>

[Antiarchitect]

I've reviewed the code of this extension and it seems bucketName should be set via org.eclipse.hawkbit.repository.s3.bucketName=my-bucket-name (without artifact). Configuration namespaces for bucketName and enabled do not match - it's a bit confusing.

[schabdo]

bucketName should be set

Exactly. And I don't think you need to set aws.s3.endpoint. This should be handled by the AWS SDK automatically. The bucketName just needs to be the name of the bucket and of course hawkBit needs permission to access it.

I have application-custom.properties

If you named the property file like this you need to enable the profile custom in SpringBoot

[Antiarchitect]

Setting org.eclipse.hawkbit.repository.s3.bucketName=my-bucket-name has no success. is "--spring.profiles.active=custom" enough in Dockerfile ENTRYPOINT? Now I'm trying to disable aws.s3.entrypoint. Seems like the extension is not enabled at all

[schabdo]

This should do the trick:

ENTRYPOINT ["java","-Dspring.profiles.active=custom","-jar", ....

You can easily verify the activated profile in the console output by searching for this line:

INFO 1 --- [ main] org.eclipse.hawkbit.app.Start : The following profiles are active: custom

[Antiarchitect]

I see that profile is active - confirmed. I'm not sure if artifact-repository-s3 is enabled and I've set configuration properly.

[schabdo]

Just enable Spring Boot's debug mode via property -Ddebug=true(cf. logging to console) and you'll get all kinds of output. The one which is of interest is whether or not the S3RepositoryAutoConfiguration is loaded by Spring

[Antiarchitect]

Seems like negative - nothing similar in logs.

org.eclipse.hawkbit.artifact.repository.s3.enabled=true
org.eclipse.hawkbit.repository.s3.bucketName=enapter-hawkbit

Are these correct settings?

[Antiarchitect]

My Dockerfile is multistaged:

FROM maven:3.6.3-jdk-8-openj9 AS artifact-repository-s3

RUN set -x \
    && apt-get update \
    && apt-get install -y git \
    && apt-get clean -y
RUN set -x \
    && git clone https://github.com/eclipse/hawkbit-extensions.git /build \
    && cd /build \
    && git reset --hard 22a4c60a1072327179bbf1dcd4e65086895e55a0
RUN set -x \
    && cd /build/hawkbit-extension-artifact-repository-s3 \
    && mvn clean install -DskipTests

FROM maven:3.6.3-jdk-8-openj9 AS hawkbit

RUN set -x \
    && apt-get update \
    && apt-get install -y git \
    && apt-get clean -y
RUN set -x \
    && git clone https://github.com/Enapter/hawkbit.git /build \
    && cd /build \
    && git reset --hard d16c73f5406606167115934ba2a0fd511c763ef7

COPY --from=artifact-repository-s3 /build/hawkbit-extension-artifact-repository-s3/target/hawkbit-extension-artifact-repository-s3-0.3.0-SNAPSHOT.jar "/build/hawkbit-runtime/hawkbit-update-server/src/main/resources/hawkbit-extension-artifact-repository-s3.jar"

COPY ./application-enapter.properties /build/hawkbit-runtime/hawkbit-update-server/src/main/resources/

RUN set -x \
    && cd /build \
    && mvn clean install -DskipTests

FROM openjdk:8u242-jre-slim

ENV HAWKBIT_HOME=/opt/hawkbit

EXPOSE 8080

COPY --from=hawkbit /build/hawkbit-runtime/hawkbit-update-server/target/hawkbit-update-server-0.3.0-SNAPSHOT.jar "${HAWKBIT_HOME}/hawkbit-update-server.jar"

VOLUME "${HAWKBIT_HOME}/data"

WORKDIR $HAWKBIT_HOME

ENTRYPOINT ["java","-jar","hawkbit-update-server.jar","--debug", "--spring.profiles.active=enapter", "-Xms768m -Xmx768m -XX:MaxMetaspaceSize=250m -XX:MetaspaceSize=250m -Xss300K -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+UseCompressedOops -XX:+HeapDumpOnOutOfMemoryError"]

Am I doing build correctly?

[Antiarchitect]

@schabdo Should the dependencies be set in general hawkbit pom.xml file or just inside the hawkbit-update-server

[Antiarchitect]

Now I'm here:

 java.lang.IllegalStateException: Error processing condition on org.eclipse.hawkbit.artifact.repository.S3RepositoryAutoConfiguration.awsCredentialsProvider
    at org.springframework.boot.autoconfigure.condition.SpringBootCondition.matches(SpringBootCondition.java:64)
    at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:108)
    at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsForBeanMethod(ConfigurationClassBeanDefinitionReader.java:181)
    at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsForConfigurationClass(ConfigurationClassBeanDefinitionReader.java:141)
    at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitions(ConfigurationClassBeanDefinitionReader.java:117)
    at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:327)
    at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:232)
    at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:275)
    at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:95)
    at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:705)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:531)
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:142)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:775)
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:316)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248)
    at org.eclipse.hawkbit.app.Start.main(Start.java:35)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
    at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)
 Caused by: java.lang.IllegalStateException: @ConditionalOnMissingBean did not specify a bean using type, name or annotation and the attempt to deduce the bean's type failed
    at org.springframework.boot.autoconfigure.condition.OnBeanCondition$BeanSearchSpec.validate(OnBeanCondition.java:451)
    at org.springframework.boot.autoconfigure.condition.OnBeanCondition$BeanSearchSpec.<init>(OnBeanCondition.java:441)
    at org.springframework.boot.autoconfigure.condition.OnBeanCondition$BeanSearchSpec.<init>(OnBeanCondition.java:416)
    at org.springframework.boot.autoconfigure.condition.OnBeanCondition.getMatchOutcome(OnBeanCondition.java:158)
    at org.springframework.boot.autoconfigure.condition.SpringBootCondition.matches(SpringBootCondition.java:47)
    ... 25 common frames omitted
 Caused by: org.springframework.boot.autoconfigure.condition.OnBeanCondition$BeanTypeDeductionException: Failed to deduce bean type for org.eclipse.hawkbit.artifact.repository.S3RepositoryAutoConfiguration.awsCredentialsProvider
    at org.springframework.boot.autoconfigure.condition.OnBeanCondition$BeanSearchSpec.addDeducedBeanTypeForBeanMethod(OnBeanCondition.java:496)
    at org.springframework.boot.autoconfigure.condition.OnBeanCondition$BeanSearchSpec.addDeducedBeanType(OnBeanCondition.java:483)
    at org.springframework.boot.autoconfigure.condition.OnBeanCondition$BeanSearchSpec.<init>(OnBeanCondition.java:435)
    ... 28 common frames omitted
 Caused by: java.lang.ClassNotFoundException: com.amazonaws.auth.AWSCredentialsProvider
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
    at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:93)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.springframework.util.ClassUtils.forName(ClassUtils.java:275)
    at org.springframework.boot.autoconfigure.condition.OnBeanCondition$BeanSearchSpec.getReturnType(OnBeanCondition.java:505)
    at org.springframework.boot.autoconfigure.condition.OnBeanCondition$BeanSearchSpec.addDeducedBeanTypeForBeanMethod(OnBeanCondition.java:491)
    ... 30 common frames omitted

And the Dockerfile is:

FROM maven:3.6.3-jdk-8-openj9 AS artifact-repository-s3

RUN set -x \
    && apt-get update \
    && apt-get install -y git \
    && apt-get clean -y
RUN set -x \
    && git clone https://github.com/eclipse/hawkbit-extensions.git /build \
    && cd /build \
    && git reset --hard 22a4c60a1072327179bbf1dcd4e65086895e55a0
RUN set -x \
    && cd /build/hawkbit-extension-artifact-repository-s3 \
    && mvn clean install -DskipTests

FROM maven:3.6.3-jdk-8-openj9 AS hawkbit

RUN set -x \
    && apt-get update \
    && apt-get install -y git \
    && apt-get clean -y
RUN set -x \
    && git clone https://github.com/Enapter/hawkbit.git /build \
    && cd /build \
    && git reset --hard d91f10a82242f8c2fe1b2d3e0cb0615779a12b74

COPY --from=artifact-repository-s3 /build/hawkbit-extension-artifact-repository-s3/target/hawkbit-extension-artifact-repository-s3-0.3.0-SNAPSHOT.jar "/build/hawkbit-runtime/hawkbit-update-server/src/main/resources/hawkbit-extension-artifact-repository-s3.jar"

COPY ./application-enapter.properties /build/hawkbit-runtime/hawkbit-update-server/src/main/resources/

RUN set -x \
    && cd /build \
    && mvn install:install-file \
        -Dfile=/build/hawkbit-runtime/hawkbit-update-server/src/main/resources/hawkbit-extension-artifact-repository-s3.jar \
        -DgroupId=org.eclipse.hawkbit \
        -DartifactId=hawkbit-extension-artifact-repository-s3 \
        -Dversion=SNAPSHOT \
        -Dpackaging=jar \
        -DgeneratePom=true \
    && mvn clean install -DskipTests

FROM openjdk:8u242-jre-slim

ENV HAWKBIT_HOME=/opt/hawkbit

EXPOSE 8080

COPY --from=hawkbit /build/hawkbit-runtime/hawkbit-update-server/target/hawkbit-update-server-0.3.0-SNAPSHOT.jar "${HAWKBIT_HOME}/hawkbit-update-server.jar"

VOLUME "${HAWKBIT_HOME}/data"

WORKDIR $HAWKBIT_HOME

ENTRYPOINT ["java","-jar","hawkbit-update-server.jar","--spring.profiles.active=enapter", "-Xms768m -Xmx768m -XX:MaxMetaspaceSize=250m -XX:MetaspaceSize=250m -Xss300K -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+UseCompressedOops -XX:+HeapDumpOnOutOfMemoryError"]

[schabdo]

Interesting, I never tried to build S3 extension separately and copy the artefact later on straight into the hawkbit-update-server.

Should the dependencies be set in general hawkbit pom.xml file or just inside the hawkbit-update-server

Exactly, that's the way I do it. Just add the maven dependency to the hawkbit-update-server pom.xml so as soon as you build the hawkbit-update-server Maven will do all the transitive dependency resolution for you and you shouldn't get ClassNotFoundException due to missing AWS SDK on the classpath

[schabdo]

@thmai11 may I ask you for help? Maybe you already have something similar up and running in a single Dockerfile. That would be awesome!

[thmai11]

Hello, You may take a look at my multi-stage docker file here : https://github.com/thmai11/hawkbit-s3-backend/blob/master/Dockerfile. Also check the read me for more detailed explanation. I don't see any problem building the extension in a separate stage but I don't see it added as a dependencies to hawkbit via the pom.xml which might be the issue? My docker is also available here: https://hub.docker.com/u/drgnzn Let me know if I can be of any help!

[Antiarchitect]

@schabdo @thmai11 Now I see. Thank you, guys. Will try when I have time for this.

[Antiarchitect]

Current error:

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.amazonaws.services.s3.AmazonS3]: Factory method 'amazonS3' threw exception; nested exception is java.lang.IllegalStateException: Only one of Region or EndpointConfiguration may be set.

But seems I have fix for that - will create a new PR

[thmai11]

Only one of Region or EndpointConfiguration may be set. Could you provide us the argument you send to hawkbit server when you launch it?

[Antiarchitect]

Only one of Region or EndpointConfiguration may be set. Could you provide us the argument you send to hawkbit server when you launch it?

Wait a bit - if I can fix it by myself I will create a PR

[Antiarchitect]

Here it goes: https://github.com/eclipse/hawkbit-extensions/pull/55