search

eclipse / hawkbit

Eclipse hawkBit™
https://projects.eclipse.org/projects/iot.hawkbit
Eclipse Public License 2.0
444 stars 186 forks source link

Change hawkbit with aws s3 storage from https download links to http links temporary #1738

Open MaMelnik opened 1 month ago

MaMelnik commented 1 month ago

Hello,

we use the hawkbit update server version 0.4.1 together with the aws s3 extension to update our devices. The hawkbit runs in a docker container and stores the settings in a maria db. Artifacts are stored in the aws s3.

The system works fine but because of an error in the production a few devices did not get the right firmware with the certificate to use https. To fix the issue we have to update the devices using http. The aws s3 download link can be modified from "https:..." to "http:..." and the download works (tested with my web browser).

I tried to changed the following properties:

hawkbit.artifact.url.protocols.download-http.protocol=http hawkbit.artifact.url.protocols.download-cdn-http.protocol=http

But i still get https download links. Maybe these properties are not effective because of the s3 extension? Is there a possibility to make direct changes in the source code? I only need the modified hawkbit to update a few devices and after that i can switch back to the encrypted version.

I am currently digging into the source code and trying to figure out where the links are generated and how to modify them. I do not have a local test environment at the moment so it is quite difficult to debug. Is there a howto/manual available to set it up?

My colleague who set the whole system up is not available anymore so i have to deal with the problem myself.

Thanks in advance for any help.

strailov commented 1 month ago

Hello @MaMelnik ! Thanks for your interest in hawkBit!

Could you provide more insights of how is your instance of hawkbit configured to work with s3 ?

MaMelnik commented 1 month ago

Hello strailov, thanks for the quick response.

The s3 settings in the application.properties (hawkbit-runtime\hawkbit-update-server\src\main\resources):

server.use-forward-headers=true
hawkbit.artifact.url.protocols.download-http.rel=download-http
hawkbit.artifact.url.protocols.download-http.protocol=http
hawkbit.artifact.url.protocols.download-http.supports=DMF,DDI
hawkbit.artifact.url.protocols.download-http.hostname=${AWS_BUCKET}.s3.${AWS_REGION}.amazonaws.com
hawkbit.artifact.url.protocols.download-http.ref={protocol}://{hostname}/{tenant}/{artifactSHA1}

hawkbit.artifact.url.protocols.md5sum-http.rel=md5sum-http
hawkbit.artifact.url.protocols.md5sum-http.protocol=${hawkbit.artifact.url.protocols.download-http.protocol}
hawkbit.artifact.url.protocols.md5sum-http.supports=DDI
hawkbit.artifact.url.protocols.md5sum-http.hostname=${hawkbit.artifact.url.protocols.download-http.hostname}
hawkbit.artifact.url.protocols.md5sum-http.ref=${hawkbit.artifact.url.protocols.download-http.ref}.MD5SUM

# AWS S3 Access Configuration
org.eclipse.hawkbit.artifact.repository.s3.enabled=true
org.eclipse.hawkbit.repository.s3.bucketName=${AWS_BUCKET}
aws.region=${AWS_REGION}
aws.accessKeyId=${AWS_KEY}
aws.secretKey=${AWS_SECRET}

The s3 extension is included in the pom.xml dependencies (hawkbit-runtime\hawkbit-update-server):

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.eclipse.hawkbit</groupId>
        <artifactId>hawkbit-runtime-parent</artifactId>
        <version>${revision}</version>
    </parent>
    <artifactId>hawkbit-update-server</artifactId>
    <name>hawkBit :: Runtime :: Update Server (Monolith)</name>

    <properties>
        <spring.app.class>org.eclipse.hawkbit.doc.Start</spring.app.class>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.eclipse.hawkbit</groupId>
            <artifactId>hawkbit-boot-starter</artifactId>
            <version>${project.version}</version>
        </dependency>
        <dependency>
            <groupId>org.mariadb.jdbc</groupId>
            <artifactId>mariadb-java-client</artifactId>
            <version>3.3.2</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>8.0.33</version>
      </dependency>
      <dependency>
            <groupId>org.eclipse.hawkbit</groupId>
            <artifactId>hawkbit-extension-artifact-repository-s3</artifactId>
            <version>${project.version}</version>
      </dependency>
    </dependencies>
</project>
strailov commented 1 month ago

I did some digging and in the code it seems that the extension creates ClientConfiguration with default constructor. So it seems that the client config itself should hold info for tthe protocol ... However I do not see how it could be configured without changes ... So I suppose here the s3 is responsible for the protocol and the extension does not provide a way to configure it.

MaMelnik commented 1 month ago

Thanks for the assistance.

I am not sure what you mean with ClientConfiguration. Do you have a hint where to look in the source code?

It seems that changing the aws s3 settings from https to http is not possible in the configuration interface.

At the moment I only see two possible solutions:

  1. Change the source code (for example InputStream in the aws s3 extension) to hard-coded file/link with http-source
  2. Build a modified version of the hawkbit without s3 storage. Use the default storage function to update the affected devices and switch back to the previous version with s3 storage afterwards

I would prefer the first solution, but I am not quite sure where the best place for this modification in the source code is. Do you have an idea? I am currently looking at the s3 extensions public InputStream getFileInputStream() function see source code

Because it is only a temporary problem I am not looking for a solution with a switch in the configuration. I only need to update the devices with missing certificates and I am fine.

strailov commented 1 month ago

I was looking at awsClientConfiguration() method in here I suspect here could be somehow configured ...

MaMelnik commented 1 month ago

Thanks for the hint.

I tried to change the aws extension but it seems to crash the hawkbit even without any source code modifications. Maybe you have an idea what is wrong with my custom version. These are the changes I made:

1. Get source code and change version I took the current version (master) from github and build a jar file from it with version 1.0.0 (changed the version in hawkbit-extensions/pom.xml)

<properties>
 <revision>1.0.0</revision>

2. Build the custom extension Build cmd C:\eclipse\hawkbit-extensions\hawkbit-extension-artifact-repository-s3> mvn clean install

3. Copy the custom extension I moved the output file (hawkbit-extension-artifact-repository-s3-1.0.0.jar) to \hawkbit\hawkbit-runtime\hawkbit-update-server\libs\hawkbit-extensions\hawkbit-extension-artifact-repository-s3\1.0.0

4. Add custom extension location to repositories and dependencies The custom s3 extension is included in the pom.xml dependencies (hawkbit-runtime\hawkbit-update-server):

<repositories>
  <repository>
      <id>in-project</id>
      <name>els aws s3 custom</name>
      <url>file://${project.basedir}/libs</url>
  </repository>
</repositories>

...

<!--
      <dependency>
            <groupId>org.eclipse.hawkbit</groupId>
            <artifactId>hawkbit-extension-artifact-repository-s3</artifactId>
            <version>${project.version}</version>
      </dependency>
-->
      <dependency>
            <groupId>hawkbit-extensions</groupId>
            <artifactId>hawkbit-extension-artifact-repository-s3</artifactId>
            <version>1.0.0</version>
      </dependency>

The maven build process for the extension and the hawkbit works, but there are some warnings during the hawkbit-server build. I do not think they are relevant, but here are the ones related to the extension:

[WARNING] The POM for hawkbit-extensions:hawkbit-extension-artifact-repository-s3:jar:1.0.0 is missing, no dependency information available
[WARNING] The artifact mysql:mysql-connector-java:jar:8.0.33 has been relocated to com.mysql:mysql-connector-j:jar:8.0.33: MySQL Connector/J artifacts moved to reverse-DNS compliant Maven 2+ coordinates.
Downloading from in-project: file://C:\hawkbit\hawkbit-runtime\hawkbit-update-server/libs/hawkbit-extensions/hawkbit-extension-artifact-repository-s3/1.0.0/hawkbit-extension-artifact-repository-s3-1.0.0.jar
[WARNING] Could not validate integrity of download from file://C:\hawkbit\hawkbit-runtime\hawkbit-update-server/libs/hawkbit-extensions/hawkbit-extension-artifact-repository-s3/1.0.0/hawkbit-extension-artifact-repository-s3-1.0.0.jar
org.eclipse.aether.transfer.ChecksumFailureException: Checksum validation failed, no checksums available
[WARNING] Checksum validation failed, no checksums available from in-project for file://C:\hawkbit\hawkbit-runtime\hawkbit-update-server/libs/hawkbit-extensions/hawkbit-extension-artifact-repository-s3/1.0.3/hawkbit-extension-artifact-repository-s3-1.0.3.jar
...

The hawkbit crashes during the startup:

Eclipse hawkBit Update Server  (v0.4.1)
using Spring Boot  (v2.7.18)
Go to https://www.eclipse.org/hawkbit for more information.
INFO 1 --- [           main] org.eclipse.hawkbit.app.Start            : Starting Start v0.4.1 using Java 17.0.2 on ea26819efedd with PID 1 (/opt/hawkbit/hawkbit-update-server.jar started by hawkbit in /opt/hawkbit)
INFO 1 --- [           main] org.eclipse.hawkbit.app.Start            : The following 1 profile is active: "mysql"
ERROR 1 --- [           main] o.s.boot.SpringApplication               : Application run failed
java.lang.IllegalStateException: Error processing condition on org.eclipse.hawkbit.artifact.repository.S3RepositoryAutoConfiguration.awsCredentialsProvider
at org.springframework.boot.autoconfigure.condition.SpringBootCondition.matches(SpringBootCondition.java:60)
...

I am not sure what causes the issue, because the extension code is unchanged.

MaMelnik commented 1 month ago

I did some further investigation and switched to my local machine for the tests. Now i get more error details. I think the issue is a ClassNotFoundException Caused by: java.lang.ClassNotFoundException: com.amazonaws.ClientConfiguration

Full log:

 ______     _ _                  _                    _    ____  _ _
 |  ____|   | (_)                | |                  | |  |  _ \(_) |
 | |__   ___| |_ _ __  ___  ___  | |__   __ ___      _| | _| |_) |_| |_
 |  __| / __| | | '_ \/ __|/ _ \ | '_ \ / _` \ \ /\ / / |/ /  _ <| | __|
 | |___| (__| | | |_) \__ \  __/ | | | | (_| |\ V  V /|   <| |_) | | |_
 |______\___|_|_| .__/|___/\___| |_| |_|\__,_| \_/\_/ |_|\_\____/|_|\__|
                | |
                |_|

Eclipse hawkBit Update Server  (v0.4.1)
using Spring Boot  (v2.7.18)

Go to https://www.eclipse.org/hawkbit for more information.

2024-05-29 16:27:14.022  INFO 1052 --- [           main] org.eclipse.hawkbit.app.Start            : Starting Start v0.4.1 using Java 17.0.11 on IMS-DEV with PID 1052 (C:\els\hawkbit\hawkbit11\hawkbit-runtime\hawkbit-update-server\target\hawkbit-update-server-0.4.1.jar started by imsadmin in C:\els\eclipse)
2024-05-29 16:27:14.107  INFO 1052 --- [           main] org.eclipse.hawkbit.app.Start            : No active profile set, falling back to 1 default profile: "default"
2024-05-29 16:27:16.324 ERROR 1052 --- [           main] o.s.boot.SpringApplication               : Application run failed

java.lang.IllegalStateException: Error processing condition on org.eclipse.hawkbit.artifact.repository.S3RepositoryAutoConfiguration.awsClientConfiguration
        at org.springframework.boot.autoconfigure.condition.SpringBootCondition.matches(SpringBootCondition.java:60)
        at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:108)
        at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsForBeanMethod(ConfigurationClassBeanDefinitionReader.java:193)
        at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsForConfigurationClass(ConfigurationClassBeanDefinitionReader.java:153)
        at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitions(ConfigurationClassBeanDefinitionReader.java:129)
        at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:343)
        at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:247)
        at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:311)
        at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:112)
        at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:756)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:572)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:147)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:732)
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:409)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:308)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1300)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1289)
        at org.eclipse.hawkbit.app.Start.main(Start.java:36)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Caused by: java.lang.IllegalStateException: @ConditionalOnMissingBean did not specify a bean using type, name or annotation and the attempt to deduce the bean's type failed
        at org.springframework.boot.autoconfigure.condition.OnBeanCondition$Spec.validate(OnBeanCondition.java:490)
        at org.springframework.boot.autoconfigure.condition.OnBeanCondition$Spec.<init>(OnBeanCondition.java:439)
        at org.springframework.boot.autoconfigure.condition.OnBeanCondition.getMatchOutcome(OnBeanCondition.java:155)
        at org.springframework.boot.autoconfigure.condition.SpringBootCondition.matches(SpringBootCondition.java:47)
        ... 25 common frames omitted
Caused by: org.springframework.boot.autoconfigure.condition.OnBeanCondition$BeanTypeDeductionException: Failed to deduce bean type for org.eclipse.hawkbit.artifact.repository.S3RepositoryAutoConfiguration.awsClientConfiguration
        at org.springframework.boot.autoconfigure.condition.OnBeanCondition$Spec.deducedBeanTypeForBeanMethod(OnBeanCondition.java:520)
        at org.springframework.boot.autoconfigure.condition.OnBeanCondition$Spec.deducedBeanType(OnBeanCondition.java:509)
        at org.springframework.boot.autoconfigure.condition.OnBeanCondition$Spec.<init>(OnBeanCondition.java:432)
        ... 27 common frames omitted
Caused by: java.lang.ClassNotFoundException: com.amazonaws.ClientConfiguration
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:592)
        at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:151)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:467)
        at org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition.resolve(FilteringSpringBootCondition.java:108)
        at org.springframework.boot.autoconfigure.condition.OnBeanCondition$Spec.getReturnType(OnBeanCondition.java:528)
        at org.springframework.boot.autoconfigure.condition.OnBeanCondition$Spec.deducedBeanTypeForBeanMethod(OnBeanCondition.java:516)
        ... 29 common frames omitted

I thought it could be caused by version difference in the dependencies but i did not find any difference (mvn dependency:tree)

Is there someone with the same issue? Most questions/error reports I can find are because of dependency problems.

I will try to solve this but I am not a maven expert.