search

eclipse / hawkbit

Eclipse hawkBit™
https://projects.eclipse.org/projects/iot.hawkbit
Eclipse Public License 2.0
444 stars 186 forks source link

Hawkbit cannot send packages to clients. (Download failed: HTTP request failed: 500) #1760

Closed dHARIZ-25 closed 1 day ago

dHARIZ-25 commented 3 days ago

Hello,

I am deploying hawkbit via docker compose. I am modifiying the SPRING_APPLICATION_JSON parameters in a way that permits me to:

It was alll working fine until last week I added the second admin user and restarted the container. the hawkbit works fine and the machines connect normally but they do not accept the download of the package.

I set my rauc client debug level to debug and rettempted the download. I get this: DEBUG: Response body: { "id" : "41", "deployment" : { "download" : "forced", "update" : "skip", "chunks" : [ { "part" : "os", "version" : "0.36.15_Test", "name" : "SM_0.36.15_Test", "artifacts" : [ { "filename" : "xxx-x.raucb", "hashes" : { "sha1" : "ea41401dac0d616837591a488c8e81e2e0c3366b", "md5" : "a5d90d809b58c29d8ca18ee306b5986f", "sha256" : "dec93707461ef4e2c3bf77d90de522ca4d18366568a605ff02b1f72a9e2d8a48" }, "size" : 111998949, "_links" : { "download-http" : { "href" : "https://link2portal:443/DEFAULT/controller/v1/machine_name/softwaremodules/7/artifacts/xxx-x.raucb" }, "md5sum-http" : { "href" : "https://link2portal:443/DEFAULT/controller/v1/machine_name/softwaremodules/7/artifacts/xxx-x.raucb.MD5SUM" } } } ] } ] } } MESSAGE: hawkBit requested to skip installation, not invoking RAUC yet. MESSAGE: New software ready for download (Name: SM_0.36.15_Test, Version: 0.36.15_Test, Size: 111998949 bytes, URL: https://link2portal:443/DEFAULT/controller/v1/machine_name/softwaremodules/7/artifacts/xxx-x.raucb) MESSAGE: Start downloading: link2portal:443/DEFAULT/controller/v1/machine_name/softwaremodules/7/artifacts/xxx-x.raucb WARNING: Download failed: HTTP request failed: 500

My docker compose looks like this:

`#
# Copyright (c) 2018 Bosch Software Innovations GmbH and others
#
# This program and the accompanying materials are made
# available under the terms of the Eclipse Public License 2.0
# which is available at https://www.eclipse.org/legal/epl-2.0/
#
# SPDX-License-Identifier: EPL-2.0
#
version: '3'

services:

  # ---------------------
  # RabbitMQ service
  # ---------------------
  rabbitmq:
    image: "rabbitmq:3-management"
    restart: always
    ports:
      - "15672:15672"
      - "5672:5672"
    labels:
      NAME: "rabbitmq"

  # ---------------------
  # MySQL service
  # ---------------------
  mysql:
    image: "mysql:8.0"
    environment:
      MYSQL_DATABASE: "hawkbit"
      # MYSQL_USER: "root" is created by default in the container for mysql 8.0+
      MYSQL_ALLOW_EMPTY_PASSWORD: "false"
      MYSQL_USER: ${MYSQL_USER}
      MYSQL_PASSWORD: ${MYSQL_USER_PASSWORD}
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
    restart: always
    ports:
      - "3306:3306"
    healthcheck:
      test: [ "CMD", "mysqladmin" ,"ping", "-h", "localhost" ]
      timeout: 20s
      retries: 10
    volumes:
      - ./mysql_data:/var/lib/mysql
    labels:
      NAME: "mysql"

  # ---------------------
  # HawkBit service
  # ---------------------
  hawkbit:
    image: "hawkbit/hawkbit-update-server:latest-mysql"
    environment:
      - 'SPRING_DATASOURCE_URL=jdbc:mariadb://mysql:3306/hawkbit?allowPublicKeyRetrieval=true&useSSL=false'
      - 'SPRING_RABBITMQ_HOST=rabbitmq'
      - 'SPRING_RABBITMQ_USERNAME=${RABBITMQ_USER}'
      - 'SPRING_RABBITMQ_PASSWORD=${RABBITMQ_PASS}'
      - 'SPRING_DATASOURCE_USERNAME=root'
      - 'SPRING_DATASOURCE_PASSWORD=${MYSQL_ROOT_PASSWORD}'
      - 'SPRING_APPLICATION_JSON = {
        "server.use-forward-headers": "true",
        "hawkbit.server.ddi.security.authentication.anonymous.enabled":"false",
        "hawkbit.server.ddi.security.authentication.gatewaytoken.enabled":"true",
        "hawkbit.server.ddi.security.authentication.gatewaytoken.key":"xxxxxxxxxxxxxxxxxxxxxxxx",
        "hawkbit.server.im.users[0].username": "${HAWKBIT_ADMIN_USER}",
        "hawkbit.server.im.users[0].password": "{noop}${HAWKBIT_ADMIN_USER_PASSWORD}",
        "hawkbit.server.im.users[0].firstname": "ADMIN",
        "hawkbit.server.im.users[0].lastname": "ACCOUNT",
        "hawkbit.server.im.users[0].permissions": "ALL",

        "hawkbit.server.im.users[1].username": "${HAWKBIT_GUEST_USER}",
        "hawkbit.server.im.users[1].password": "{noop}${HAWKBIT_GUEST_USER_PASSWORD}",
        "hawkbit.server.im.users[1].firstname": "GUEST",
        "hawkbit.server.im.users[1].lastname": "ACCOUNT",
        "hawkbit.server.im.users[1].permissions": "READ_TARGET,UPDATE_TARGET,CREATE_TARGET,DELETE_TARGET,READ_REPOSITORY,UPDATE_REPOSITORY,CREATE_REPOSITORY,DELETE_REPOSITORY,DOWNLOAD_REPOSITORY_ARTIFACT,READ_ROLLOUT,CREATE_ROLLOUT,HANDLE_ROLLOUT,APPROVE_ROLLOUT,DELETE_ROLLOUT,UPDATE_ROLLOUT",

        "hawkbit.server.im.users[2].username": "${HAWKBIT_ADMIN_TKR}",
        "hawkbit.server.im.users[2].password": "{noop}${HAWKBIT_ADMIN_TKR_PASSWORD}",
        "hawkbit.server.im.users[2].firstname": "ADMIN",
        "hawkbit.server.im.users[2].lastname": "ACCOUNT",
        "hawkbit.server.im.users[2].permissions": "ALL",

        "hawkbit.artifact.url.protocols.download-http.rel":"download-http",
        "hawkbit.artifact.url.protocols.download-http.protocol": "https",
        "hawkbit.artifact.url.protocols.download-http.port": "443",
        "hawkbit.server.forward-headers-strategy":"NATIVE",
        "hawkbit.artifact.url.protocols.download-http.hostname": "link2portal",
        "hawkbit.artifact.url.protocols.download-http.ref": "{protocolRequest}://{hostnameRequest}:{port}/{tenant}/controller/v1/{controllerId}/softwaremodules/{softwareModuleId}/artifacts/{artifactFileName}"
}'
    restart: always
    ports:
      - "8080:8080"
    depends_on:
      mysql:
        condition: service_healthy
    labels:
      NAME: "hawkbit"
volumes:
  artifactrepo:
    driver: local
``

Please let me know if i have anything wrong or outdated in my compose configuration.

strailov commented 2 days ago

Hello @dHARIZ-25 !

Are you able to see any logs from the hawkBit container ?

dHARIZ-25 commented 2 days ago

Hello again! Yess that is what I get from the compose logs:

`docker-hawkbit-1   | 2024-07-02 11:22:35.223  INFO 1 --- [           main] org.eclipse.hawkbit.app.Start            : Started Start in 26.207 seconds (JVM running for 27.096)
docker-hawkbit-1   | 2024-07-02 11:22:55.058  INFO 1 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
docker-hawkbit-1   | 2024-07-02 11:22:55.059  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
docker-hawkbit-1   | 2024-07-02 11:22:55.065  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 6 ms
docker-hawkbit-1   | 2024-07-02 11:26:08.698  INFO 1 --- [nio-8080-exec-6] o.v.spring.servlet.Vaadin4SpringServlet  : Found 1 custom init parameter provider(s)
docker-hawkbit-1   | 2024-07-02 11:26:08.713  INFO 1 --- [nio-8080-exec-6] o.v.s.s.Vaadin4SpringServletService      : Using custom Vaadin4Spring servlet service
docker-hawkbit-1   | 2024-07-02 11:26:08.745  INFO 1 --- [nio-8080-exec-6] o.v.spring.servlet.Vaadin4SpringServlet  : Using custom SystemMessagesProvider org.eclipse.hawkbit.ui.LocalizedSystemMessagesProvider@7e9e73
docker-hawkbit-1   | 2024-07-02 11:26:08.749  INFO 1 --- [nio-8080-exec-6] o.v.spring.servlet.Vaadin4SpringServlet  : Custom Vaadin4Spring servlet initialization completed
docker-hawkbit-1   | 2024-07-02 12:12:52.896  INFO 1 --- [nio-8080-exec-4] o.apache.coyote.http11.Http11Processor   : Error parsing HTTP request header
docker-hawkbit-1   |  Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.
docker-hawkbit-1   | 
docker-hawkbit-1   | 
docker-hawkbit-1   | java.lang.IllegalArgumentException: Invalid character found in the request target [/index.php?s=/index/\think\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=Hello ]. The valid characters are defined in RFC 7230 and RFC 3986
docker-hawkbit-1   |    at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:482)
docker-hawkbit-1   |    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:263)
docker-hawkbit-1   |    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
docker-hawkbit-1   |    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928)
docker-hawkbit-1   |    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1794)
docker-hawkbit-1   |    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
docker-hawkbit-1   |    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
docker-hawkbit-1   |    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
docker-hawkbit-1   |    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
docker-hawkbit-1   |    at java.base/java.lang.Thread.run(Unknown Source)
docker-hawkbit-1   | 
docker-hawkbit-1   | 
docker-hawkbit-1   | 2024-07-02 15:13:53.414  INFO 1 --- [nio-8080-exec-3] o.apache.coyote.http11.Http11Processor   : Error parsing HTTP request header
docker-hawkbit-1   |  Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.
docker-hawkbit-1   | 
docker-hawkbit-1   | 
docker-hawkbit-1   | java.lang.IllegalArgumentException: Invalid character found in the request target [/index.php?s=/index/\think\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=Hello ]. The valid characters are defined in RFC 7230 and RFC 3986
docker-hawkbit-1   |    at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:482)
docker-hawkbit-1   |    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:263)
docker-hawkbit-1   |    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
docker-hawkbit-1   |    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928)
docker-hawkbit-1   |    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1794)
docker-hawkbit-1   |    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
docker-hawkbit-1   |    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
docker-hawkbit-1   |    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
docker-hawkbit-1   |    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
docker-hawkbit-1   |    at java.base/java.lang.Thread.run(Unknown Source)
docker-hawkbit-1   | 
docker-hawkbit-1   | 
docker-hawkbit-1   | 2024-07-02 18:46:58.582  WARN 1 --- [nio-8080-exec-2] .w.s.m.s.DefaultHandlerExceptionResolver : Resolved [org.springframework.web.HttpRequestMethodNotSupportedException: Request method 'POST' not supported]
docker-hawkbit-1   | 2024-07-02 23:17:04.041  WARN 1 --- [nio-8080-exec-5] .w.s.m.s.DefaultHandlerExceptionResolver : Resolved [org.springframework.web.HttpRequestMethodNotSupportedException: Request method 'POST' not supported]
docker-hawkbit-1   | 2024-07-03 00:51:39.211  WARN 1 --- [nio-8080-exec-5] .w.s.m.s.DefaultHandlerExceptionResolver : Resolved [org.springframework.web.HttpRequestMethodNotSupportedException: Request method 'POST' not supported]`
strailov commented 2 days ago

The problem seems to be that Tomcat does not like the url. In this case th URL seems to be well formed. However, Tomcat might be interpreting some part of it as containing invalid characters. I suspect this machinename controllerId might be problematic - is there a way of changing it without underscore ( ) ?

But the error message seems to be for other url - /index.php?s=/index/\think\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=Hello

I think [ ] are forbidden also

dHARIZ-25 commented 2 days ago

The download doesn't work for all machines, I used to be able to download normally. So I don't think it's the name (please check the screenshot and tell me if the format can be problematic) image

But after adding the seconnd admin user and running docker compose up -d. The link parsing doesnt seem to work anymore. image Can it be a change in the latest docker image or spring boot uncompatibility?

I usually never encounter this problem updating my machines until 28-06-2024 when I modified the user priviledges and ran sudo docker compose down && sudo docker compose up -d.

This is the link in a clearer format from another machine's logs:

MESSAGE: Start downloading: https://udte-prtl.lrh.cloud:443/DEFAULT/controller/v1/4153-3414-0002-5/softwaremodules/7/artifacts/xxx-xxxx-xxx-stm32mp1-xxxx.raucb
WARNING: Download failed: HTTP request failed: 500
strailov commented 2 days ago

This seems valid also ... Any idea where does /index.php?s=/index/\think\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=Hello may come from ?

dHARIZ-25 commented 2 days ago

No Idea honestly, I have never built/tested that url. I thought it was coming from hawkbit at first.

dHARIZ-25 commented 2 days ago

I think I knew from where it comes see here (you may need to translate the page 😆 )

Any idea how to update tomcat configuration?

strailov commented 1 day ago

Didn't manage to translate the page :( But you can configure tomcat in application.properties. You can check this tutorial.

dHARIZ-25 commented 1 day ago

Hello, I wanted to say that I could surpass this error by simply:

  1. Delete the Software Module
  2. Recreate the exact same Distribution with the newly created SM.
  3. Reattempt the update.
  4. Verify the download on client (-->Success)

I think whenever I redeploy my Hawkbit server I need to reupload the package in the SM. I am still not sure why this happens but for now I can work with this workaround.

Thank you for your support :) I will mark this as closed!

strailov commented 1 day ago

This could be related to the Artifact Storage - if you are using local storage could cause problems with docker I guess.