search

xirixiz / dsmr-reader-docker

DSMR Reader in Docker.
https://hub.docker.com/r/xirixiz/dsmr-reader-docker
112 stars 33 forks source link

Unable to boot after recent upgrade #315

Closed Diondk closed 1 year ago

Diondk commented 1 year ago

Support guidelines

I've found an issue and checked that ...

Description

I updated dsmr reader to the latest version that has been published today and i get this error.

I am running on unraid

Expected behaviour

Should start normally

Actual behaviour

Fails to start

Steps to reproduce

upgrade from the version in december to the latest

Docker info

n/a i am running unraid

Version

Linux Tower 5.19.17-Unraid #2 SMP PREEMPT_DYNAMIC Wed Nov 2 11:54:15 PDT 2022 x86_64 Intel(R) Core(TM) i5-10400F CPU @ 2.90GHz GenuineIntel GNU/Linux

Docker compose

na

Container logs

[2023-01-30 10:49:00 +0100] [771] [INFO] Starting gunicorn 20.1.0 [2023-01-30 10:49:00 +0100] [771] [INFO] Listening at: unix:/tmp/gunicorn--dsmr_webinterface.socket (771) [2023-01-30 10:49:00 +0100] [771] [INFO] Using worker: sync [2023-01-30 10:49:00 +0100] [776] [INFO] Booting worker with pid: 776 [2023-01-30 10:49:00 +0100] [776] [ERROR] Exception in worker process Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker worker.init_process() File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/base.py", line 134, in init_process self.load_wsgi() File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi self.wsgi = self.app.wsgi() ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/gunicorn/app/base.py", line 67, in wsgi self.callable = self.load() ^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 58, in load return self.load_wsgiapp() ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp return util.import_app(self.app_uri) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/gunicorn/util.py", line 359, in import_app mod = importlib.import_module(module) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 1206, in _gcd_import File "", line 1178, in _find_and_load File "", line 1149, in _find_and_load_unlocked File "", line 690, in _load_unlocked File "", line 940, in exec_module File "", line 241, in _call_with_frames_removed File "/app/dsmrreader/wsgi.py", line 16, in application = get_wsgi_application() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application django.setup(set_prefix=False) File "/usr/local/lib/python3.11/site-packages/django/init.py", line 19, in setup configure_logging(settings.LOGGING_CONFIG, settings.LOGGING) ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/django/conf/init.py", line 82, in getattr self._setup(name) File "/usr/local/lib/python3.11/site-packages/django/conf/init.py", line 69, in _setup self._wrapped = Settings(settings_module) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/django/conf/init.py", line 170, in init mod = importlib.import_module(self.SETTINGS_MODULE) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 1206, in _gcd_import File "", line 1178, in _find_and_load File "", line 1149, in _find_and_load_unlocked File "", line 690, in _load_unlocked File "", line 940, in exec_module File "", line 241, in _call_with_frames_removed File "/app/dsmrreader/settings.py", line 4, in from dsmrreader.config.production import File "/app/dsmrreader/config/production.py", line 1, in from dsmrreader.config.defaults import File "/app/dsmrreader/config/defaults.py", line 16, in DSMRREADER_LOGLEVEL = config( ^^^^^^^ File "/usr/local/lib/python3.11/site-packages/decouple.py", line 245, in call return self.config(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/decouple.py", line 107, in call return self.get(args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/decouple.py", line 101, in get return cast(value) ^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/decouple.py", line 309, in call raise ValueError(( ValueError: Value not in list: 'warning'; valid values are ['DEBUG', 'WARNING', 'ERROR'] [2023-01-30 10:49:00 +0100] [776] [INFO] Worker exiting (pid: 776) [2023-01-30 10:49:00 +0100] [771] [INFO] Shutting down: Master [2023-01-30 10:49:00 +0100] [771] [INFO] Reason: Worker failed to boot. Starting DSMR Reader - webinterface... Starting DSMR Reader - backend...

Additional info

No response

mupsje commented 1 year ago

same here indeed

xirixiz commented 1 year ago

Hi, you need to provide a little more info. Can you share your docker-compose file? @mupsje do you also have unraid? Do you both use a tool like portainer?

The error is clear at least. DSMRREADER_LOGLEVEL has been set, but incorrectly (warning instead of WARNING).

mupsje commented 1 year ago

No, I use Docker Compose inside container station QNAP Yes I use Portainer also

version: '3'

services:
  dsmrdb:
    # When using Postgres, release 13.x is supported only
    # due to the limited availability of client packages, especially for arm32v7
    image: postgres:14-alpine
    container_name: dsmrdb
    restart: unless-stopped
    volumes:
      - /share/docker/dsmrdb:/var/lib/postgresql/data
    networks:
      vlan86:
        ipv4_address: ##################
    mac_address: ###############
    environment:
      - PUID=502
      - PGID=100
      - TZ=Europe/Amsterdam
      - PG_TZ=Europe/Amsterdam
      - POSTGRES_USER=dsmrreader
      - POSTGRES_PASSWORD=dsmrreader
      - POSTGRES_DB=dsmrreader
    healthcheck:
      # postgres is the default user, please update with
      # the DJANGO_DATABASE_USER used for dsmr-reader-docker
      # default for DSMR Reader is dsmrreader
      test: [ "CMD-SHELL", "pg_isready -U dsmrreader" ]
      interval: 10s
      timeout: 5s
      retries: 10

  dsmr:
    image: xirixiz/dsmr-reader-docker:latest
    depends_on:
      dsmrdb:
        condition: service_healthy
    container_name: dsmr
    links:
      - dsmrdb
    cap_add:
      - NET_ADMIN
    restart: always
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /share/docker/dsmr_backups:/app/backups
    networks:
      vlan86:
        ipv4_address: ##############
    mac_address: ################
    environment:
      - PUID=502
      - PGID=100
      - TZ=Europe/Amsterdam
      - DJANGO_TIME_ZONE=Europe/Amsterdam
      - VIRTUAL_HOST=###############
    ports:
      - 7777:80
      - 7779:443
    devices:
      - "/dev/ttyUSB0:/dev/ttyUSB0"
    # healthcheck:
    #   disable: true
    healthcheck:
      test:
        [
          "CMD",
          "curl",
          "-Lsf",
          "http://127.0.0.1/about",
          "-o",
          "/dev/null",
          "-w",
          "HTTP_%{http_code}"
        ]
      interval: 10s
      timeout: 5s
      retries: 10

volumes:
  dsmrdb: null
  dsmrdb_backups: null

networks:
  vlan20:
    name: qnet-static-vlan20
    external: true
  vlan30:
    name: qnet-static-vlan30
    external: true
  vlan40:
    name: qnet-static-vlan40
    external: true
  vlan86:
    name: qnet-static-vlan86
    external: true

Log output:

2023/01/30 11:21:07 [crit] 301#301: *13 connect() to unix:///tmp/gunicorn--dsmrwebinterface.socket failed (2: No such file or directory) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /about HTTP/1.1", upstream: "http://unix:///tmp/gunicorn--dsmr_webinterface.socket:/about", host: "127.0.0.1" 127.0.0.1 - - [30/Jan/2023:11:21:07 +0100] "GET /about HTTP/1.1" 502 150 "-" "curl/7.83.1" "-" Starting DSMR Reader - datalogger... Starting DSMR Reader - backend... Starting DSMR Reader - webinterface... Starting DSMR Reader - backend... Starting DSMR Reader - datalogger... Starting DSMR Reader - webinterface... Starting DSMR Reader - backend... Starting DSMR Reader - datalogger... Starting DSMR Reader - webinterface... Starting DSMR Reader - backend... Starting DSMR Reader - datalogger... Starting DSMR Reader - webinterface... Starting DSMR Reader - webinterface... Starting DSMR Reader - datalogger... Starting DSMR Reader - backend... Starting DSMR Reader - webinterface... Starting DSMR Reader - datalogger... Starting DSMR Reader - backend... Starting DSMR Reader - backend... Starting DSMR Reader - datalogger... Starting DSMR Reader - webinterface... Starting DSMR Reader - webinterface...

xirixiz commented 1 year ago

Portainer sucks imho :). This happened already a lot of time to multiple users. Please look in the closed section: https://github.com/xirixiz/dsmr-reader-docker/issues?q=is%3Aissue+is%3Aclosed+portainer

Most probably you'll find the solution there. You need to remove "things" from portainer and start again.

mupsje commented 1 year ago

okay gonna try now, but fyi, the update went trough the Watchtower. I only use Portainer to check, not for updates . I shall remove and reinstall again.

mupsje commented 1 year ago

nope, still same issue, problem is Health check. " Unhealthy"

Same error:

[crit] 301#301: *13 connect() to unix:///tmp/gunicorn--dsmr_webinterface.socket failed (2: No such file or directory) while connecting to upstream, client: 127.0.0.1, server: _, request: "GET /about HTTP/1.1", upstream: "[http://unix:///tmp/gunicorn--dsmr_webinterface.socket:/about](http://unix///tmp/gunicorn--dsmr_webinterface.socket:/about)", host: "127.0.0.1"
127.0.0.1 - - [30/Jan/2023:11:21:07 +0100] "GET /about HTTP/1.1" 502 150 "-" "curl/7.83.1" "-"
xirixiz commented 1 year ago

The problem isn't the healthcheck. The healthcheck failing is the result of a process failing to start. Basically nothing has been changed in the Docker image itself, only to DSMR Reader, so I`m for 99% sure this issue is caused (again) by Portainer. I've had these discussions already a lot of times, and it all turned out to be Portainer in the end.... so apologies if you notice my little temper poppin up. Please don't take it personal ;). Let's try to solve it together.

Did you fully remove the Docker image(s) of DSMR Reader? When pulled fresh, it should all be working again. I validated the images myself via AMD64 and ARM, both are working fine from my side. Maybe you can enable debug logging if it's not working after removing it all?

Also I have the feeling you're facing another error then @Diondk is facing. Did you see something similar to this:

File "/usr/local/lib/python3.11/site-packages/decouple.py", line 309, in call
raise ValueError((
ValueError: Value not in list: 'warning'; valid values are ['DEBUG', 'WARNING', 'ERROR']
[2023-01-30 10:49:00 +0100] [776] [INFO] Worker exiting (pid: 776)
[2023-01-30 10:49:00 +0100] [771] [INFO] Shutting down: Master
[2023-01-30 10:49:00 +0100] [771] [INFO] Reason: Worker failed to boot.
Starting DSMR Reader - webinterface...
Starting DSMR Reader - backend...
Diondk commented 1 year ago

Hi, you need to provide a little more info. Can you share your docker-compose file? @mupsje do you also have unraid? Do you both use a tool like portainer?

The error is clear at least. DSMRREADER_LOGLEVEL has been set, but incorrectly (warning instead of WARNING).

pff good thing you are reading the errors correctly because i didnt see it, i changed the DMSRREADER_LOGLEVEL from warning to WARNING and now it works.

mupsje commented 1 year ago

No problem with the temper. (I completely understand)

But if it again a "Portainer issue", what's not My case as I mentioned before, then maybe investigate that problem to solve in the future. :) I only use Portainer to debug logs, nothing more.. The update went through " Watchtower"

I use "ContainerStation" on a QNAP NAS.

Anyway, Indeed I removed and pruned all images and containers, also the complete Volumes (temp also) And did a full new setup.

Still same error.

mupsje commented 1 year ago

I tried the " arm32v7-5.9.0-2022.12.05 " TAG

Then it is working again. So the new update is not ok for armv7 then I suppose ?

mupsje commented 1 year ago

now it is working with the OLD TAG, How iyo is the best way to update then ? (I removed Watchtower label for now)

Greets and thanks for your patience

mupsje commented 1 year ago

Still having issues to update, please advice how to update to latest. when I change inside compose file yaml from the working image:

also tried to change the DUID and DGID without succes.

still getting same error, getting bit frustrated. ;(

192.168.86.101 - - [05/Feb/2023:12:06:51 +0100] "GET / HTTP/1.1" 502 150 "-" "python-requests/2.25.0" "-" 2023/02/05 12:06:51 [crit] 298#298: 11 connect() to unix:///tmp/gunicorn--dsmrwebinterface.socket failed (2: No such file or directory) while connecting to upstream, client: 192.168.86.101, server: , request: "GET / HTTP/1.1", upstream: "http://unix:///tmp/gunicorn--dsmr_webinterface.socket:/", host: "192.168.86.233" Starting DSMR Reader - backend... Starting DSMR Reader - datalogger... Starting DSMR Reader - webinterface... Starting DSMR Reader - backend... Starting DSMR Reader - datalogger... Starting DSMR Reader - webinterface... 2023/02/05 12:06:53 [crit] 298#298: 13 connect() to unix:///tmp/gunicorn--dsmrwebinterface.socket failed (2: No such file or directory) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /about HTTP/1.1", upstream: "http://unix:///tmp/gunicorn--dsmr_webinterface.socket:/about", host: "127.0.0.1" 127.0.0.1 - - [05/Feb/2023:12:06:53 +0100] "GET /about HTTP/1.1" 502 150 "-" "curl/7.83.1" "-" Starting DSMR Reader - backend... Starting DSMR Reader - datalogger... Starting DSMR Reader - webinterface... Starting DSMR Reader - backend... Starting DSMR Reader - datalogger... Starting DSMR Reader - webinterface... Starting DSMR Reader - backend... Starting DSMR Reader - datalogger...

full compose file QNAP

version: '3.0'

services:
  dsmrdb:
    # When using Postgres, release 13.x is supported only
    # due to the limited availability of client packages, especially for arm32v7
    image: postgres:14-alpine
    container_name: dsmrdb
    restart: unless-stopped
    volumes:
      - /share/docker/dsmrdb:/var/lib/postgresql/data
    networks:
      vlan86:
        ipv4_address: 192.168.86.234
    mac_address: 02:42:97:8e:cf:46
    environment:
#      - PUID=502
#      - PGID=100
      - DUID=803
      - DGID=803
      - TZ=Europe/Amsterdam
      - PG_TZ=Europe/Amsterdam
      - POSTGRES_USER=dsmrreader
      - POSTGRES_PASSWORD=dsmrreader
      - POSTGRES_DB=dsmrreader
    healthcheck:
      # postgres is the default user, please update with
      # the DJANGO_DATABASE_USER used for dsmr-reader-docker
      # default for DSMR Reader is dsmrreader
      test: [ "CMD-SHELL", "pg_isready -U dsmrreader" ]
      interval: 10s
      timeout: 5s
      retries: 10

  dsmr:
#    image: xirixiz/dsmr-reader-docker:arm32v7-5.9.0-2022.12.05
    image: xirixiz/dsmr-reader-docker:arm32v7-5.10.3-2023.02.01
#    image: xirixiz/dsmr-reader-docker:latest        
    depends_on:
      dsmrdb:
        condition: service_healthy
    container_name: dsmr
    links:
      - dsmrdb
    cap_add:
      - NET_ADMIN
    restart: always
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /share/docker/dsmr_backups:/app/backups
    networks:
      vlan86:
        ipv4_address: 192.168.86.233
    mac_address: 02:42:59:81:57:b1
    environment:
#      - PUID=502
#      - PGID=100
      - DUID=803
      - DGID=803
      - TZ=Europe/Amsterdam
      - DJANGO_TIME_ZONE=Europe/Amsterdam
#      - VIRTUAL_HOST=192.168.86.234
      - VIRTUAL_HOST=localhost
    ports:
      - 7777:80
      - 7779:443
    devices:
      - "/dev/ttyUSB0:/dev/ttyUSB0"
    #healthcheck:
    #   disable: true
    healthcheck:
      test:
        [
          "CMD",
          "curl",
          "-Lsf",
          "http://127.0.0.1/about",
          "-o",
          "/dev/null",
          "-w",
          "HTTP_%{http_code}"
        ]
      interval: 10s
      timeout: 5s
      retries: 10

volumes:
  dsmrdb: null
  dsmrdb_backups: null

networks:
  vlan20:
    name: qnet-static-vlan20
    external: true
  vlan30:
    name: qnet-static-vlan30
    external: true
  vlan40:
    name: qnet-static-vlan40
    external: true
  vlan86:
    name: qnet-static-vlan86
    external: true
mupsje commented 1 year ago

@xirixiz ?

xirixiz commented 1 year ago

Hi @mupsje, I`m afraid I cannot help you with this as it's not DSMR Reader related. It is related to portainer or anything else within your infrastructure configuration. The only thing what you could do is to keep it simple and try from there.

As I see errors related to this, it could be that this is causing the issue, though I would find it weird as it was working before and nothing changed for DSMR Reader for this.

    networks:
      vlan86:
        ipv4_address: 192.168.86.233

Verder kwam ik deze issues tegen: https://github.com/dsmrreader/dsmr-reader/issues/381 https://github.com/dsmrreader/dsmr-reader/issues/668

We kunnen uiteraard ook @dennissiemensma nog even vragen of hij een richting heeft om verder te analyzeren :).

mupsje commented 1 year ago

Ow Nederlands..... fijn ;)

Dank alvast, ik heb zelf gisteren ook compleet alles verwijderd en alles opnieuw opgezet, maar helaas dan ook hetzelfde probleem. Als ik dan de oude image configureer dan draait alles weer. Echt wierd.

Waarom is er een probleem met de Vlan ?
Ik draai zo ongeveer 20 op deze manier, ik moet wel zeggen dat een QNAP niet het makkelijkste is ;)

IK zal me eens gaan inlezen op die andere. Nogmaals, Portainer is in dit verhaal niet ter orde .

groet

dennissiemensma commented 1 year ago

We kunnen uiteraard ook @dennissiemensma nog even vragen of hij een richting heeft om verder te analyzeren :).

Hebben jullie nog meer logs? Want de ...

2023/02/05 12:06:53 [crit] 298#298: *13 connect() to unix:///tmp/gunicorn--dsmr_webinterface.socket failed 
(2: No such file or directory) while connecting to upstream, client: 127.0.0.1, server: _, request: 
"GET /about HTTP/1.1", upstream: "[http://unix:///tmp/gunicorn--dsmr_webinterface.socket:/about]
(http://unix///tmp/gunicorn--dsmr_webinterface.socket:/about)", host: "127.0.0.1"

... is denk ik een gevolg dat Gunicorn (de app server voor DSMR-reader) niet draait en dan is het de vraag waarom die niet wilt starten (of crasht).

dennissiemensma commented 1 year ago

Gunicorn zit trouwens in het "webinterface" proces, ter referentie.

xirixiz commented 1 year ago

Sorry, I have been quiet busy. I wlll respond asap!