search

xirixiz / dsmr-reader-docker

DSMR Reader in Docker.
https://hub.docker.com/r/xirixiz/dsmr-reader-docker
113 stars 33 forks source link

Problems with dsmrreader started week ago no daily stats #307

Closed hemant5400z closed 1 year ago

hemant5400z commented 1 year ago

Support guidelines

I've found an issue and checked that ...

Description

Hi,

Running version 5.8 in Docker version, starting last week, it seems that dsmr keeps crashing for no reason: Process behind schedule: Weather update (4 days, 8 hours ago) Process behind schedule: Calculate quarter hour electricity peaks (4 days, 10 hours ago) Process behind schedule: Generate day and hour statistics (4 days, 10 hours ago) Process behind schedule: Generate consumption data (5 days ago) Day statistics are lagging behind (5 days, 15 hours ago) Too many unprocessed readings: 43095 (7 months, 1 week ago)

Above I can see in the notifications.

In the portal/webinterface I'm sometimes redirected to:

Server Error

Sorry, something unexpected happened. Exception:

OperationalError: server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. Traceback:

File "/usr/local/lib/python3.10/site-packages/django/core/handlers/base.py", line 181, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/usr/local/lib/python3.10/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func response = view_func(request, *args, **kwargs)

File "/usr/local/lib/python3.10/site-packages/django/contrib/admin/sites.py", line 414, in login return LoginView.as_view(**defaults)(request)

File "/usr/local/lib/python3.10/site-packages/django/views/generic/base.py", line 70, in view return self.dispatch(request, *args, **kwargs)

File "/usr/local/lib/python3.10/site-packages/django/utils/decorators.py", line 43, in _wrapper return bound_method(*args, **kwargs)

File "/usr/local/lib/python3.10/site-packages/django/views/decorators/debug.py", line 89, in sensitive_post_parameters_wrapper return view(request, *args, **kwargs)

File "/usr/local/lib/python3.10/site-packages/django/utils/decorators.py", line 43, in _wrapper return bound_method(*args, **kwargs)

File "/usr/local/lib/python3.10/site-packages/django/utils/decorators.py", line 130, in _wrapped_view response = view_func(request, *args, **kwargs)

File "/usr/local/lib/python3.10/site-packages/django/utils/decorators.py", line 43, in _wrapper return bound_method(*args, **kwargs)

File "/usr/local/lib/python3.10/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func response = view_func(request, *args, **kwargs)

File "/usr/local/lib/python3.10/site-packages/django/contrib/auth/views.py", line 63, in dispatch return super().dispatch(request, *args, **kwargs)

File "/usr/local/lib/python3.10/site-packages/django/views/generic/base.py", line 98, in dispatch return handler(request, *args, **kwargs)

File "/usr/local/lib/python3.10/site-packages/django/views/generic/edit.py", line 141, in post if form.is_valid():

File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 175, in is_valid return self.is_bound and not self.errors

File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 170, in errors self.full_clean()

File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 373, in full_clean self._clean_form()

File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 400, in _clean_form cleaned_data = self.clean()

File "/usr/local/lib/python3.10/site-packages/django/contrib/auth/forms.py", line 202, in clean self.user_cache = authenticate(self.request, username=username, password=password)

File "/usr/local/lib/python3.10/site-packages/django/views/decorators/debug.py", line 42, in sensitive_variables_wrapper return func(*func_args, **func_kwargs)

File "/usr/local/lib/python3.10/site-packages/django/contrib/auth/init.py", line 76, in authenticate user = backend.authenticate(request, **credentials)

File "/usr/local/lib/python3.10/site-packages/django/contrib/auth/backends.py", line 42, in authenticate user = UserModel._default_manager.get_by_natural_key(username)

File "/usr/local/lib/python3.10/site-packages/django/contrib/auth/base_user.py", line 45, in get_by_natural_key return self.get(**{self.model.USERNAME_FIELD: username})

File "/usr/local/lib/python3.10/site-packages/django/db/models/manager.py", line 85, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs)

File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 431, in get num = len(clone)

File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 262, in len self._fetch_all()

File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 1324, in _fetch_all self._result_cache = list(self._iterable_class(self))

File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 51, in iter results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)

File "/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql cursor.execute(sql, params)

File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 66, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)

File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers return executor(sql, params, many, context)

File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 79, in _execute with self.db.wrap_database_errors:

File "/usr/local/lib/python3.10/site-packages/django/db/utils.py", line 90, in exit raise dj_exc_value.with_traceback(traceback) from exc_value

File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params)

Not sure what is happening and why this appress all of a sudden.

Expected behaviour

Expect to have daily sats and no errors

Actual behaviour

Continues error and restarts, the docker itself stays active/online but seems something internally in the dsmr container that restarts.

Steps to reproduce

Normal run of docker

Docker info

Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b1dc45ec561bd867c4805eee786caab7cc83acae
 runc version: 89783e1862a2cc04647ab15b6e88a0af3d66fac3
 init version: 12b6a20 (expected: de40ad0)
 Security Options:
  apparmor
 Kernel Version: 3.10.105
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 3.867GiB
 Name: HAGPAZ1-S-05
 ID: WYNU:4ZHT:AUDS:DN5A:G7N6:NVZS:TJCN:4NUF:GKWI:MU73:4AYY:THNS
 Docker Root Dir: /volume1/@docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:

Version

Docker compose

No yaml info

Container logs

Server Error

Sorry, something unexpected happened. Exception:

OperationalError: server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. Traceback:

File "/usr/local/lib/python3.10/site-packages/django/core/handlers/base.py", line 181, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/usr/local/lib/python3.10/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func response = view_func(request, *args, **kwargs)

File "/usr/local/lib/python3.10/site-packages/django/contrib/admin/sites.py", line 414, in login return LoginView.as_view(**defaults)(request)

File "/usr/local/lib/python3.10/site-packages/django/views/generic/base.py", line 70, in view return self.dispatch(request, *args, **kwargs)

File "/usr/local/lib/python3.10/site-packages/django/utils/decorators.py", line 43, in _wrapper return bound_method(*args, **kwargs)

File "/usr/local/lib/python3.10/site-packages/django/views/decorators/debug.py", line 89, in sensitive_post_parameters_wrapper return view(request, *args, **kwargs)

File "/usr/local/lib/python3.10/site-packages/django/utils/decorators.py", line 43, in _wrapper return bound_method(*args, **kwargs)

File "/usr/local/lib/python3.10/site-packages/django/utils/decorators.py", line 130, in _wrapped_view response = view_func(request, *args, **kwargs)

File "/usr/local/lib/python3.10/site-packages/django/utils/decorators.py", line 43, in _wrapper return bound_method(*args, **kwargs)

File "/usr/local/lib/python3.10/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func response = view_func(request, *args, **kwargs)

File "/usr/local/lib/python3.10/site-packages/django/contrib/auth/views.py", line 63, in dispatch return super().dispatch(request, *args, **kwargs)

File "/usr/local/lib/python3.10/site-packages/django/views/generic/base.py", line 98, in dispatch return handler(request, *args, **kwargs)

File "/usr/local/lib/python3.10/site-packages/django/views/generic/edit.py", line 141, in post if form.is_valid():

File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 175, in is_valid return self.is_bound and not self.errors

File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 170, in errors self.full_clean()

File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 373, in full_clean self._clean_form()

File "/usr/local/lib/python3.10/site-packages/django/forms/forms.py", line 400, in _clean_form cleaned_data = self.clean()

File "/usr/local/lib/python3.10/site-packages/django/contrib/auth/forms.py", line 202, in clean self.user_cache = authenticate(self.request, username=username, password=password)

File "/usr/local/lib/python3.10/site-packages/django/views/decorators/debug.py", line 42, in sensitive_variables_wrapper return func(*func_args, **func_kwargs)

File "/usr/local/lib/python3.10/site-packages/django/contrib/auth/init.py", line 76, in authenticate user = backend.authenticate(request, **credentials)

File "/usr/local/lib/python3.10/site-packages/django/contrib/auth/backends.py", line 42, in authenticate user = UserModel._default_manager.get_by_natural_key(username)

File "/usr/local/lib/python3.10/site-packages/django/contrib/auth/base_user.py", line 45, in get_by_natural_key return self.get(**{self.model.USERNAME_FIELD: username})

File "/usr/local/lib/python3.10/site-packages/django/db/models/manager.py", line 85, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs)

File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 431, in get num = len(clone)

File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 262, in len self._fetch_all()

File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 1324, in _fetch_all self._result_cache = list(self._iterable_class(self))

File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 51, in iter results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)

File "/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql cursor.execute(sql, params)

File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 66, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)

File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers return executor(sql, params, many, context)

File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 79, in _execute with self.db.wrap_database_errors:

File "/usr/local/lib/python3.10/site-packages/django/db/utils.py", line 90, in exit raise dj_exc_value.with_traceback(traceback) from exc_value

File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params)

Additional info

No response

xirixiz commented 1 year ago

Hi, I see you're running very old Docker components on your Synology. Are you running on an old version of DSM perhaps?

hemant5400z commented 1 year ago

Hi yes, running on my synology for years without problems it started 4th december not sure why en how to solve.

hemant5400z commented 1 year ago

Tried to vacumm but it is not completing I think the sisue started when I did run this on the 4th:

vacuumdb: vacuuming database "dsmrreader" WARNING: terminating connection because of crash of another server process DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. HINT: In a moment you should be able to reconnect to the database and repeat your command.

Hemant

hemant5400z commented 1 year ago

I can see in the Database Log:

2022-12-12 10:51:42 | stdout | 2022-12-12 11:51:42.025 CET [1] LOG:  database system is ready to accept connections -- | -- | -- 2022-12-12 10:51:41 | stdout | 2022-12-12 11:51:41.832 CET [1199] LOG:  redo done at 31/4754C8D0 2022-12-12 10:51:41 | stdout | 2022-12-12 11:51:41.824 CET [1199] LOG:  redo starts at 31/47546730 2022-12-12 10:51:41 | stdout | 2022-12-12 11:51:41.814 CET [1199] LOG:  database system was not properly shut down; automatic recovery in progress 2022-12-12 10:51:41 | stdout | 2022-12-12 11:51:41.731 CET [1199] LOG:  database system was interrupted; last known up at 2022-12-12 11:50:34 CET 2022-12-12 10:51:41 | stdout | 2022-12-12 11:51:41.067 CET [1] LOG:  all server processes terminated; reinitializing 2022-12-12 10:51:41 | stdout | 2022-12-12 11:51:41.063 CET [1192] HINT:  In a moment you should be able to reconnect to the database and repeat your command. 2022-12-12 10:51:41 | stdout | 2022-12-12 11:51:41.063 CET [1192] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2022-12-12 10:51:41 | stdout | 2022-12-12 11:51:41.063 CET [1192] WARNING:  terminating connection because of crash of another server process 2022-12-12 10:51:41 | stdout | 2022-12-12 11:51:41.055 CET [1196] HINT:  In a moment you should be able to reconnect to the database and repeat your command. 2022-12-12 10:51:41 | stdout | 2022-12-12 11:51:41.055 CET [1196] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2022-12-12 10:51:41 | stdout | 2022-12-12 11:51:41.055 CET [1196] WARNING:  terminating connection because of crash of another server process 2022-12-12 10:51:41 | stdout | 2022-12-12 11:51:41.054 CET [1] LOG:  terminating any other active server processes 2022-12-12 10:51:41 | stdout | 2022-12-12 11:51:41.054 CET [1] DETAIL:  Failed process was running: UPDATE "dsmr_datalogger_dsmrreading" SET "processed" = true WHERE "dsmr_datalogger_dsmrreading"."id" = 11657273 2022-12-12 10:51:41 | stdout | 2022-12-12 11:51:41.054 CET [1] LOG:  server process (PID 1198) was terminated by signal 11: Segmentation fault 2022-12-12 10:50:34 | stdout | 2022-12-12 11:50:34.269 CET [1] LOG:  database system is ready to accept connections 2022-12-12 10:50:34 | stdout | 2022-12-12 11:50:34.202 CET [1188] LOG:  redo done at 31/47546678 2022-12-12 10:50:34 | stdout | 2022-12-12 11:50:34.202 CET [1188] LOG:  invalid record length at 31/475466B8: wanted 24, got 0 2022-12-12 10:50:34 | stdout | 2022-12-12 11:50:34.195 CET [1188] LOG:  redo starts at 31/47543C80 2022-12-12 10:50:34 | stdout | 2022-12-12 11:50:34.183 CET [1188] LOG:  database system was not properly shut down; automatic recovery in progress 2022-12-12 10:50:34 | stdout | 2022-12-12 11:50:34.070 CET [1188] LOG:  database system was interrupted; last known up at 2022-12-12 11:49:27 CET 2022-12-12 10:50:33 | stdout | 2022-12-12 11:50:33.991 CET [1] LOG:  all server processes terminated; reinitializing
xirixiz commented 1 year ago

Is jouw nas toevallig uitgevallen? Ik heb zoiets eerder gezien. Ik geloof na een stroomstoring bij iemand. Ik geloof dat de datum/tijd toen ergens niet meer goed stond wat allerlhande klachten veroorzaakte.

hemant5400z commented 1 year ago

De NAS is niet uitgevallen, heb datum nog nagekeken van NAS en Dockers die lopen gelijk met mijn NTP server.

Ik heb een backup teruggezet van 2 December 2022. Die werkte weer tot het moment dat ik de Vacuum heb uitgevoerd:

docker exec -ti dsmr bash -c '/app/cleandb.sh' vacuumdb: vacuuming database "dsmrreader" WARNING: terminating connection because of crash of another server process DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. HINT: In a moment you should be able to reconnect to the database and repeat your command.

2022-12-12 13:29:49 stdout 2022-12-12 14:29:49.278 CET [138] LOG: redo done at 30/8CF39F28 2022-12-12 13:29:49 stdout 2022-12-12 14:29:49.019 CET [159] FATAL: the database system is in recovery mode 2022-12-12 13:29:45 stdout 2022-12-12 14:29:45.975 CET [158] FATAL: the database system is in recovery mode 2022-12-12 13:29:40 stdout 2022-12-12 14:29:40.506 CET [157] FATAL: the database system is in recovery mode 2022-12-12 13:29:33 stdout 2022-12-12 14:29:30.632 CET [156] FATAL: the database system is in recovery mode 2022-12-12 13:29:24 stdout 2022-12-12 14:29:24.526 CET [155] FATAL: the database system is in recovery mode 2022-12-12 13:29:15 stdout 2022-12-12 14:29:15.583 CET [154] FATAL: the database system is in recovery mode 2022-12-12 13:29:07 stdout 2022-12-12 14:29:07.717 CET [153] FATAL: the database system is in recovery mode 2022-12-12 13:29:02 stdout 2022-12-12 14:29:02.983 CET [152] FATAL: the database system is in recovery mode 2022-12-12 13:29:02 stdout 2022-12-12 14:29:02.944 CET [151] FATAL: the database system is in recovery mode 2022-12-12 13:29:02 stdout 2022-12-12 14:29:02.909 CET [150] FATAL: the database system is in recovery mode 2022-12-12 13:29:02 stdout 2022-12-12 14:29:02.871 CET [149] FATAL: the database system is in recovery mode 2022-12-12 13:29:02 stdout 2022-12-12 14:29:02.795 CET [148] FATAL: the database system is in recovery mode 2022-12-12 13:29:02 stdout 2022-12-12 14:29:02.720 CET [147] FATAL: the database system is in recovery mode 2022-12-12 13:29:02 stdout 2022-12-12 14:29:02.637 CET [146] FATAL: the database system is in recovery mode 2022-12-12 13:29:02 stdout 2022-12-12 14:29:02.568 CET [145] FATAL: the database system is in recovery mode 2022-12-12 13:29:02 stdout 2022-12-12 14:29:02.489 CET [144] FATAL: the database system is in recovery mode 2022-12-12 13:29:02 stdout 2022-12-12 14:29:02.402 CET [143] FATAL: the database system is in recovery mode 2022-12-12 13:29:02 stdout 2022-12-12 14:29:02.256 CET [142] FATAL: the database system is in recovery mode 2022-12-12 13:29:02 stdout 2022-12-12 14:29:02.127 CET [141] FATAL: the database system is in recovery mode 2022-12-12 13:29:02 stdout 2022-12-12 14:29:02.015 CET [138] LOG: redo starts at 30/6D1C86A8

Dan blijft de database weer vreemd doen.

hemant5400z commented 1 year ago

De log blijft dat herhalen met:

2022-12-12 13:40:29 | stdout | 2022-12-12 14:40:29.461 CET [1] LOG:  database system is ready to accept connections -- | -- | -- 2022-12-12 13:40:29 | stdout | 2022-12-12 14:40:29.441 CET [877] FATAL:  the database system is in recovery mode 2022-12-12 13:40:29 | stdout | 2022-12-12 14:40:29.353 CET [875] LOG:  redo done at 30/8D081B10 2022-12-12 13:40:29 | stdout | 2022-12-12 14:40:29.352 CET [875] LOG:  invalid record length at 30/8D081B38: wanted 24, got 0 2022-12-12 13:40:29 | stdout | 2022-12-12 14:40:29.349 CET [875] LOG:  redo starts at 30/8D076778 2022-12-12 13:40:29 | stdout | 2022-12-12 14:40:29.332 CET [875] LOG:  database system was not properly shut down; automatic recovery in progress 2022-12-12 13:40:29 | stdout | 2022-12-12 14:40:29.322 CET [876] FATAL:  the database system is in recovery mode 2022-12-12 13:40:29 | stdout | 2022-12-12 14:40:29.191 CET [875] LOG:  database system was interrupted; last known up at 2022-12-12 14:40:19 CET 2022-12-12 13:40:29 | stdout | 2022-12-12 14:40:29.084 CET [1] LOG:  all server processes terminated; reinitializing 2022-12-12 13:40:29 | stdout | 2022-12-12 14:40:29.064 CET [872] HINT:  In a moment you should be able to reconnect to the database and repeat your command. 2022-12-12 13:40:29 | stdout | 2022-12-12 14:40:29.064 CET [872] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2022-12-12 13:40:29 | stdout | 2022-12-12 14:40:29.064 CET [872] WARNING:  terminating connection because of crash of another server process 2022-12-12 13:40:29 | stdout | 2022-12-12 14:40:29.060 CET [874] HINT:  In a moment you should be able to reconnect to the database and repeat your command. 2022-12-12 13:40:29 | stdout | 2022-12-12 14:40:29.060 CET [874] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2022-12-12 13:40:29 | stdout | 2022-12-12 14:40:29.060 CET [874] WARNING:  terminating connection because of crash of another server process 2022-12-12 13:40:29 | stdout | 2022-12-12 14:40:29.059 CET [869] HINT:  In a moment you should be able to reconnect to the database and repeat your command. 2022-12-12 13:40:29 | stdout | 2022-12-12 14:40:29.059 CET [869] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2022-12-12 13:40:29 | stdout | 2022-12-12 14:40:29.059 CET [869] WARNING:  terminating connection because of crash of another server process 2022-12-12 13:40:29 | stdout | 2022-12-12 14:40:29.052 CET [1] LOG:  terminating any other active server processes 2022-12-12 13:40:29 | stdout | 2022-12-12 14:40:29.050 CET [1] DETAIL:  Failed process was running: UPDATE "dsmr_datalogger_dsmrreading" SET "processed" = true WHERE "dsmr_datalogger_dsmrreading"."id" = 11657273 2022-12-12 13:40:29 | stdout | 2022-12-12 14:40:29.050 CET [1] LOG:  server process (PID 873) was terminated by signal 11: Segmentation fault 2022-12-12 13:40:19 | stdout | 2022-12-12 14:40:19.938 CET [1] LOG:  database system is ready to accept connections 2022-12-12 13:40:19 | stdout | 2022-12-12 14:40:19.857 CET [865] LOG:  redo done at 30/8D0766D8 2022-12-12 13:40:19 | stdout | 2022-12-12 14:40:19.850 CET [865] LOG:  redo starts at 30/8D075608 2022-12-12 13:40:19 | stdout | 2022-12-12 14:40:19.834 CET [865] LOG:  database system was not properly shut down; automatic recovery in progress 2022-12-12 13:40:19 | stdout | 2022-12-12 14:40:19.731 CET [865] LOG:  database system was interrupted; last known up at 2022-12-12 14:40:11 CET 2022-12-12 13:40:19 | stdout | 2022-12-12 14:40:19.663 CET [1] LOG:  all server processes terminated; reinitializing 2022-12-12 13:40:19 | stdout | 2022-12-12 14:40:19.658 CET [860] HINT:  In a moment you should be able to reconnect to the database and repeat your command. 2022-12-12 13:40:19 | stdout | 2022-12-12 14:40:19.658 CET [860] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2022-12-12 13:40:19 | stdout | 2022-12-12 14:40:19.658 CET [860] WARNING:  terminating connection because of crash of another server process 2022-12-12 13:40:19 | stdout | 2022-12-12 14:40:19.656 CET [863] HINT:  In a moment you should be able to reconnect to the database and repeat your command. 2022-12-12 13:40:19 | stdout | 2022-12-12 14:40:19.656 CET [863] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2022-12-12 13:40:19 | stdout | 2022-12-12 14:40:19.656 CET [863] WARNING:  terminating connection because of crash of another server process
xirixiz commented 1 year ago

In dat geval zou ik vacuum uitzetten/disablen. Daar is namelijk een optie voor (zie readme). Het is natuurlijk geen oplossing, maar het lijkt iets omgeving specifiek te zijn. Iets met tijden of iets anders dat niet helemaal lekker gaat. Je kan het uiteraard verder onderzoeken, maar dit staat los van dsmr reader in Docker uiteraard :).

Ik denk dat het beste is om de db te restoren met een andere naam zodat je op die omgeving kan testen met vaccuum.

Een tweede optie zou zijn om Docker bij te werken, DSMR Reader en vervolgens ook de Postgres container naar een meer recentere versie bij te werken, en het dan nog eens te proberen.

hemant5400z commented 1 year ago

Ik Heb de retention gedisabled, maar probleem blijft.

Ik heb hetzelfde gedaan met de laatste versie van dsmrreader 5.8, geet exact hetzelfde probleem. Het lijft alsof ie na de Vacuum blijft hangen in een loop.

Hemant

hemant5400z commented 1 year ago

Na de Vacuum, is de gaat de logger over zijn nek, ik zie in de log de volgende melding:

DETAIL: Failed process was running: UPDATE "dsmr_datalogger_dsmrreading" SET "processed" = true WHERE "dsmr_datalogger_dsmrreading"."id" = 11657273

xirixiz commented 1 year ago

Laten we de expert erbij halen :). @dennissiemensma heb je zoiets dergelijks vaker gezien? Ik kon het niet echt vinden tussen de gesloten issues, maar wie weet!

dennissiemensma commented 1 year ago

Alle foutmeldingen wijzen erop dat iets met de database niet lekker is. Welke versie van PostgreSQL gebruik je?

Als je een goede DB-backup hebt, zou ik je aanraden deze tip van Bram te proberen:

Een tweede optie zou zijn om Docker bij te werken, DSMR Reader en vervolgens ook de Postgres container naar een meer recentere versie bij te werken, en het dan nog eens te proberen.

hemant5400z commented 1 year ago

Hoi,

Postgres versie is 12.5. Ik heb in de Database probleem record opgezocht en vervolgens verwijder, daarna zijn alle berichten en statestieken verwerkt. Daarmee lijkt het probleem opgelost.

" Everything seems to run smoothly. Any issues, such as missing data or stale processes, will be reported here."

Snap alleen niet waarom dat geberut na aan Vacuum.

hemant

xirixiz commented 1 year ago

Ik vermoed dat er al corruptie in de database aanwezig was door de crash, met als resultaat dat het vacuum proces niet naar behoren draait. Hoe dan ook fijn dat het issue is verholpen. Ik zou overings nog steeds even kijken om DSM te updaten, alsmede de containers die je draait. DSM met name, daar zitten zo nu en dan nog wel wat vulnarabiliteis in...

hemant5400z commented 1 year ago

Ik heb 2 versies draaien van dsmrreader, oudere en laatste versie.

De reden heirvoor is Influx koppeling, aangezien je met versie 5.x naar Influx 2.0 moet en dus naar buckets (waar ik geen fan van ben). En geen zin heb om alle queries te herschijven va de verschillende dashboard heb ik de 4.19 versie aangehouden.

Zou mooi zijn als je in dsmrreader zelf de keuze kon maken of je 1.8 of 2.x wilt gebruiken en dus over wil naar buckets.

Hemant

xirixiz commented 1 year ago

Daar ben ik het mee eens voor wat betreft de wijzigingen tussen influxdb 1.x en 2.x. Het feit is wel dat het oud en unmaintained is, dus in dat geval is het ook logisch dat het niet eer supported is in DSMR Reader (python). Vernieuwing is niet altijd beter, maar stilstand is in dit geval achteruitgang. Hoe langer je wacht, des te lastiger het wordt om alles weer bij te werken naar supported versies.

My two cents 😄✌️

dennissiemensma commented 1 year ago

De reden heirvoor is Influx koppeling, aangezien je met versie 5.x naar Influx 2.0 moet en dus naar buckets (waar ik geen fan van ben). En geen zin heb om alle queries te herschijven va de verschillende dashboard heb ik de 4.19 versie aangehouden.

Zou mooi zijn als je in dsmrreader zelf de keuze kon maken of je 1.8 of 2.x wilt gebruiken en dus over wil naar buckets.

De keuze voor InfluxDB 2 is geen persoonlijke voorkeur, maar simpelweg noodzaak. Zoals Bram al aangeeft is stilstand achteruitgang en software gaat eenmaal door.

Ik zit zelf niet heel diep in InfluxDB, maar deze kwam ik tegen bij zoeken naar 1.x support (mei 2021):

InfluxDB 1.8 will continue to be maintained and receive defect fixes through the end of 2021. But InfluxDB 1.8 (and subsequent maintenance releases) will be the last official release on the 1.x line that will be built and distributed by InfluxData.

Dus blijkbaar is 1.x inmiddels al bijna een jaar end-of-life en ondersteunt de zelfs maker het niet meer. Ze patchen dus ook geen beveiligingslekken meer. Je bent uiteraard absoluut vrij om zelf anders te kiezen, voor je eigen redenen of voorkeuren. Echter als je komende tijd InfluxDB niet upgrade, zul je dat op den duur alsnog moeten doen.

Mijn advies is om dat niet oneindig uit te stellen. Niet perse vanwege DSMR-reader, maar omdat je uiteindelijk steeds meer software tegen gaat komen die incompatible raakt.

dennissiemensma commented 1 year ago

Voor DSMR-reader is het dan ook simpel: Ik hanteer tenminste mee met de end-of-life policies van de software die ik gebruik: