search

dsmrreader / dsmr-reader

DSMR-telegram reader & data visualizer for hobbyists. Free for non-commercial use.
https://dsmr-reader.readthedocs.io
Other
458 stars 95 forks source link

🙋Data (gas/elektra) geïmporteerd vanuit domotics wel in database, maar alleen gas is zichtbaar in archief e.d. #1654

Closed fjbosscha closed 1 year ago

fjbosscha commented 1 year ago

Description

Sinds afgelopen week ben ik overgestapt op DSMR-reader. Omdat ik een aantal jaren historie in domoticz heb te zitten, heb ik deze geïmporteerd binnen DSMR-reader. Ik zie dat data wel in de betreffende tabellen is gekomen (dsmr_consumption_electricityconsumption en dsmr_consumption_gasconsumption) maar via de archief optie, zie ik alleen maar gas-waarden.

Ik heb al in de andere issues gekeken, maar zie niet direct wat waar mis gaat.

DSMR-reader version

5.3

DSMR-reader platform

No response

Debug info dump

DSMR-READER
    App / Python / Database                                                        v5.3 / v3.9.2 / postgresql
    BE sleep / DL sleep / Retention / Override                                     1.0s / 1.0s / 672h / False
    Latest telegram version read / Parser settings                                                 "50" / "4"

DATA
    Telegrams total (est.)                                                                             158839
    Consumption records electricity / gas (est.)                                                103613 / 3230
dennissiemensma commented 1 year ago

Je zou #1630 kunnen bekijken. Daar staat een soortgelijk issue.

https://github.com/dsmrreader/dsmr-reader/issues/1630#issuecomment-1129653748

fjbosscha commented 1 year ago

Bedankt voor je reactie. Die had ik inderdaad ook al gezien, maar mijn data is, in mijn optiek, correct.

dsmrreader=> select read_at from dsmr_consumption_electricityconsumption;
        read_at         
------------------------
 2019-02-08 00:00:00+01
 2019-02-08 23:59:00+01
 2019-02-09 00:00:00+01
 2019-02-11 23:59:00+01
 2019-02-12 00:00:00+01
 2019-02-14 23:59:00+01
 2019-02-15 00:00:00+01
 2019-02-15 23:59:00+01
 2019-02-16 00:00:00+01
 2019-02-16 23:59:00+01
 2019-02-17 00:00:00+01
 2019-02-17 23:59:00+01
 2019-02-18 00:00:00+01
 2019-02-18 23:59:00+01
 2019-02-19 00:00:00+01
 2019-02-19 23:59:00+01
 2019-02-20 00:00:00+01
 2019-02-20 23:59:00+01
 2019-02-21 00:00:00+01
 2019-02-21 23:59:00+01
 2019-02-22 00:00:00+01
 2019-02-22 23:59:00+01
 2019-02-23 00:00:00+01
 2019-02-23 23:59:00+01
 2019-02-24 00:00:00+01
 2019-02-24 23:59:00+01
 2019-02-25 00:00:00+01
 2019-02-25 23:59:00+01
 2019-02-26 00:00:00+01
 2019-02-26 23:59:00+01
 2019-02-27 00:00:00+01
 2019-02-27 23:59:00+01
 2019-02-28 00:00:00+01
dsmrreader=> select read_at from dsmr_consumption_gasconsumption;
        read_at         
------------------------
 2022-06-28 21:45:08+02
 2022-06-28 21:50:01+02
 2020-01-10 23:59:00+01
 2020-01-11 00:00:00+01
 2020-01-11 23:59:00+01
 2020-01-12 00:00:00+01
 2020-01-12 23:59:00+01
 2020-01-13 00:00:00+01
 2020-01-13 23:59:00+01
 2020-01-14 00:00:00+01
 2020-01-14 23:59:00+01
 2020-01-15 00:00:00+01
 2020-01-15 23:59:00+01
 2020-01-16 00:00:00+01
 2020-01-16 23:59:00+01
 2020-01-17 00:00:00+01
 2020-01-17 23:59:00+01
 2020-01-18 00:00:00+01
 2020-01-18 23:59:00+01
 2020-01-19 00:00:00+01
 2020-01-19 23:59:00+01
 2020-01-20 00:00:00+01
 2020-01-20 23:59:00+01
 2020-01-21 00:00:00+01
 2020-01-21 23:59:00+01
 2020-01-22 00:00:00+01
 2020-01-22 23:59:00+01
 2020-01-23 00:00:00+01
 2020-01-23 23:59:00+01

Heb truncate dsmr_stats_daystatistics gedraaid en de processen weer aangezet.

Debug aangezet en zie nu in de dsmr_backend.log allemaal de volgende meldingen:

2022-07-04 19:26:23,478 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:26:23,497 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:26:24.490464+02:00 (ETA 0:00:00.999936)
2022-07-04 19:26:23,509 DEBUG    persistent_clients run                              31 | CLIENTS: Running 1 active client(s)
2022-07-04 19:26:23,513 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:26:24,521 DEBUG    schedule     execute_scheduled_processes      32 | SP: 2 backend service(s) ready to run
2022-07-04 19:26:24,521 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "PVOutput export" (dsmr_pvoutput.services.run)
2022-07-04 19:26:24,637 WARNING  services     get_export_data                 110 | PVOutput: Data found, but not in sync. Last data timestamp is before expected (2022-07-04 14:47:48+02:00 < 2022-07-04 14:50:00+02:00)
2022-07-04 19:26:24,637 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:26:24,651 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:26:25.647331+02:00 (ETA 0:00:00.999938)
2022-07-04 19:26:24,660 DEBUG    persistent_clients run                              31 | CLIENTS: Running 1 active client(s)
2022-07-04 19:26:24,664 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:26:25,670 DEBUG    schedule     execute_scheduled_processes      32 | SP: 3 backend service(s) ready to run
2022-07-04 19:26:25,671 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "PVOutput export" (dsmr_pvoutput.services.run)
2022-07-04 19:26:25,760 WARNING  services     get_export_data                 110 | PVOutput: Data found, but not in sync. Last data timestamp is before expected (2022-07-04 14:47:48+02:00 < 2022-07-04 14:50:00+02:00)
2022-07-04 19:26:25,761 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:26:25,778 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:26:26.772076+02:00 (ETA 0:00:00.999945)
2022-07-04 19:26:25,778 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Calculate quarter hour electricity peaks" (dsmr_consumption.services.run_quarter_hour_peaks)
2022-07-04 19:26:25,783 DEBUG    services     run_quarter_hour_peaks           52 | Quarter hour peaks: Using 2022-07-04 19:26:00+02:00 as fuzzy start
2022-07-04 19:26:25,787 DEBUG    services     run_quarter_hour_peaks           70 | Quarter hour peaks: Ready but awaiting any new readings after 2022-07-04 19:15:00+02:00, postponing for a bit...
2022-07-04 19:26:25,793 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Calculate quarter hour electricity peaks" to 2022-07-04 19:26:30+02:00 (ETA 0:00:04.212378)
2022-07-04 19:26:25,803 DEBUG    persistent_clients run                              31 | CLIENTS: Running 1 active client(s)
2022-07-04 19:26:25,809 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:26:26,816 DEBUG    schedule     execute_scheduled_processes      32 | SP: 2 backend service(s) ready to run
2022-07-04 19:26:26,817 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "PVOutput export" (dsmr_pvoutput.services.run)
2022-07-04 19:26:26,916 WARNING  services     get_export_data                 110 | PVOutput: Data found, but not in sync. Last data timestamp is before expected (2022-07-04 14:47:48+02:00 < 2022-07-04 14:50:00+02:00)

Weet niet of dit moet kloppen!

fjbosscha commented 1 year ago

Zat wat vervuiling in (had de database op een test-systeem gezet, waarbij de mqtt en pvoutput ook liepen.

logging is nu:

2022-07-04 19:32:52,267 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:32:52,292 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:32:53.283032+02:00 (ETA 0:00:00.999914)
2022-07-04 19:32:52,316 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:32:53,325 DEBUG    schedule     execute_scheduled_processes      32 | SP: 1 backend service(s) ready to run
2022-07-04 19:32:53,325 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:32:53,349 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:32:54.341110+02:00 (ETA 0:00:00.999914)
2022-07-04 19:32:53,368 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:32:54,376 DEBUG    schedule     execute_scheduled_processes      32 | SP: 1 backend service(s) ready to run
2022-07-04 19:32:54,377 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:32:54,400 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:32:55.392694+02:00 (ETA 0:00:00.999914)
2022-07-04 19:32:54,421 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:32:55,430 DEBUG    schedule     execute_scheduled_processes      32 | SP: 2 backend service(s) ready to run
2022-07-04 19:32:55,430 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:32:55,454 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:32:56.446108+02:00 (ETA 0:00:00.999913)
2022-07-04 19:32:55,454 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Calculate quarter hour electricity peaks" (dsmr_consumption.services.run_quarter_hour_peaks)
2022-07-04 19:32:55,461 DEBUG    services     run_quarter_hour_peaks           52 | Quarter hour peaks: Using 2022-07-04 19:32:00+02:00 as fuzzy start
2022-07-04 19:32:55,468 DEBUG    services     run_quarter_hour_peaks           70 | Quarter hour peaks: Ready but awaiting any new readings after 2022-07-04 19:30:00+02:00, postponing for a bit...
2022-07-04 19:32:55,478 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Calculate quarter hour electricity peaks" to 2022-07-04 19:33:00+02:00 (ETA 0:00:04.530197)
2022-07-04 19:32:55,499 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:32:56,508 DEBUG    schedule     execute_scheduled_processes      32 | SP: 1 backend service(s) ready to run
2022-07-04 19:32:56,510 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:32:56,534 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:32:57.526560+02:00 (ETA 0:00:00.999912)
2022-07-04 19:32:56,556 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:32:57,565 DEBUG    schedule     execute_scheduled_processes      32 | SP: 1 backend service(s) ready to run
2022-07-04 19:32:57,567 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:32:57,592 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:32:58.583766+02:00 (ETA 0:00:00.999911)
2022-07-04 19:32:57,609 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:32:58,618 DEBUG    schedule     execute_scheduled_processes      32 | SP: 1 backend service(s) ready to run
2022-07-04 19:32:58,620 DEBUG    schedule     execute_scheduled_processes      35 | SP: Running "Generate consumption data" (dsmr_consumption.services.run)
2022-07-04 19:32:58,647 DEBUG    schedule     reschedule                       80 | SP: Rescheduled "Generate consumption data" to 2022-07-04 19:32:59.638447+02:00 (ETA 0:00:00.999905)
2022-07-04 19:32:58,673 DEBUG    mixins       run_loop                         75 | dsmr_backend.management.commands.dsmr_backend: Sleeping 1.0s
2022-07-04 19:32:59,682 DEBUG    schedule     execute_scheduled_processes      32 | SP: 1 backend service(s) ready to run
dennissiemensma commented 1 year ago

Hoe draai je DSMR-reader? Docker of native?

dennissiemensma commented 1 year ago

Staat er wel data in de dsmr_stats_daystatistics tabel?

fjbosscha commented 1 year ago

Ik draai deze native. Er staat geen data in dsmr_stats_daystatistics

dsmrreader=> select  * from dsmr_stats_daystatistics;                                                                                                                  id | day | total_cost | electricity1 | electricity2 | electricity1_returned | electricity2_returned | electricity1_cost | electricity2_cost | gas | gas_cost | average_temperature | highest_temperature | lowest_temperature | fixed_cost | electricity1_reading | electricity1_returned_reading | electricity2_reading | electricity2_returned_reading | gas_reading 
----+-----+------------+--------------+--------------+-----------------------+-----------------------+-------------------+-------------------+-----+----------+---------------------+---------------------+--------------------+------------+----------------------+-------------------------------+----------------------+-------------------------------+-------------
(0 rows)
fjbosscha commented 1 year ago

ik heb nog even in de "running" tabellen gekeken en daar staan in dsmr_stats_daystatistics wel waarden bij Electricity1_reading, electricity1_returned_reading, electricity2_reading, electricity2_returned_reading en gas. Wat mij hierbij opvalt is dat hier geen waarden zijn bij electricity1, electricity2, electricity1_returned en electricity2_returned. Bij gas staan er wel de dagelijkse waarden.

fjbosscha commented 1 year ago 
  329 | 2020-01-08 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  0.000 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |             9648.898 |                      2956.777 |             8158.299 |                      6917.227 |            
  330 | 2020-01-09 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  0.000 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |             9651.773 |                      2956.777 |             8164.627 |                      6917.659 |            
  331 | 2020-01-11 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  6.304 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |               14.423 |                         0.120 |                0.000 |                         0.000 |       9.134
  332 | 2020-01-12 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  7.424 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |               20.704 |                         0.615 |                0.000 |                         0.000 |      15.438
  333 | 2020-01-13 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  7.257 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |               29.230 |                         0.615 |                0.000 |                         0.000 |      22.862
  334 | 2020-01-14 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  6.747 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |               30.535 |                         0.615 |                5.621 |                         0.252 |      30.119
  335 | 2020-01-15 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  5.822 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |               32.128 |                         0.615 |               11.657 |                         1.695 |      36.866
  336 | 2020-01-16 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  6.104 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |               35.107 |                         0.615 |               18.425 |                         1.695 |      42.688
  337 | 2020-01-17 |       0.00 |        0.000 |        0.000 |                 0.000 |                 0.000 |              0.00 |              0.00 |  6.144 |     0.00 |                 0.0 |                 0.0 |                0.0 |       0.00 |               36.625 |                         0.615 |               25.185 |                         2.612 |

Er is, op 11 januari 2020 wel een meterwissel geweest. Kan dat invloed hierop hebben. Vanaf 11 januari 2020 is de meter aangepast, zodat ook gas nu met de slimmemeter mee komt

dennissiemensma commented 1 year ago

Klopt je data in dsmr_consumption_electricityconsumption dan wel? Dat is de tussentabel voordat het naar dsmr_stats_daystatisticsgaat

dennissiemensma commented 1 year ago

Je kunt nog dsmr_stats_daystatistics en dsmr_stats_hourstatistics leeggooien en dan in de URL /admin/dsmr_backend/scheduledprocess/ het proces Generate day and hour statistics zoeken, aanpassen naar vandaag + nu.

Dan hergenereert die als het goed is weer de statistieken. In de debug logs zie je dat dan ook.

Je kunt ook nog wat andere proberen, zoals:

sudo su - dsmr
./manage.py dsmr_stats_reconstruct_missing_day_statistics

Nu ik het weer teruglees, die was hier ook voor gemaakt. Zie https://github.com/dsmrreader/dsmr-reader/issues/1302#issuecomment-792328271. Probeer die anders eerst maar.

PGTrynwalden commented 1 year ago

Hoi Dennis, Heb het laatste commando gedraaid en de ontbrekende data staat erin.

Dank voor het meedenken.

dennissiemensma commented 1 year ago

Fijn om te horen! Ik zal het bij de andere issues zetten.