Closed frankforpresident closed 4 years ago
Bedankt voor je melding. Wil je dit eens proberen:
sudo su - dsmr
./manage.py dsmr_sqlsequencereset
Aangezien je Docker gebruikt, heb je wellicht het inloggen als dsmr
user niet nodig, maar voor de volledigheid vermeld ik het.
Hi Dennis,
Ik zie geen effect na dit commando, ik blijf duplicate key errors krijgen. Ook na het herstarten van de container
Je kunt het dan nog handmatig proberen in de postgresql container:
sudo su - postgres
psql dsmrreader
Doe dan:
BEGIN;
SELECT setval(pg_get_serial_sequence('"dsmr_datalogger_meterstatistics"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_datalogger_meterstatistics";
COMMIT;
ERROR: duplicate key value violates unique constraint "dsmr_datalogger_meterstatistics_pkey" DETAIL: Key (id)=(1) already exists. STATEMENT: INSERT INTO "dsmr_datalogger_meterstatistics" ("id", "timestamp", "dsmr_version", "electricity_tariff", "power_failure_count", "long_power_failure_count", "voltage_sag_count_l1", "voltage_sag_count_l2", "voltage_sag_count_l3", "voltage_swell_count_l1", "voltage_swell_count_l2", "voltage_swell_count_l3", "rejected_telegrams", "latest_telegram") VALUES (1, '2020-06-04T18:52:52.574136+00:00'::timestamptz, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 0, NULL) RETURNING "dsmr_datalogger_meterstatistics"."id" ERROR: duplicate key value violates unique constraint "dsmr_datalogger_meterstatistics_pkey" DETAIL: Key (id)=(1) already exists.
Deze table is trouwens leeg?
SELECT * FROM "dsmr_datalogger_meterstatistics" LIMIT 50 (0.002 s) Edit
No rows.
Je kunt het vergelijken met de autoincrement van mysql. Dat wordt apart bijgehouden. In postgres is dat ook zo en soms botst dat, al is de foutmelding wat obscuur.
Krijg je trouwens wel output op de eerdere query van mij?
dsmrreader=# BEGIN;
BEGIN
dsmrreader=# SELECT setval(pg_get_serial_sequence('"dsmr_datalogger_meterstatistics"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_datalogger_meterstatistics";
setval
--------
1
(1 row)
dsmrreader=# COMMIT;
COMMIT
En weet je zeker dat je een recente foutmelding bekijkt? Het tijd stip uit je comment van 6 minuten terug is 2020-06-04T18:52:52.574136+00:00
, dat is in CEST rond 20:52
zal een copy/past dingetje zijn denk ik, het is er nog steeds :(
ERROR: duplicate key value violates unique constraint "dsmr_datalogger_meterstatistics_pkey"
DETAIL: Key (id)=(1) already exists.
STATEMENT: INSERT INTO "dsmr_datalogger_meterstatistics" ("id", "timestamp", "dsmr_version", "electricity_tariff", "power_failure_count", "long_power_failure_count", "voltage_sag_count_l1", "voltage_sag_count_l2", "voltage_sag_count_l3", "voltage_swell_count_l1", "voltage_swell_count_l2", "voltage_swell_count_l3", "rejected_telegrams", "latest_telegram") VALUES (1, '2020-06-04T19:49:33.085527+00:00'::timestamptz, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 0, NULL) RETURNING "dsmr_datalogger_meterstatistics"."id"
ERROR: duplicate key value violates unique constraint "dsmr_datalogger_meterstatistics_pkey"
DETAIL: Key (id)=(1) already exists.
STATEMENT: INSERT INTO "dsmr_datalogger_meterstatistics" ("id", "timestamp", "dsmr_version", "electricity_tariff", "power_failure_count", "long_power_failure_count", "voltage_sag_count_l1", "voltage_sag_count_l2", "voltage_sag_count_l3", "voltage_swell_count_l1", "voltage_swell_count_l2", "voltage_swell_count_l3", "rejected_telegrams", "latest_telegram") VALUES (1, '2020-06-04T19:49:34.154890+00:00'::timestamptz, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 0, NULL) RETURNING "dsmr_datalogger_meterstatistics"."id"
Ik krijg wel een response terug van postgres (via adminer)
1
Pfoe, uhm je kunt dan nog handmatig alle tabellen doen, maar ik vermoed niet dat dat het gaat oplossen:
BEGIN;
SELECT setval(pg_get_serial_sequence('"django_admin_log"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "django_admin_log";
SELECT setval(pg_get_serial_sequence('"auth_permission"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "auth_permission";
SELECT setval(pg_get_serial_sequence('"auth_group_permissions"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "auth_group_permissions";
SELECT setval(pg_get_serial_sequence('"auth_group"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "auth_group";
SELECT setval(pg_get_serial_sequence('"auth_user_groups"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "auth_user_groups";
SELECT setval(pg_get_serial_sequence('"auth_user_user_permissions"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "auth_user_user_permissions";
SELECT setval(pg_get_serial_sequence('"auth_user"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "auth_user";
SELECT setval(pg_get_serial_sequence('"django_content_type"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "django_content_type";
SELECT setval(pg_get_serial_sequence('"dsmr_api_apisettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_api_apisettings";
SELECT setval(pg_get_serial_sequence('"dsmr_backend_backendsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_backend_backendsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_backend_emailsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_backend_emailsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_backend_scheduledprocess"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_backend_scheduledprocess";
SELECT setval(pg_get_serial_sequence('"dsmr_backup_backupsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_backup_backupsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_backup_dropboxsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_backup_dropboxsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_backup_emailbackupsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_backup_emailbackupsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_consumption_electricityconsumption"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_consumption_electricityconsumption";
SELECT setval(pg_get_serial_sequence('"dsmr_consumption_gasconsumption"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_consumption_gasconsumption";
SELECT setval(pg_get_serial_sequence('"dsmr_consumption_energysupplierprice"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_consumption_energysupplierprice";
SELECT setval(pg_get_serial_sequence('"dsmr_consumption_consumptionsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_consumption_consumptionsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_datalogger_dataloggersettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_datalogger_dataloggersettings";
SELECT setval(pg_get_serial_sequence('"dsmr_datalogger_retentionsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_datalogger_retentionsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_datalogger_dsmrreading"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_datalogger_dsmrreading";
SELECT setval(pg_get_serial_sequence('"dsmr_datalogger_meterstatistics"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_datalogger_meterstatistics";
SELECT setval(pg_get_serial_sequence('"dsmr_frontend_notification"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_frontend_notification";
SELECT setval(pg_get_serial_sequence('"dsmr_frontend_frontendsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_frontend_frontendsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_mindergas_mindergassettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_mindergas_mindergassettings";
SELECT setval(pg_get_serial_sequence('"dsmr_mqtt_mqttbrokersettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_mqtt_mqttbrokersettings";
SELECT setval(pg_get_serial_sequence('"dsmr_mqtt_jsondaytotalsmqttsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_mqtt_jsondaytotalsmqttsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_mqtt_splittopicdaytotalsmqttsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_mqtt_splittopicdaytotalsmqttsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_mqtt_rawtelegrammqttsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_mqtt_rawtelegrammqttsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_mqtt_jsontelegrammqttsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_mqtt_jsontelegrammqttsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_mqtt_splittopictelegrammqttsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_mqtt_splittopictelegrammqttsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_mqtt_splittopicmeterstatisticsmqttsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_mqtt_splittopicmeterstatisticsmqttsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_mqtt_jsongasconsumptionmqttsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_mqtt_jsongasconsumptionmqttsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_mqtt_splittopicgasconsumptionmqttsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_mqtt_splittopicgasconsumptionmqttsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_mqtt_message"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_mqtt_message";
SELECT setval(pg_get_serial_sequence('"dsmr_notification_notificationsetting"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_notification_notificationsetting";
SELECT setval(pg_get_serial_sequence('"dsmr_notification_statusnotificationsetting"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_notification_statusnotificationsetting";
SELECT setval(pg_get_serial_sequence('"dsmr_pvoutput_pvoutputapisettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_pvoutput_pvoutputapisettings";
SELECT setval(pg_get_serial_sequence('"dsmr_pvoutput_pvoutputaddstatussettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_pvoutput_pvoutputaddstatussettings";
SELECT setval(pg_get_serial_sequence('"dsmr_stats_daystatistics"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_stats_daystatistics";
SELECT setval(pg_get_serial_sequence('"dsmr_stats_hourstatistics"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_stats_hourstatistics";
SELECT setval(pg_get_serial_sequence('"dsmr_stats_electricitystatistics"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_stats_electricitystatistics";
SELECT setval(pg_get_serial_sequence('"dsmr_stats_note"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_stats_note";
SELECT setval(pg_get_serial_sequence('"dsmr_weather_weathersettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_weather_weathersettings";
SELECT setval(pg_get_serial_sequence('"dsmr_weather_temperaturereading"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_weather_temperaturereading";
COMMIT;
Wat voor issues met Postgresql had je dan precies? Normaal gesproken lost de sequencereset dit namelijk gewoon op bij deze fout. Helemaal als er nog geen data in je tabel staat.
En wat geeft dit voor resultaat?
SELECT pg_get_serial_sequence('"dsmr_datalogger_meterstatistics"','id'), max("id") FROM "dsmr_datalogger_meterstatistics";
Server was uitgevallen en postgres starten niet meer goed op.
Heb dit probleem kunnen oplossen door een transation log reset uit te voeren met:
gosu postgres pg_resetxlog -f /var/lib/postgres/data
De select komt met NULL terug.
SELECT pg_get_serial_sequence('"dsmr_datalogger_meterstatistics"','id'), max("id") FROM "dsmr_datalogger_meterstatistics";
pg_get_serial_sequence max
public.dsmr_datalogger_meterstatistics_id_seq NULL
Heb die lijst van selects geprobeerd maar geen effect
En in de rest van de tabellen staat nog wel data neem ik aan?
SELECT schemaname,relname,n_live_tup
FROM pg_stat_user_tables
ORDER BY n_live_tup DESC;
En wat geeft dit terug (kan zijn dat het alleen op de commandline kan en een alias is):
\dS dsmr_datalogger_meterstatistics
\di dsmr_datalogger_meterstatistics_pkey
Welke serverversie van postgres gebruik je trouwens? Ik kan me bijna niet voorstellen dat er issues zijn tussen eventuele andere versies, maar toch...
postgres 9.6
er zit data in
schemaname | relname | n_live_tup |
---|---|---|
public | dsmr_datalogger_dsmrreading | 9861 |
public | dsmr_mqtt_message | 48 |
public | dsmr_backend_scheduledprocess | 8 |
public | dsmr_datalogger_meterstatistics | 1 |
public | dsmr_mqtt_jsondaytotalsmqttsettings | 0 |
de rest is 0
Ben nog even aan het zien hoe ik via cli kan werken
Als de rest 0 is, dan mis je volgens mij een hoop data hoor. Ik ben bang dat je dan erg veel kwijt bent en het is dan de vraag hoeveel je kunt hergenereren op basis van de data in dsmr_datalogger_dsmrreading
.
Ik neem aan dat dit geen nieuwe installatie is toch? Zo nee, heb je nog een backup?
Wat ook nog wrang is, is dat er in dsmr_datalogger_meterstatistics
wel wat lijkt te zitten. Dat zeggende, bovenstaande query is geen exacte select maar een schatting. Je zou nog even wat andere tabellen kunnen checken met een exacte select.
Ter vergelijking bij mij:
schemaname | relname | n_live_tup
------------+-------------------------------------------------+------------
public | dsmr_datalogger_dsmrreading | 141105
public | dsmr_consumption_electricityconsumption | 87516
public | dsmr_stats_hourstatistics | 38536
public | dsmr_consumption_gasconsumption | 38505
public | dsmr_weather_temperaturereading | 37941
public | dsmr_stats_daystatistics | 1620
public | django_admin_log | 233
public | django_migrations | 194
public | django_session | 86
public | dsmr_stats_note | 57
public | django_content_type | 45
public | auth_permission | 35
public | dsmr_frontend_notification | 33
public | dsmr_backend_scheduledcall | 10
public | dsmr_backend_scheduledprocess | 8
public | dsmr_consumption_energysupplierprice | 7
public | auth_user | 2
public | dsmr_api_apisettings | 1
public | dsmr_weather_weathersettings | 1
public | dsmr_mqtt_rawtelegrammqttsettings | 1
public | dsmr_datalogger_meterstatistics | 1
public | dsmr_notification_notificationsetting | 1
public | dsmr_backend_emailsettings | 1
public | dsmr_datalogger_retentionsettings | 1
public | dsmr_backup_emailbackupsettings | 1
public | dsmr_notification_statusnotificationsetting | 1
public | dsmr_backend_backendsettings | 1
public | dsmr_mqtt_splittopictelegrammqttsettings | 1
public | dsmr_mqtt_mqttbrokersettings | 1
public | dsmr_mqtt_jsontelegrammqttsettings | 1
public | dsmr_mqtt_jsondaytotalsmqttsettings | 1
public | dsmr_mqtt_splittopicgasconsumptionmqttsettings | 1
public | dsmr_pvoutput_pvoutputapisettings | 1
public | dsmr_mqtt_jsongasconsumptionmqttsettings | 1
public | dsmr_stats_electricitystatistics | 1
public | dsmr_mqtt_splittopicmeterstatisticsmqttsettings | 1
public | dsmr_frontend_frontendsettings | 1
public | dsmr_backup_backupsettings | 1
public | dsmr_pvoutput_pvoutputaddstatussettings | 1
public | dsmr_backup_dropboxsettings | 1
public | dsmr_mqtt_splittopicdaytotalsmqttsettings | 1
public | dsmr_mqtt_message | 1
public | dsmr_datalogger_dataloggersettings | 1
public | dsmr_mindergas_mindergassettings | 1
public | dsmr_consumption_consumptionsettings | 1
public | auth_group_permissions | 0
public | auth_group | 0
public | auth_user_groups | 0
public | auth_user_user_permissions | 0
Ik ben gestart vanaf januari met het verzamelen van deze data. Nu werkt de app nog wel. alleen live data komt niet meer binnen. Ik kan perfect in het verleden kijken. Heb geen backup van deze DB.
Is het misschien mogelijk om deze table (dsmr_datalogger_meterstatistics) te droppen en opnieuw aan te maken?
schemaname | relname | n_live_tup |
---|---|---|
public | dsmr_datalogger_dsmrreading | 10549 |
public | dsmr_mqtt_message | 30 |
public | dsmr_backend_scheduledprocess | 8 |
public | dsmr_datalogger_meterstatistics | 1 |
public | dsmr_mqtt_jsondaytotalsmqttsettings | 0 |
public | dsmr_mqtt_splittopicgasconsumptionmqttsettings | 0 |
public | auth_permission | 0 |
public | django_migrations | 0 |
public | auth_group | 0 |
public | auth_user | 0 |
public | dsmr_frontend_frontendsettings | 0 |
public | dsmr_consumption_consumptionsettings | 0 |
public | auth_group_permissions | 0 |
public | auth_user_groups | 0 |
public | dsmr_pvoutput_pvoutputapisettings | 0 |
public | dsmr_mqtt_mqttbrokersettings | 0 |
public | dsmr_stats_note | 0 |
public | dsmr_mqtt_splittopicdaytotalsmqttsettings | 0 |
public | django_admin_log | 0 |
public | dsmr_datalogger_dataloggersettings | 0 |
public | dsmr_stats_electricitystatistics | 0 |
public | dsmr_mqtt_splittopicmeterstatisticsmqttsettings | 0 |
public | dsmr_notification_notificationsetting | 0 |
public | dsmr_notification_statusnotificationsetting | 0 |
public | dsmr_consumption_energysupplierprice | 0 |
public | dsmr_mindergas_mindergassettings | 0 |
public | dsmr_mqtt_jsontelegrammqttsettings | 0 |
public | dsmr_api_apisettings | 0 |
public | django_session | 0 |
public | dsmr_mqtt_splittopictelegrammqttsettings | 0 |
public | dsmr_stats_hourstatistics | 0 |
public | dsmr_consumption_electricityconsumption | 0 |
public | dsmr_pvoutput_pvoutputaddstatussettings | 0 |
public | dsmr_mqtt_rawtelegrammqttsettings | 0 |
public | dsmr_datalogger_retentionsettings | 0 |
public | dsmr_frontend_notification | 0 |
public | django_content_type | 0 |
public | dsmr_stats_daystatistics | 0 |
public | dsmr_backup_backupsettings | 0 |
public | dsmr_backend_emailsettings | 0 |
public | dsmr_backup_dropboxsettings | 0 |
public | dsmr_backup_emailbackupsettings | 0 |
public | dsmr_mqtt_jsongasconsumptionmqttsettings | 0 |
public | auth_user_user_permissions | 0 |
public | dsmr_weather_temperaturereading | 0 |
public | dsmr_weather_weathersettings | 0 |
public | dsmr_consumption_gasconsumption | 0 |
public | dsmr_backend_backendsettings | 0 |
Oke dat is vreemd. Zorg er sowieso voor dat je even een backup maakt, als je die nog niet had.
Wellicht kun je eens de vacuumer + analyzer over de database gooien en daarna nogmaals kijken naar bovenstaande aantallen:
vacuumdb f -F -v -z -d dsmrreader
Ik zal er morgen naar kijken. Begint wat laat te worden.
Alvast super bedankt voor de support!
Het droppen en opnieuw aanmaken van de tabel is overigens wat lastig, omdat alles gemanaged wordt met automatische migraties. Je kunt nog wel overwegen om een dump te maken, de database te verwijderen, opnieuw aanmaken en de dump weer in te lezen.
pg_dump -d dsmrreader > dump.sql
Als je DB heel groot is, kun je er eventueel gzip tussen gooien.
@dennissiemensma,
Raar maar waar, PG heeft zichzelf kunnen herstellen. Ik zou er deze morgen naar kijken en het issue was al opgelost, eerst dacht ik dat het door de vacuum kwam maar nu ik in de logs kijk is het zal sinds 2020-06-07T00:04:08 opgelost. Heb nog wel aan andere services gewerkt dit weekend. Misschien related, geen idee.
Ohwel, ik heb nu wel een pg_dump job draaien om mijn DB te back-uppen voor als het nodig is.
Alvast bedankt voor de support.
Fijn om te horen!
Wat gebruik je?
Heb een postgres issue gehad waardoor deze niet meer wou opstarten. heb dit kunnen oplossen maar loop nu tegen een dsmr issue aan.
Heb de foutmelding al proberen op te lossen via https://github.com/dennissiemensma/dsmr-reader/issues/737 en https://github.com/dennissiemensma/dsmr-reader/issues/866 maar zonder succes.
Hopelijk kan iemand mij hier mee helpen