dsmrreader / dsmr-reader

DSMR-telegram reader & data visualizer for hobbyists. Free for non-commercial use.
https://dsmr-reader.readthedocs.io
Other
463 stars 95 forks source link

ERROR: duplicate key value violates unique constraint "dsmr_datalogger_meterstatistics_pkey" #1010

Closed frankforpresident closed 4 years ago

frankforpresident commented 4 years ago

Wat gebruik je?


Heb een postgres issue gehad waardoor deze niet meer wou opstarten. heb dit kunnen oplossen maar loop nu tegen een dsmr issue aan.

ERROR:  duplicate key value violates unique constraint "dsmr_datalogger_meterstatistics_pkey"
DETAIL:  Key (id)=(1) already exists.
STATEMENT:  INSERT INTO "dsmr_datalogger_meterstatistics" ("id", "timestamp", "dsmr_version", "electricity_tariff", "power_failure_count", "long_power_failure_count", "voltage_sag_count_l1", "voltage_sag_count_l2", "voltage_sag_count_l3", "voltage_swell_count_l1", "voltage_swell_count_l2", "voltage_swell_count_l3", "rejected_telegrams", "latest_telegram") VALUES (1, '2020-06-04T18:42:20.491933+00:00'::timestamptz, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 0, NULL) RETURNING "dsmr_datalogger_meterstatistics"."id"
ERROR:  duplicate key value violates unique constraint "dsmr_datalogger_meterstatistics_pkey"
DETAIL:  Key (id)=(1) already exists.
STATEMENT:  INSERT INTO "dsmr_datalogger_meterstatistics" ("id", "timestamp", "dsmr_version", "electricity_tariff", "power_failure_count", "long_power_failure_count", "voltage_sag_count_l1", "voltage_sag_count_l2", "voltage_sag_count_l3", "voltage_swell_count_l1", "voltage_swell_count_l2", "voltage_swell_count_l3", "rejected_telegrams", "latest_telegram") VALUES (1, '2020-06-04T18:42:20.742991+00:00'::timestamptz, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 0, NULL) RETURNING "dsmr_datalogger_meterstatistics"."id"
ERROR:  duplicate key value violates unique constraint "dsmr_datalogger_meterstatistics_pkey"
DETAIL:  Key (id)=(1) already exists.
STATEMENT:  INSERT INTO "dsmr_datalogger_meterstatistics" ("id", "timestamp", "dsmr_version", "electricity_tariff", "power_failure_count", "long_power_failure_count", "voltage_sag_count_l1", "voltage_sag_count_l2", "voltage_sag_count_l3", "voltage_swell_count_l1", "voltage_swell_count_l2", "voltage_swell_count_l3", "rejected_telegrams", "latest_telegram") VALUES (1, '2020-06-04T18:42:21.250794+00:00'::timestamp

Heb de foutmelding al proberen op te lossen via https://github.com/dennissiemensma/dsmr-reader/issues/737 en https://github.com/dennissiemensma/dsmr-reader/issues/866 maar zonder succes.

bash-5.0# ./manage.py showmigrations
admin
 [X] 0001_initial
 [X] 0002_logentry_remove_auto_add
 [X] 0003_logentry_add_action_flag_choices
auth
 [X] 0001_initial
 [X] 0002_alter_permission_name_max_length
 [X] 0003_alter_user_email_max_length
 [X] 0004_alter_user_username_opts
 [X] 0005_alter_user_last_login_null
 [X] 0006_require_contenttypes_0002
 [X] 0007_alter_validators_add_error_messages
 [X] 0008_alter_user_username_max_length
 [X] 0009_alter_user_last_name_max_length
 [X] 0010_alter_group_name_max_length
 [X] 0011_update_proxy_permissions
contenttypes
 [X] 0001_initial
 [X] 0002_remove_content_type_name
dsmr_api
 [X] 0001_api_settings
 [X] 0002_generate_random_auth_key
 [X] 0003_create_api_user
dsmr_backend
 [X] 0001_backend_language
 [X] 0002_email_settings
 [X] 0003_scheduled_processes
 [X] 0004_backend_schedule_process_active_flag
 [X] 0005_schedule_auto_update_check
 [X] 0006_backend_auto_update_check
 [X] 0007_schedule_stats_generator
 [X] 0008_scheduled_process_fields
 [X] 0009_update_email_settings_mail_from
 [X] 0010_process_sleep
 [X] 0011_disable_capabilities
 [X] 0012_fix_email_from
 [X] 0013_allow_higher_backend_sleep
 [X] 0014_verbose_field_translations
dsmr_backup
 [X] 0001_initial
 [X] 0002_settings_documentation
 [X] 0003_dropbox_sync_field
 [X] 0004_custom_backup_path
 [X] 0005_email_backup_settings
 [X] 0006_scheduled_email_backup
 [X] 0007_remove_backupsettings_compress
 [X] 0008_verbose_field_translations
 [X] 0009_compression_level
 [X] 0010_schedule_backup
 [X] 0011_remove_backupsettings_latest_backup
dsmr_consumption
 [X] 0001_squashed_0004_recalculate_gas_consumption
 [X] 0002_verbose_text
 [X] 0003_electricity_consumption_indexes
 [X] 0004_merge_electricity_tariffs
 [X] 0005_phase_currently_delivered
 [X] 0006_dsmr_firmware_v5
 [X] 0007_settings_documentation
 [X] 0008_energysupplier_return_prices
 [X] 0009_consumption_settings_text
 [X] 0010_phases_currently_returned
 [X] 0011_track_phase_voltages
 [X] 0012_add_voltage_indexes
 [X] 0013_schedule_generate_consumption
 [X] 0014_gas_grouping
 [X] 0015_track_power_current
dsmr_datalogger
 [X] 0001_squashed_0005_optional_gas_readings
 [X] 0002_meta_names
 [X] 0003_telegram_checksum
 [X] 0004_phase_currently_delivered
 [X] 0005_verify_telegram_crc_setting
 [X] 0006_dsmr_firmware_v5
 [X] 0007_dsmrreading_timestamp_index
 [X] 0008_dsmrreading_help_text
 [X] 0009_retention_settings
 [X] 0010_phases_currently_returned
 [X] 0011_raw_telegram_insight
 [X] 0012_track_phase_voltages
 [X] 0013_remove_datalogger_track_settings
 [X] 0014_remove_optional_crc_check
 [X] 0015_datalogger_foreign_countries
 [X] 0016_process_sleep
 [X] 0017_log_telegrams
 [X] 0018_meterstatistics_no_autonow
 [X] 0019_verbose_field_translations
 [X] 0020_track_power_current
 [X] 0021_schedule_retention_data_rotation
 [X] 0022_three_month_retention
dsmr_frontend
 [X] 0001_initial
 [X] 0002_recent_history_weeks
 [X] 0003_drop_history_page
 [X] 0004_chart_colors
 [X] 0005_notifications
 [X] 0006_notifications_initial
 [X] 0007_merge_electricity_tariffs
 [X] 0008_merge_electricity_tariffs_notification
 [X] 0009_docs_no_reverse_match
 [X] 0010_drop_reverse_dashboard_graphs_setting
 [X] 0011_phase_currently_delivered
 [X] 0012_frontendsettings_dashboard_graph_width
 [X] 0013_echart_graphs_message
 [X] 0014_phases_currently_returned
 [X] 0015_notification_meta
 [X] 0016_v200_release
 [X] 0017_v201_release
 [X] 0018_v202_release
 [X] 0019_v210_release
 [X] 0020_v230_release
 [X] 0021_v260_release
 [X] 0022_v270_release
 [X] 0023_v290_release
 [X] 0024_v2100_release
 [X] 0025_increase_max_dashboard_graph_width
 [X] 0026_v2140_release
 [X] 0027_v215_release
 [X] 0028_v3_0_fix_upgrade_redirect
 [X] 0029_gas_graph_style
 [X] 0030_electricity_graph_style
 [X] 0031_tariff_names
 [X] 0032_v3_4_0_release
 [X] 0033_django_colorfield_update
 [X] 0034_mysql_timezone_support
 [X] 0035_graphs_range
 [X] 0036_graphs_zoom
dsmr_mindergas
 [X] 0001_mindergas
 [X] 0002_mindergas_notification
 [X] 0003_mindergas_next_export_datetime
 [X] 0004_mindergas_latest_sync
 [X] 0005_schedule_mindergas_export
dsmr_mqtt
 [X] 0001_mqtt_settings
 [X] 0002_daytotalsmqttsettings
 [X] 0003_splittopicmeterstatisticsmqttsettings
 [X] 0004_phases_currently_returned
 [X] 0005_mqtt_use_local_timezone
 [X] 0006_mqtt_refactoring
 [X] 0007_mqtts
 [X] 0008_mqtt_null_payload
 [X] 0009_remove_optional_debug
 [X] 0010_mqtt_gas_consumption
 [X] 0011_mqtt_meta_description
 [X] 0012_mqtt_voltage_defaults
 [X] 0013_process_sleep
 [X] 0014_mqtt_telegram_defaults
dsmr_notification
 [X] 0001_initial
 [X] 0002_notificationsetting_next_notification
 [X] 0003_notification_notification
 [X] 0004_statusnotificationsetting
 [X] 0005_notify_my_android_ended_support
 [X] 0006_support_for_pushover
 [X] 0007_support_for_telegram
dsmr_pvoutput
 [X] 0001_pvoutput_settings
 [X] 0002_pvoutput_latest_sync
dsmr_stats
 [X] 0001_squashed_0016_drop_stats_settings
 [X] 0002_regenerate_missing_gas_data
 [X] 0003_hour_statistics_gas_default
 [X] 0004_hour_statistics_gas_default_retroactive
 [X] 0005_statistics_exportverbose_names
 [X] 0006_min_max_temperature_statistics
 [X] 0007_min_max_temperature_statistics_retroactive
 [X] 0008_meta_names
 [X] 0009_statistics_editable
 [X] 0010_statistics_models_index
 [X] 0011_note_model_index
 [X] 0012_electricity_statistics
 [X] 0013_all_time_low
 [X] 0014_day_total_cost_index
dsmr_weather
 [X] 0001_weather_models
 [X] 0002_meta
 [X] 0003_next_sync_setting
 [X] 0004_next_sync_setting_retroactive
 [X] 0005_weather_refactor_scheduling
 [X] 0006_schedule_weather_update
sessions
 [X] 0001_initial
bash-5.0# ./manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, dsmr_api, dsmr_backend, dsmr_backup, dsmr_consumption, dsmr_datalogger, dsmr_frontend, dsmr_mindergas, dsmr_mqtt, dsmr_notification, dsmr_pvoutput, dsmr_stats, dsmr_weather, sessions
Running migrations:
  No migrations to apply.

Hopelijk kan iemand mij hier mee helpen

dennissiemensma commented 4 years ago

Bedankt voor je melding. Wil je dit eens proberen:

sudo su - dsmr
./manage.py dsmr_sqlsequencereset
dennissiemensma commented 4 years ago

Aangezien je Docker gebruikt, heb je wellicht het inloggen als dsmr user niet nodig, maar voor de volledigheid vermeld ik het.

frankforpresident commented 4 years ago

Hi Dennis,

Ik zie geen effect na dit commando, ik blijf duplicate key errors krijgen. Ook na het herstarten van de container

dennissiemensma commented 4 years ago

Je kunt het dan nog handmatig proberen in de postgresql container:

sudo su - postgres
psql dsmrreader

Doe dan:

BEGIN;
SELECT setval(pg_get_serial_sequence('"dsmr_datalogger_meterstatistics"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_datalogger_meterstatistics";
COMMIT;
frankforpresident commented 4 years ago

ERROR: duplicate key value violates unique constraint "dsmr_datalogger_meterstatistics_pkey" DETAIL: Key (id)=(1) already exists. STATEMENT: INSERT INTO "dsmr_datalogger_meterstatistics" ("id", "timestamp", "dsmr_version", "electricity_tariff", "power_failure_count", "long_power_failure_count", "voltage_sag_count_l1", "voltage_sag_count_l2", "voltage_sag_count_l3", "voltage_swell_count_l1", "voltage_swell_count_l2", "voltage_swell_count_l3", "rejected_telegrams", "latest_telegram") VALUES (1, '2020-06-04T18:52:52.574136+00:00'::timestamptz, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 0, NULL) RETURNING "dsmr_datalogger_meterstatistics"."id" ERROR: duplicate key value violates unique constraint "dsmr_datalogger_meterstatistics_pkey" DETAIL: Key (id)=(1) already exists.

frankforpresident commented 4 years ago

Deze table is trouwens leeg?

SELECT * FROM "dsmr_datalogger_meterstatistics" LIMIT 50 (0.002 s) Edit

No rows.

dennissiemensma commented 4 years ago

Je kunt het vergelijken met de autoincrement van mysql. Dat wordt apart bijgehouden. In postgres is dat ook zo en soms botst dat, al is de foutmelding wat obscuur.

Krijg je trouwens wel output op de eerdere query van mij?

dsmrreader=# BEGIN;
BEGIN
dsmrreader=# SELECT setval(pg_get_serial_sequence('"dsmr_datalogger_meterstatistics"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_datalogger_meterstatistics";
 setval 
--------
      1
(1 row)

dsmrreader=# COMMIT;
COMMIT
dennissiemensma commented 4 years ago

En weet je zeker dat je een recente foutmelding bekijkt? Het tijd stip uit je comment van 6 minuten terug is 2020-06-04T18:52:52.574136+00:00, dat is in CEST rond 20:52

frankforpresident commented 4 years ago

zal een copy/past dingetje zijn denk ik, het is er nog steeds :(

ERROR:  duplicate key value violates unique constraint "dsmr_datalogger_meterstatistics_pkey"
DETAIL:  Key (id)=(1) already exists.
STATEMENT:  INSERT INTO "dsmr_datalogger_meterstatistics" ("id", "timestamp", "dsmr_version", "electricity_tariff", "power_failure_count", "long_power_failure_count", "voltage_sag_count_l1", "voltage_sag_count_l2", "voltage_sag_count_l3", "voltage_swell_count_l1", "voltage_swell_count_l2", "voltage_swell_count_l3", "rejected_telegrams", "latest_telegram") VALUES (1, '2020-06-04T19:49:33.085527+00:00'::timestamptz, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 0, NULL) RETURNING "dsmr_datalogger_meterstatistics"."id"
ERROR:  duplicate key value violates unique constraint "dsmr_datalogger_meterstatistics_pkey"
DETAIL:  Key (id)=(1) already exists.
STATEMENT:  INSERT INTO "dsmr_datalogger_meterstatistics" ("id", "timestamp", "dsmr_version", "electricity_tariff", "power_failure_count", "long_power_failure_count", "voltage_sag_count_l1", "voltage_sag_count_l2", "voltage_sag_count_l3", "voltage_swell_count_l1", "voltage_swell_count_l2", "voltage_swell_count_l3", "rejected_telegrams", "latest_telegram") VALUES (1, '2020-06-04T19:49:34.154890+00:00'::timestamptz, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 0, NULL) RETURNING "dsmr_datalogger_meterstatistics"."id"

Ik krijg wel een response terug van postgres (via adminer)

setval

1

dennissiemensma commented 4 years ago

Pfoe, uhm je kunt dan nog handmatig alle tabellen doen, maar ik vermoed niet dat dat het gaat oplossen:

BEGIN;
SELECT setval(pg_get_serial_sequence('"django_admin_log"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "django_admin_log";
SELECT setval(pg_get_serial_sequence('"auth_permission"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "auth_permission";
SELECT setval(pg_get_serial_sequence('"auth_group_permissions"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "auth_group_permissions";
SELECT setval(pg_get_serial_sequence('"auth_group"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "auth_group";
SELECT setval(pg_get_serial_sequence('"auth_user_groups"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "auth_user_groups";
SELECT setval(pg_get_serial_sequence('"auth_user_user_permissions"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "auth_user_user_permissions";
SELECT setval(pg_get_serial_sequence('"auth_user"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "auth_user";
SELECT setval(pg_get_serial_sequence('"django_content_type"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "django_content_type";
SELECT setval(pg_get_serial_sequence('"dsmr_api_apisettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_api_apisettings";
SELECT setval(pg_get_serial_sequence('"dsmr_backend_backendsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_backend_backendsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_backend_emailsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_backend_emailsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_backend_scheduledprocess"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_backend_scheduledprocess";
SELECT setval(pg_get_serial_sequence('"dsmr_backup_backupsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_backup_backupsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_backup_dropboxsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_backup_dropboxsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_backup_emailbackupsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_backup_emailbackupsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_consumption_electricityconsumption"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_consumption_electricityconsumption";
SELECT setval(pg_get_serial_sequence('"dsmr_consumption_gasconsumption"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_consumption_gasconsumption";
SELECT setval(pg_get_serial_sequence('"dsmr_consumption_energysupplierprice"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_consumption_energysupplierprice";
SELECT setval(pg_get_serial_sequence('"dsmr_consumption_consumptionsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_consumption_consumptionsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_datalogger_dataloggersettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_datalogger_dataloggersettings";
SELECT setval(pg_get_serial_sequence('"dsmr_datalogger_retentionsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_datalogger_retentionsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_datalogger_dsmrreading"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_datalogger_dsmrreading";
SELECT setval(pg_get_serial_sequence('"dsmr_datalogger_meterstatistics"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_datalogger_meterstatistics";
SELECT setval(pg_get_serial_sequence('"dsmr_frontend_notification"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_frontend_notification";
SELECT setval(pg_get_serial_sequence('"dsmr_frontend_frontendsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_frontend_frontendsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_mindergas_mindergassettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_mindergas_mindergassettings";
SELECT setval(pg_get_serial_sequence('"dsmr_mqtt_mqttbrokersettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_mqtt_mqttbrokersettings";
SELECT setval(pg_get_serial_sequence('"dsmr_mqtt_jsondaytotalsmqttsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_mqtt_jsondaytotalsmqttsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_mqtt_splittopicdaytotalsmqttsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_mqtt_splittopicdaytotalsmqttsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_mqtt_rawtelegrammqttsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_mqtt_rawtelegrammqttsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_mqtt_jsontelegrammqttsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_mqtt_jsontelegrammqttsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_mqtt_splittopictelegrammqttsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_mqtt_splittopictelegrammqttsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_mqtt_splittopicmeterstatisticsmqttsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_mqtt_splittopicmeterstatisticsmqttsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_mqtt_jsongasconsumptionmqttsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_mqtt_jsongasconsumptionmqttsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_mqtt_splittopicgasconsumptionmqttsettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_mqtt_splittopicgasconsumptionmqttsettings";
SELECT setval(pg_get_serial_sequence('"dsmr_mqtt_message"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_mqtt_message";
SELECT setval(pg_get_serial_sequence('"dsmr_notification_notificationsetting"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_notification_notificationsetting";
SELECT setval(pg_get_serial_sequence('"dsmr_notification_statusnotificationsetting"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_notification_statusnotificationsetting";
SELECT setval(pg_get_serial_sequence('"dsmr_pvoutput_pvoutputapisettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_pvoutput_pvoutputapisettings";
SELECT setval(pg_get_serial_sequence('"dsmr_pvoutput_pvoutputaddstatussettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_pvoutput_pvoutputaddstatussettings";
SELECT setval(pg_get_serial_sequence('"dsmr_stats_daystatistics"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_stats_daystatistics";
SELECT setval(pg_get_serial_sequence('"dsmr_stats_hourstatistics"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_stats_hourstatistics";
SELECT setval(pg_get_serial_sequence('"dsmr_stats_electricitystatistics"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_stats_electricitystatistics";
SELECT setval(pg_get_serial_sequence('"dsmr_stats_note"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_stats_note";
SELECT setval(pg_get_serial_sequence('"dsmr_weather_weathersettings"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_weather_weathersettings";
SELECT setval(pg_get_serial_sequence('"dsmr_weather_temperaturereading"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "dsmr_weather_temperaturereading";
COMMIT;
dennissiemensma commented 4 years ago

Wat voor issues met Postgresql had je dan precies? Normaal gesproken lost de sequencereset dit namelijk gewoon op bij deze fout. Helemaal als er nog geen data in je tabel staat.

dennissiemensma commented 4 years ago

En wat geeft dit voor resultaat?

SELECT pg_get_serial_sequence('"dsmr_datalogger_meterstatistics"','id'), max("id") FROM "dsmr_datalogger_meterstatistics";
frankforpresident commented 4 years ago

Server was uitgevallen en postgres starten niet meer goed op.

Heb dit probleem kunnen oplossen door een transation log reset uit te voeren met: gosu postgres pg_resetxlog -f /var/lib/postgres/data

De select komt met NULL terug.

SELECT pg_get_serial_sequence('"dsmr_datalogger_meterstatistics"','id'), max("id") FROM "dsmr_datalogger_meterstatistics";

pg_get_serial_sequence  max
public.dsmr_datalogger_meterstatistics_id_seq   NULL

Heb die lijst van selects geprobeerd maar geen effect

dennissiemensma commented 4 years ago

En in de rest van de tabellen staat nog wel data neem ik aan?

SELECT schemaname,relname,n_live_tup 
  FROM pg_stat_user_tables 
  ORDER BY n_live_tup DESC;
dennissiemensma commented 4 years ago

En wat geeft dit terug (kan zijn dat het alleen op de commandline kan en een alias is):

\dS dsmr_datalogger_meterstatistics

\di dsmr_datalogger_meterstatistics_pkey
dennissiemensma commented 4 years ago

Welke serverversie van postgres gebruik je trouwens? Ik kan me bijna niet voorstellen dat er issues zijn tussen eventuele andere versies, maar toch...

frankforpresident commented 4 years ago

postgres 9.6

er zit data in

schemaname relname n_live_tup
public dsmr_datalogger_dsmrreading 9861
public dsmr_mqtt_message 48
public dsmr_backend_scheduledprocess 8
public dsmr_datalogger_meterstatistics 1
public dsmr_mqtt_jsondaytotalsmqttsettings 0

de rest is 0

Ben nog even aan het zien hoe ik via cli kan werken

dennissiemensma commented 4 years ago

Als de rest 0 is, dan mis je volgens mij een hoop data hoor. Ik ben bang dat je dan erg veel kwijt bent en het is dan de vraag hoeveel je kunt hergenereren op basis van de data in dsmr_datalogger_dsmrreading.

Ik neem aan dat dit geen nieuwe installatie is toch? Zo nee, heb je nog een backup?

Wat ook nog wrang is, is dat er in dsmr_datalogger_meterstatistics wel wat lijkt te zitten. Dat zeggende, bovenstaande query is geen exacte select maar een schatting. Je zou nog even wat andere tabellen kunnen checken met een exacte select.

dennissiemensma commented 4 years ago

Ter vergelijking bij mij:

 schemaname |                     relname                     | n_live_tup 
------------+-------------------------------------------------+------------
 public     | dsmr_datalogger_dsmrreading                     |     141105
 public     | dsmr_consumption_electricityconsumption         |      87516
 public     | dsmr_stats_hourstatistics                       |      38536
 public     | dsmr_consumption_gasconsumption                 |      38505
 public     | dsmr_weather_temperaturereading                 |      37941
 public     | dsmr_stats_daystatistics                        |       1620
 public     | django_admin_log                                |        233
 public     | django_migrations                               |        194
 public     | django_session                                  |         86
 public     | dsmr_stats_note                                 |         57
 public     | django_content_type                             |         45
 public     | auth_permission                                 |         35
 public     | dsmr_frontend_notification                      |         33
 public     | dsmr_backend_scheduledcall                      |         10
 public     | dsmr_backend_scheduledprocess                   |          8
 public     | dsmr_consumption_energysupplierprice            |          7
 public     | auth_user                                       |          2
 public     | dsmr_api_apisettings                            |          1
 public     | dsmr_weather_weathersettings                    |          1
 public     | dsmr_mqtt_rawtelegrammqttsettings               |          1
 public     | dsmr_datalogger_meterstatistics                 |          1
 public     | dsmr_notification_notificationsetting           |          1
 public     | dsmr_backend_emailsettings                      |          1
 public     | dsmr_datalogger_retentionsettings               |          1
 public     | dsmr_backup_emailbackupsettings                 |          1
 public     | dsmr_notification_statusnotificationsetting     |          1
 public     | dsmr_backend_backendsettings                    |          1
 public     | dsmr_mqtt_splittopictelegrammqttsettings        |          1
 public     | dsmr_mqtt_mqttbrokersettings                    |          1
 public     | dsmr_mqtt_jsontelegrammqttsettings              |          1
 public     | dsmr_mqtt_jsondaytotalsmqttsettings             |          1
 public     | dsmr_mqtt_splittopicgasconsumptionmqttsettings  |          1
 public     | dsmr_pvoutput_pvoutputapisettings               |          1
 public     | dsmr_mqtt_jsongasconsumptionmqttsettings        |          1
 public     | dsmr_stats_electricitystatistics                |          1
 public     | dsmr_mqtt_splittopicmeterstatisticsmqttsettings |          1
 public     | dsmr_frontend_frontendsettings                  |          1
 public     | dsmr_backup_backupsettings                      |          1
 public     | dsmr_pvoutput_pvoutputaddstatussettings         |          1
 public     | dsmr_backup_dropboxsettings                     |          1
 public     | dsmr_mqtt_splittopicdaytotalsmqttsettings       |          1
 public     | dsmr_mqtt_message                               |          1
 public     | dsmr_datalogger_dataloggersettings              |          1
 public     | dsmr_mindergas_mindergassettings                |          1
 public     | dsmr_consumption_consumptionsettings            |          1
 public     | auth_group_permissions                          |          0
 public     | auth_group                                      |          0
 public     | auth_user_groups                                |          0
 public     | auth_user_user_permissions                      |          0
frankforpresident commented 4 years ago

Ik ben gestart vanaf januari met het verzamelen van deze data. Nu werkt de app nog wel. alleen live data komt niet meer binnen. Ik kan perfect in het verleden kijken. Heb geen backup van deze DB.

Is het misschien mogelijk om deze table (dsmr_datalogger_meterstatistics) te droppen en opnieuw aan te maken?

schemaname relname n_live_tup
public dsmr_datalogger_dsmrreading 10549
public dsmr_mqtt_message 30
public dsmr_backend_scheduledprocess 8
public dsmr_datalogger_meterstatistics 1
public dsmr_mqtt_jsondaytotalsmqttsettings 0
public dsmr_mqtt_splittopicgasconsumptionmqttsettings 0
public auth_permission 0
public django_migrations 0
public auth_group 0
public auth_user 0
public dsmr_frontend_frontendsettings 0
public dsmr_consumption_consumptionsettings 0
public auth_group_permissions 0
public auth_user_groups 0
public dsmr_pvoutput_pvoutputapisettings 0
public dsmr_mqtt_mqttbrokersettings 0
public dsmr_stats_note 0
public dsmr_mqtt_splittopicdaytotalsmqttsettings 0
public django_admin_log 0
public dsmr_datalogger_dataloggersettings 0
public dsmr_stats_electricitystatistics 0
public dsmr_mqtt_splittopicmeterstatisticsmqttsettings 0
public dsmr_notification_notificationsetting 0
public dsmr_notification_statusnotificationsetting 0
public dsmr_consumption_energysupplierprice 0
public dsmr_mindergas_mindergassettings 0
public dsmr_mqtt_jsontelegrammqttsettings 0
public dsmr_api_apisettings 0
public django_session 0
public dsmr_mqtt_splittopictelegrammqttsettings 0
public dsmr_stats_hourstatistics 0
public dsmr_consumption_electricityconsumption 0
public dsmr_pvoutput_pvoutputaddstatussettings 0
public dsmr_mqtt_rawtelegrammqttsettings 0
public dsmr_datalogger_retentionsettings 0
public dsmr_frontend_notification 0
public django_content_type 0
public dsmr_stats_daystatistics 0
public dsmr_backup_backupsettings 0
public dsmr_backend_emailsettings 0
public dsmr_backup_dropboxsettings 0
public dsmr_backup_emailbackupsettings 0
public dsmr_mqtt_jsongasconsumptionmqttsettings 0
public auth_user_user_permissions 0
public dsmr_weather_temperaturereading 0
public dsmr_weather_weathersettings 0
public dsmr_consumption_gasconsumption 0
public dsmr_backend_backendsettings 0
dennissiemensma commented 4 years ago

Oke dat is vreemd. Zorg er sowieso voor dat je even een backup maakt, als je die nog niet had.

Wellicht kun je eens de vacuumer + analyzer over de database gooien en daarna nogmaals kijken naar bovenstaande aantallen:

vacuumdb f -F -v -z -d dsmrreader
frankforpresident commented 4 years ago

Ik zal er morgen naar kijken. Begint wat laat te worden.

Alvast super bedankt voor de support!

dennissiemensma commented 4 years ago

Het droppen en opnieuw aanmaken van de tabel is overigens wat lastig, omdat alles gemanaged wordt met automatische migraties. Je kunt nog wel overwegen om een dump te maken, de database te verwijderen, opnieuw aanmaken en de dump weer in te lezen.

pg_dump -d dsmrreader > dump.sql

Als je DB heel groot is, kun je er eventueel gzip tussen gooien.

frankforpresident commented 4 years ago

@dennissiemensma,

Raar maar waar, PG heeft zichzelf kunnen herstellen. Ik zou er deze morgen naar kijken en het issue was al opgelost, eerst dacht ik dat het door de vacuum kwam maar nu ik in de logs kijk is het zal sinds 2020-06-07T00:04:08 opgelost. Heb nog wel aan andere services gewerkt dit weekend. Misschien related, geen idee.

Ohwel, ik heb nu wel een pg_dump job draaien om mijn DB te back-uppen voor als het nodig is.

Alvast bedankt voor de support.

dennissiemensma commented 4 years ago

Fijn om te horen!