Python/Django ir SQL abstraction layer - SQAlchemy klaidos

[JuliusLADP]

Daug užklausų į Saugykla baigiasi klaida. Logas - pilnas Django klaidų:

Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: DETAIL: Key (_id)=(c04afccf-55a3-4168-999f-e484db6ff332) already exists. Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: The above exception was the direct cause of the following exception: Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: Traceback (most recent call last): Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/spinta/backends/postgresql/commands/write.py", line 43, in insert Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: connection.execute(qry, patch) Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1200, in execute Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS) Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 313, in _execute_on_connection Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: return connection._execute_clauseelement( Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1389, in _execute_clauseelement Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: ret = self._execute_context( Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1748, in _execute_context Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: self._handle_dbapi_exception( Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1929, in _handle_dbapiexception Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: util.raise( Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_ Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: raise exception Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_context Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: self.dialect.do_execute( Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 716, in do_execute Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: cursor.execute(statement, parameters) Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "datasets/gov/vrk/rinkeju_demografiniai_duomenys/Zymejimas_pkey" Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: DETAIL: Key (_id)=(c04afccf-55a3-4168-999f-e484db6ff332) already exists. Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: [SQL: INSERT INTO "datasets/gov/vrk/rinkeju_demografiniai_duomenys/Zymejimas" (_txn, _created, _id, _revision, vda_prime_key, rinkejo_asm_k, rinkimu_turo_id, faktines_apygardos_id) VALUES (%(_txn)s, TIMEZONE('utc', CURRENT_TIMESTAMP), %(_id)s, %(_revision)s, %(vda_prime_key)s, %(rinkejo_asm_k)s, %(rinkimu_turo_id)s, %(faktines_apygardos_id)s)] Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: [parameters: {'_txn': UUID('f5c41785-06cf-4c8a-86d3-22973199028e'), '_id': UUID('c04afccf-55a3-4168-999f-e484db6ff332'), '_revision': '48e2fd9c-23ef-4c03-bbde-fda59d8bbc15', 'vda_prime_key': 'ab973c76a79da2b', 'rinkejo_asm_k': 'f914984dd083544', 'rinkimu_turo_id': '1906', 'faktines_apygardos_id': None}] Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: (Background on this error at: http://sqlalche.me/e/14/gkpj) Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: During handling of the above exception, another exception occurred: Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: Traceback (most recent call last): Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: await app(scope, receive, sender) Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/starlette/routing.py", line 74, in app Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: response = await f(request) Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/spinta/api/init.py", line 272, in homepage Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: return await create_http_response(context, params, request) Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/spinta/utils/response.py", line 230, in create_http_response Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: return await commands.push( Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/spinta/commands/write.py", line 91, in push Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: status_code, response = await _batch_response( Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/spinta/commands/write.py", line 1183, in _batch_response Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: async for data in results: Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/spinta/accesslog/init.py", line 174, in log_async_response Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: async for row in rows: Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/spinta/commands/write.py", line 181, in push_stream Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: async for data in dstream: Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/spinta/backends/postgresql/commands/changes.py", line 25, in create_changelog_entry Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: async for data in dstream: Sep 27 07:05:58 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/spinta/backends/postgresql/commands/write.py", line 55, in insert

Dar ištrauka: Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "datasets/gov/vrk/rinkeju_demografiniai_duomenys/Zymejimas_pkey" Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: DETAIL: Key (_id)=(47fcfe23-80ad-4576-ac53-5603cb98b613) already exists. Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: [SQL: INSERT INTO "datasets/gov/vrk/rinkeju_demografiniai_duomenys/Zymejimas" (_txn, _created, _id, _revision, vda_prime_key, rinkejo_asm_k, rinkimu_turo_id, faktines_apygardos_id) VALUES (%(_txn)s, TIMEZONE('utc', CURRENT_TIMESTAMP), %(_id)s, %(_revision)s, %(vda_prime_key)s, %(rinkejo_asm_k)s, %(rinkimu_turo_id)s, %(faktines_apygardos_id)s)] Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: [parameters: {'_txn': UUID('24da7300-1f88-4709-b9c9-508c67db4c44'), '_id': UUID('47fcfe23-80ad-4576-ac53-5603cb98b613'), '_revision': '063243e0-156e-42f3-80e4-f626b0db0909', 'vda_prime_key': 'feadbd3a41c2375', 'rinkejo_asm_k': '31e40204fefdc94', 'rinkimu_turo_id': '264', 'faktines_apygardos_id': None}] Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: (Background on this error at: http://sqlalche.me/e/14/gkpj) Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: During handling of the above exception, another exception occurred: Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: Traceback (most recent call last): Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: await app(scope, receive, sender) Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/starlette/routing.py", line 74, in app Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: response = await f(request) Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/spinta/api/init.py", line 272, in homepage Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: return await create_http_response(context, params, request) Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/spinta/utils/response.py", line 230, in create_http_response Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: return await commands.push( Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/spinta/commands/write.py", line 91, in push Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: status_code, response = await _batch_response( Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/spinta/commands/write.py", line 1183, in _batch_response Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: async for data in results: Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/spinta/accesslog/init.py", line 174, in log_async_response Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: async for row in rows: Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/spinta/commands/write.py", line 181, in push_stream Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: async for data in dstream: Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/spinta/backends/postgresql/commands/changes.py", line 25, in create_changelog_entry Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: async for data in dstream: Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: File "/opt/spinta/env/lib/python3.9/site-packages/spinta/backends/postgresql/commands/write.py", line 55, in insert Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: raise exceptions.MultipleErrors(error_list) Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: spinta.exceptions.MultipleErrors: Multiple errors: Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: - Given value already exists. Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: Context: Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: component: spinta.components.Property Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: manifest: default Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: schema: 33370 Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: dataset: datasets/gov/vrk/rinkeju_demografiniai_duomenys Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: model: datasets/gov/vrk/rinkeju_demografiniai_duomenys/Zymejimas Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: entity: Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: property: _id Sep 27 07:17:19 IVPK-ADS-WWW01 gunicorn[94156]: attribute: None

[JuliusLADP]

Reikalingas Justino vertinimas

[JustinasKen]

Na iš pirmo žvligsnio atrodo, kad bando supushinti jau egzistuojančius duomenis ?

į datasets/gov/vrk/rinkeju_demografiniai_duomenys/Zymejimas duomenų rinkinį

[JustinasKen]

geresnis klausymas kodėl, matau, kad šitas įrašas jau buvo įrašytas: https://get.data.gov.lt/datasets/gov/vrk/rinkeju_demografiniai_duomenys/Zymejimas/47fcfe23-80ad-4576-ac53-5603cb98b613/:changes/-10

2024-09-27T10:39:14.362791

Valandas galime ignoruoti (serverio timezone gali neatitikti), bet svarbu pamatyti dieną ir minutes.

bando rašyti: Sep 27 07:17:19

ta pati diena, bet minutės smarkiai skiriasi.

Reiktų patikrinti ar pas juos nėra dublikuojančių duomenų, jei nėra būtų gerai sužinoti ką tai įstaigai rašo push metu.

[JuliusLADP]

Eiga: iš pradžių tiesiog buvo nulūžęs eksportas ir kai jis pasileido iš naujo, bandė vėl tą patį įrašą įkelti. Programuotojas patikino, kad duomenys nesidubliuoja.

Pagarbiai

[JustinasKen]

Yra galimybė pabandyti paleisti sinchronizaciją, jei push state neatitinka realybės.

Prie push komandos pridėti --sync argumentą. Tik patariama, jei nori, pasidaryti push state duomenų bazės kopiją, jei sync komanda blogai suveiktu.

[JuliusLADP]

Ar tai "receptas", kurį reikia taikyti, kai "Daug užklausų į Saugykla baigiasi klaida. Logas - pilnas Django klaidų:"

[JustinasKen]

Na tai galimas sprendimas, jei yra daug dublikavimo klaidų. Visas --sync principas, kad bando iš serverio pasiimti duomenis ir tikrinti ar lokali push state duomenų bazė atitinka realybę, kad tiksliau galėtų atlikti push komandą.

[Grigorjeva11]

@JuliusLADP Laukiame komentaro, ar reikia imtis papildomu veiksmų, ar galime uždaryti uždavinį.