opendatadiscovery / odd-collectors

Apache License 2.0
8 stars 9 forks source link

500 error while pulling big data with CKAN adaptor. Bug appear in case of wrong construction of DataEntityList. #20

Closed ValeriyWorld closed 9 months ago

ValeriyWorld commented 10 months ago

Error: Caused by: io.r2dbc.postgresql.ExceptionFactory$PostgresqlDataIntegrityViolationException: insert or update on table "group_entity_relations" violates foreign key constraint "group_entity_relations_fk_entity"

DataEntityList formation (commented variant not working): image

Assumption: data was too big for being combined into one chunk request, so it was separated into multiple chunks, and data was not able to be grouped properly.

Possible chunk bottleneck in aiohttp library in class ClientSession: read_bufsize: int = 2**16

ValeriyWorld commented 9 months ago

closed with #21