search

datamade / nyc-council-councilmatic

NYC Council version of Councilmatic
MIT License
7 stars 3 forks source link

IntegrityError: (psycopg2.IntegrityError) duplicate key value violates unique constraint "councilmatic_core_bill_slug_key" #147

Closed hancush closed 5 years ago

hancush commented 5 years ago

DETAIL: Key (slug)=(int-0969-2018) already exists.

https://sentry.io/datamade/nyc-council-councilmatic/issues/750193666/?query=is%3Aunresolved%20import_data

indeed, there are two bills in the ocd api with an identifier that would be slugified as "int-0969-2018": https://ocd.datamade.us/bills/?identifier__endswith=0969-2018

one in them is now in legistar (and the ocd api/councilmatic database) as 0959-2018.

we should delete the erroneous 0969-2018 version from the ocd api. (it was never imported to councilmatic because of this integrityerror.)

hancush commented 5 years ago

this continues to recur, though i'm not sure why – when i set up the import locally (because the raw and new tables are cleared every 15 minutes by the cron imports on the server), i did not see two versions of this bill in the raw table, only the correct one. i also don't see an outdated file cached in the downloads directory on the server. so it's a bit perplexing where the duplicate is coming from, for now...

hancush commented 5 years ago

update!

DID YOU KNOW: the staging and production instances of new york city council councilmatic are deployed on different servers??? staging is on staging, production is on the councilmatic server. 💫

this was confusing because exceptions from both staging and production are logged to sentry in a single project, without indicating what instance they come from.

also, the old nyc councilmatic site (nyc-councilmatic) is running the councilmatic server (i.e., shows up in supervisor as a running process), and there are several nyc-related directories in the datamade user's home dir:

finally, i completely forgot about the import data logs. that would have been a clue that i'd been barking up the wrong tree all along.

so, i cleared the downloads/new_york/bills folder on the staging site. let's see if these errors come back.

in the meantime, at minimum, it may be a good idea to log in sentry which instance the errors are coming from.

hancush commented 5 years ago

yay, it looks like removing the outdated bills solved the problem.

reginafcompton commented 5 years ago

@hancush - I ran import_data --update_since='2018-10-10' on the staging server without fail. I'll kick-off a rebuilding of the index, this afternoon.

reginafcompton commented 5 years ago

Same for the production server!