protocols scraper should run fast + not fail due to missing person

hasadna / knesset-data-django

Django-specific APis and modules for working with Knesset (Israeli parliament) data

3 stars 5 forks source link

protocols scraper should run fast + not fail due to missing person #14

Closed OriHoch closed 6 years ago

OriHoch commented 7 years ago

temporarily fixed by removing the speaker_id (f818759fbc2830680bf445dd4aaa6d5cdd9db246 / d78bcb8a31d50d267ab214ee39dafe0f0c692fb1)

Reproduction

scrape a package with new meetings on production
e.g. ubuntu@ip-10-209-139-40:~$./oknesset_manage.sh download_knesset_datapackage --scrape --url=https://s3.amazonaws.com/knesset-data/datapackage_last_120_days_2017-07-07.zip

expected

run in a reasonable time
not have error

actual

take a very long time to scrape protocols (minutes per meeting)
after a few meetings it ended with error an error which I failed to copy here, but it was something about not finding related speaker_id in persons

alonisser commented 7 years ago

@OriHoch I can't run it in production since you removed the scraping, can you scroll up and paste the exception you got?

OriHoch commented 7 years ago

@alonisser I neglected to copy the error

The problem is that you are saving the member id in the speaker_id, but the speaker_id is defined as relation to persons, not members

ProtocolPart model

    speaker = models.ForeignKey('persons.Person', blank=True, null=True, related_name='protocol_parts')

I guess you might miss this error if you happen to have the same member_id as an existing person_id (but it might be a different person)

OriHoch commented 7 years ago

Here are steps to reproduce the same error (on production DB): ./manage.py shell_plus

part = ProtocolPart.objects.get(pk=6268439)
mk = Member.objects.get(pk=924)
person = Person.objects.get(mk=mk)
part.speaker_id = mk.id
part.save()

IntegrityError: insert or update on table "committees_protocolpart" violates foreign key constraint "speaker_id_refs_id_6023887a"
DETAIL:  Key (speaker_id)=(924) is not present in table "persons_person".

part.speaker_id = person.id
part.save()

alonisser commented 7 years ago

LOL I can swear then used to be logic to enforce having the same id, somewhere in the listener creating persons. THE REAL PROBLEM IS THAT NOW I CANT FIND ANY OF THE PERSON CREATING LOGIC. Any ideas where it disappeared

alonisser commented 7 years ago

Ok, found it. Hate the Damm Django signals antipattern , can be hidden anywhere

OriHoch commented 6 years ago

thanks, added to draft release v1.0.7

need to publish this draft release (after some testing maybe..) and update it in open knesset (let me know if you want me to do it)

alonisser commented 6 years ago

Hmm. I actually added it to 1.0.6 and released it.. :( I tested it on my machine and looked legit

OriHoch commented 6 years ago

right, sorry.. missed it