open-contracting / kingfisher-collect

Downloads OCDS data and stores it on disk
https://kingfisher-collect.readthedocs.io
BSD 3-Clause "New" or "Revised" License
13 stars 12 forks source link

Use Pydantic #1068

Closed jpmckinney closed 5 months ago

jpmckinney commented 5 months ago

Builds on #1066

995

Some differences:

jpmckinney commented 5 months ago

Tests pass and I also ran:

scrapy crawlall --sample 1 --loglevel=ERROR

I added this to the crawall command to skip the ones with authentication and the many broken Digiwhist spiders:

if spider_name.startswith(('openopps', 'paraguay')) or spider_name.endswith('digiwhist'):
    continue
yolile commented 5 months ago

Ah, nice and clear!