chore(pipeline): optimize monenfant scraping - Githubissues

gip-inclusion / data-inclusion

data·inclusion aggrège les données de l'insertion sociale et professionnelle

https://api.data.inclusion.beta.gouv.fr/api/v0/docs

MIT License

6 stars 1 forks source link

chore(pipeline): optimize monenfant scraping #241

Closed vmttn closed 3 months ago

vmttn commented 4 months ago

The initial extraction results give : 75k scrapped results for 7500 uniques results ultimately.

This is inefficient and thus the extraction is unnecessary long.

To fix this :

drop the radius from 30km to 5km by default
drop the radius to 1km in cities with arrondissements
increase the number of cities scrapped to 2000 to keep the global coverage

In addition,

fix the tests
clean up the staging model (disambiguate columns names, use sql style cast)
increase the number of retries for extraction tasks from 2 to 15