Insee 2016 - Githubissues

anthill / open-moulinette

Scripts to clean Open-Data.

MIT License

40 stars 20 forks source link

Insee 2016 #58

Closed garaud closed 6 years ago

garaud commented 6 years ago

Makefile updated

garaud commented 6 years ago

I've some troubles in mk_data.py.

A pdb.set_trace() when reading "revenu par unité de consomation"
An assertion with this traceback

Traceback (most recent call last):
  File "mk_data.py", line 226, in <module>
    compare_geo(data, revenu_uc, debug=True)
  File "open-moulinette/insee/comparison.py", line 56, in compare_geo
    compare_var(tab1, tab2, var)
  File "open-moulinette/insee/comparison.py", line 32, in compare_var
    assert max(tab1[var].value_counts()) == 1
AssertionError

when reading "les diplomes 2011"

armgilles commented 6 years ago

Still on WIP on your PR.

Some important points :

Region's code with new files are from 2016 style (new region code)
Hard to to keep reference of Iris / Code insee on more than 6 years of data (some cities move or fusion). The way to go is have the most up-to-date reference. So the output on Iris, City code Insee, Region and Departement should be on the most up-to-date data (2016 is this case). Some geo features name (Iris name, Citie names) changes. We keep the last one (most up-to-date)

With this kind of rules, we are allowed to aggregate features on Code Insee, Departement and Region with no lost data.

armgilles commented 6 years ago

You can pull my branch.

I update documentation files 2016.
Create quality Iris indicator based on number of lack of information (nan_count & iris_quality).

Seems good to me to merge

vallettea commented 6 years ago

thanks guys !