Closed GoulartNogueira closed 2 years ago
Going further, lots of the non-match are due to the wrong Letter Case. If I lowercase both columns:
cidades['name_muni'] = cidades['name_muni'].str.lower().str.strip()
escolas['name_muni'] = escolas['name_muni'].str.lower().str.strip()
Then, the no-matching goes to 0.06%
Here is the full list of name_muni on school dataset with no correspondence in municipality dataset:
escolas[escolas.code_muni.isna()][['abbrev_state','name_muni']].value_counts()
abbrev_state | name_muni | count |
---|---|---|
BA | santa teresinha | 39 |
ES | atílio vivacqua | 16 |
RN | augusto severo | 16 |
CE | ererê | 15 |
PB | quixabá | 15 |
MG | são thomé das letras | 9 |
SC | grão pará | 9 |
TO | fortaleza do tabocão | 9 |
SE | amparo de são francisco | 7 |
MG | dona eusébia | 5 |
MG | pingo d'água | 4 |
Here are some matches I found manually:
abbrev_state | muni (school dataset) | muni (municipality dataset) |
---|---|---|
BA | Santa Teresinha | Santa Terezinha |
ES | Atílio Vivacqua | Atílio Vivácqua |
RN | Augusto Severo | - |
CE | Ererê | Ereré |
PB | Quixabá | Quixaba |
MG | São Thomé Das Letras | São Tomé Das Letras |
SC | Grão Pará | Grão-Pará |
TO | Fortaleza Do Tabocão | - |
SE | Amparo De São Francisco | Amparo Do São Francisco |
MG | Dona Eusébia | Dona Euzébia |
MG | Pingo D'Água | Pingo-D'Água |
Most names have a single-character error, like an accent, hyphen or S-Z changed.
Hi @GoulartNogueira . Thank you for bringing this issue to our attention. The municipality data comes from IBGE, while the schools data come from INEP. This is to say two things:
Thank you for the quick answer! I tried brute force checking all schools versus all cities, but:
It takes more than one hour (using the simplified geometry).
Also, some schools fall into 2 cities and others fall into none.
So I guess I could use the state + city name and then just double check using spacial data.
Anyways, after I make a trustful merge, can I somehow upload the new muni_code to the schools table, to help other users? I want to contribute to the community.
The fact that "some schools fall into 2 cities and others fall into none" is a geocoding problem in the original data set at Inep. I know they are continuously improving this geocoging, so hopefully this won't be an issue in the next update of the data.
There are also a few complicated situations because the muni_name
in the data set refers to the municipal government officially responsible for the school, but the school location can be very close to the border and imprecise.
Trying to get the
code_muni
for each school, I've found some problems:Finally, I suggest to add the _codemuni for each school, as it's a more trustworthy index, comparing to _namemuni and state.