Table preprocessing - Githubissues

gazetteerhk / census_explorer

Explore Hong Kong's neighborhoods through visualizations of census data

http://gazetteer.hk

MIT License

42 stars 12 forks source link

Table preprocessing #17

Open hupili opened 10 years ago

hupili commented 10 years ago

Redirected from: https://github.com/hxu/hk_census_explorer/pull/14#discussion-diff-9334029

Pointers:

hupili commented 10 years ago

Note. After filtering out empty value cells, only two cases of "none" row names:

table 5, a17-a103, those values in brackets.
h177: the row name is "None"..

Those are filtered out directly at current version for simplicity. Add back later if one find them useful.

hxu commented 10 years ago

Two questions regarding the data munging:

What happened to the rows in the Monthly Income table that exclude foreign domestic helpers? These were the values that were in parenthesis under the main table values
If you look at the data browser in the columns, there's a lot of redundant male / female / both columns. All of those columns represent the same type of data, but because they were in different tables, they have different identifiers. Do you think it is worthwhile to collapse all of them into one column value? i.e. male instead of a1_male, d1_male, g1_male, etc.

hupili commented 10 years ago

The bracket values of Monthly Income table that exclude foreign domestic helpers is noted in the comments above. This one needs special treatment. I don't know a proper place yet. It is put lower priority at my side. For visualization purpose, I don't think they will produce too much different plots.

hupili commented 10 years ago

The equivalence of a1_male, d1_male, g1_male are subject to human interpretation. I prefer to distinguish them in the identifier level. They won't affect the users because translation maps will give Male in final visualization.

For App development purpose, I think we usually don't want to mix multiple tables. In this case, only one cellid_male will be there. Then the difference is only a matter of typing. What's your use case if those are unified?