cliftonmcintosh commented 6 years ago

Once the data for population has been integrated as part of #7 , present the total population with a breakdown by gender.

This includes:

defining a table in the tables.py file based on the table created for #7.
creating a profile for demographics, like the one in the pre-federal NepalMap. This should include a total population and a breakdown by gender.
integrating that profile into the base profiles.py
creating a view for demographics that shows total population and breakdown by gender. See https://github.com/Code4Nepal/nepalmap_app/blob/dev/wazimap_np/templates/profile/profile_detail.html and https://github.com/Code4Nepal/nepalmap_app/blob/dev/wazimap_np/templates/profile/sections/demographics.html for an example of how this has been done in the pre-federal NepalMap.

Data sets for this are here: https://github.com/Code4Nepal/data/tree/master/Federal%20Data/753%20Local%20Unit%20Population%20and%20HouseHold This only has provinces 1 and 2 right now, though.

ravinepal commented 6 years ago

@Beegrekokto, @soneeka, @sunitagajurel, @theonlyNischal - pls let us know if one of you are interested in leading on this issue? thank you.

ravinepal commented 6 years ago

dear @Beegrekokto, @soneeka, @sunitagajurel, @theonlyNischal. i hope all is well. i know some of you had exams. i wanted to follow up to see if one of you are interested in leading on this issue?

cliftonmcintosh commented 6 years ago

16 needs to be addressed before this issue can be worked on.

amitness commented 6 years ago

@crackjack @cliftonmcintosh

The instructions to load data into postgres database seems to be out of date. There is no simpletables folder in sql directory. Can someone guide me with instructions to load the database dump?

amitness commented 6 years ago

Also, I'm getting an error regarding WhiteNoise when running the app with python manage.py runserver. @crackjack Did you get any such error when setting up?

And had we used Python 2 or 3 for this app?

cliftonmcintosh commented 6 years ago

@amitness

There are no simpletables yet. You should still be able to load the database tables. You can either run that command or skip it. Either way you should be able to proceed to the next steps successfully.

We could remove that from the README, but we would need to remember to add it back in as soon as we included any simpletables.

cliftonmcintosh commented 6 years ago

@amitness

Wazimap is not compatible with python 3. I do not get that exception for WhiteNoise with python manage.py runserver, and the homepage loads for me. Of course, since we have no more data, that is as far as it goes!

nepalmap_federal

amitness commented 6 years ago

@cliftonmcintosh Thanks. The problem was due to wazimap dependancy on a library called whitenoise. When I installed from our requirements.txt, the latest version of whitenoise was installed which was incompatible with how wazimap was using that library. I was able to fix it by uninstalling it with pip uninstall whitenoise and downgrading to a lower version of the library pip install whitenoise==3.3.1 .

amitness commented 6 years ago

@cliftonmcintosh @ravinepal I converted the data from local structures to the format we need for wazimap and when mapping the geography.sql codes to that, I have found some inconsitencies for some VDCs.

For example, for district 'Nawalpur', we have mappings to only 7 local levels in geography.sql

But, in the csv data repository, we have 10 VDCs for 'Nawalpur'.

Can I know the source from which the geography.sql mapping for province -> district -> local level was created? I need to cross-verify the data to proceed. The data shows that we have missing mappings.

I was thinking of recreating the geography.sql file from the data that was scraped but there is some interesting data such as: Province 5: Kapilbastu: Mayadevi Gaunpalika.csv Nawalpur: Mayadevi Gaunpalika.csv Rupandehi: Mayadevi Gaunpalika.csv

All 3 districts have same name VDC with different populate readings. Can I trust the scraped data as the ground truth and recreate a mapping from it for geography.sql?

cliftonmcintosh commented 6 years ago

Amit,

I am traveling and don’t have access to a computer to try to investigate the answer to your questions. I may not be able to get to a response until tomorrow, but I will do my best to respond promptly. On Sun, Oct 28, 2018 at 12:53 PM Amit Chaudhary notifications@github.com wrote:

@cliftonmcintosh https://github.com/cliftonmcintosh @ravinepal https://github.com/ravinepal I converted the data from local structures to the format we need for wazimap and when mapping the geography.sql codes to that, I have found some inconsitencies for some VDCs.

For example, for district 'Nawalpur', we have mappings to only 7 local levels in geography.sql

[image: image] https://user-images.githubusercontent.com/8587189/47618940-e566c280-db00-11e8-8b9c-5d7a88111d77.png

But, in the csv data repository, we have 10 VDCs for 'Nawalpur'.

[image: image] https://user-images.githubusercontent.com/8587189/47618946-fadbec80-db00-11e8-8bfb-36f0dfcd56dd.png

Can I know the source from which the geography.sql mapping for province -> district -> local level was created? I need to cross-verify the data to proceed. The data shows that we have missing mappings.

I was thinking of recreating the geography.sql file from the data that was scraped but there is some interesting data such as: Province 5: Kapilbastu: Mayadevi Gaunpalika.csv Nawalpur: Mayadevi Gaunpalika.csv Rupandehi: Mayadevi Gaunpalika.csv

All 3 districts have same name VDC with different populate readings. Can I trust the scraped data as the ground truth and recreate a mapping from it for geography.sql?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Code4Nepal/nepalmap_federal/issues/11#issuecomment-433722135, or mute the thread https://github.com/notifications/unsubscribe-auth/ADpbbEUjoiccPHQ7MzApuHXVYG_44f5xks5upeEkgaJpZM4VUf9Q .

openrijal commented 6 years ago

@amitness the geography.sql was crrated out oglf geojson and topojson files from different sources, plus a few other sources.

Nawalpur is one of the districts which was renamed from "nawalparasi". There might be descripecancies, if you find a single source of Truth, we can use that.

amitness commented 6 years ago

@cliftonmcintosh It's working now. I've skipped the local levels with discrepencies for now. We can discuss and solve it in #19. Please review the PR.

I also had found a new problem. The URL pattern used by Wazimap to detect the geocode from the URL was like this:

https://nepalmap.org/profiles/vdc-1425-bhadgau-sinawari/

Here vdc-1425 was splitted into 'vdc' and '1425' as geo_level and geo_code respectively.

But, for federal data, we have geo_code in format such as 'pro-06'.

http://127.0.0.1:8000/profiles/province-pro-6-province-no-6/

So, wazimap was treating 'province' as geo_level and 'pro' as the geo_code instead of 'pro-6' causing errors. This was hardcoded in wazimap app itself. So the urls for provinces, district and local level were not working

To fix it, I created a new urls.py, imported all urls from wazimap, replaced this specific pattern with our new pattern and set our urls as the default root conf url in settings.

amitness commented 6 years ago

@crackjack Can you help me out with this?

urlpatterns.append(
  url(
        regex   = '^profiles/(?P<geography_id>\w+-\w+-\w+)(-(?P<slug>[\w-]+))?/$',
        view    = cache_page(STANDARD_CACHE_TIME)(GeographyDetailView.as_view()),
        kwargs  = {},
        name    = 'geography_detail_country',
    )
)

This is the regex I replaced the existing Wazimap URL with. With this regex, it matches province, district and local levels.

http://127.0.0.1:8000/profiles/local-loc-4010-jhapa/
http://127.0.0.1:8000/profiles/district-dis-28-jhapa/

But, the below URL is not matched, since geo_level is 'country' and geo_code turns out to be 'NP-nepal' instead of 'NP' that we need.

http://127.0.0.1:8000/profiles/country-NP-nepal/

This is the original URL pattern that was present in Wazimap and it's hardcoded.

    url(
        regex   = '^profiles/(?P<geography_id>\w+-\w+)(-(?P<slug>[\w-]+))?/$',
        view    = cache_page(STANDARD_CACHE_TIME)(GeographyDetailView.as_view()),
        kwargs  = {},
        name    = 'geography_detail',
    ),

This matches country level but don't work with province, local and district level geo-codes that we are using.

Any way to modify the regex to match both of them?

cliftonmcintosh commented 6 years ago

Would it be better to have us remove the dash in all of the geo-codes, so that they become things like "loc4010" and "dis28"? Or even use just the numeric part, so that they become "4010" and "28"? What would the effort be in making one of those changes?

On Mon, Oct 29, 2018 at 12:08 PM Amit Chaudhary notifications@github.com wrote:

@crackjack https://github.com/crackjack Can you help me out with this?

urlpatterns.append( url( regex = '^profiles/(?P\w+-\w+-\w+)(-(?P[\w-]+))?/$', view = cache_page(STANDARD_CACHE_TIME)(GeographyDetailView.as_view()), kwargs = {}, name = 'geography_detail_country', ) )

This is the regex I replaced the existing Wazimap URL with. With this regex, it matches province, district and local levels.

http://127.0.0.1:8000/profiles/local-loc-4010-jhapa/http://127.0.0.1:8000/profiles/district-dis-28-jhapa/

But, the below URL is not matched, since geo_level is 'country' and geo_code turns out to be 'NP-nepal' instead of 'NP' that we need.

http://127.0.0.1:8000/profiles/country-NP-nepal/

This is the original URL pattern that was present in Wazimap and it's hardcoded.
url(
    regex   = '^profiles/(?P<geography_id>\w+-\w+)(-(?P<slug>[\w-]+))?/$',
    view    = cache_page(STANDARD_CACHE_TIME)(GeographyDetailView.as_view()),
    kwargs  = {},
    name    = 'geography_detail',
),
This matches country level but don't work with province, local and district level geo-codes that we are using.

Any way to modify the regex to match both of them?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Code4Nepal/nepalmap_federal/issues/11#issuecomment-433993397, or mute the thread https://github.com/notifications/unsubscribe-auth/ADpbbLmgY8CBjTOncXtLZxjiQyOu5gdaks5upzYZgaJpZM4VUf9Q .

amitness commented 6 years ago

@cliftonmcintosh Previously, I tried removing the dash in the geocodes. The URLs worked but the map for that region failed to load. So I had to revert that. I think it's because the geocodes in geojson was still the old one. I'll try renaming both and see if it fixes the problem.

cliftonmcintosh commented 6 years ago

Yes, they have to match the codes in the geo files, so they would need to be modified in those as well

ravinepal commented 5 years ago

hi @amitness - just checking in to see how we can help here. we have received interests from new volunteers and i can request them to help.

cliftonmcintosh commented 5 years ago

@ravinepal

The dashes in the geocodes has been corrected by https://github.com/Code4Nepal/nepalmap_federal/pull/21.

From what I can tell, the most pressing issue is the data discrepancies described in #19. Those must be resolved in order for us to be able to present the basic population data.

amitness commented 5 years ago

@ravinepal It would be great to get them involved. The major roadblock currently is #19.

cliftonmcintosh commented 5 years ago

With @nikeshbalami's data updates and his guidance provided in a comment on #19, we should now have the ability to account for the missing data. @amitness has some tooling for processing data in a Jupyter notebook. See this comment: https://github.com/Code4Nepal/nepalmap_federal/issues/19#issuecomment-437935324.

@ravinepal if one of the volunteers wants to try to process the missing data, they are welcome to do so. This would include:

including data for the missing localities that have now all been identified in the data set.
including zero values for all national parks, wildlife reserves and development districts.
updating the aggregated totals for all levels above the local level so that they include the new figures. For example, if a missing locality is added to Banke district, then the totals for that district needs to be updated to included the locality's figures. The totals for Province 5 would need to be updated to include the locality's figures, and the totals for Nepal would need to be updated to include the locality's figures.

ravinepal commented 5 years ago

This has been resolved, I believe. @cliftonmcintosh - fine to close this?

CodeforNepal / nepalmap_federal

Present total population and population by gender #11

16 needs to be addressed before this issue can be worked on.