Upgrade to latest wazimap version for Python 2.7 end of life

[cliftonmcintosh]

This project is currently running on Wazimap 0.8.2, which uses Python 2.7. Python 2.7 is reaching end of life at the end of 2019. This project should be upgraded to the latest version of Wazimap, which includes an upgrade to Python 3.

There are breaking changes from 0.8.2 to 1.0.0 of Wazimap that will need to be dealt with. There are also breaking changes from 1.X Wazimap to 2.X Wazimap.

I would recommend a stepped approach by upgrading to 1.X and, from there, to 2.X.

Documentation The Wazimap docs up to version 1.0.0, which documents the breaking changes in 1.0.0: https://wazimap.readthedocs.io/en/latest/version_history.html The Wazimap repo's changelog that includes the update to 2.1.0: https://github.com/OpenUpSA/wazimap/blob/master/CHANGES.rst

[cliftonmcintosh]

I have updated the dependencies to the most recent version for Python 2.7. The version of wazimap that runs on Python 3 has not yet been published to PyPi. The maintainers are currently working on documentation.

https://github.com/OpenUpSA/wazimap/issues/154

[divyamani1]

@cliftonmcintosh Is anyone looking into this? I'm actually working on this and I'm checking the project with wazimap 1.2.1 and the updated requirements.

When updating requirements using pip install -r requirement.txt, django-sass-processor v0.8 is installed which doesn't support Python 2.7, so I've downgraded to 0.7.5 (which could be explicitly defined in requirements.txt, but it's redundant since we'll be upgrading to Python 3).

What should I particularly check to ensure proper functioning after upgrade?

[cliftonmcintosh]

@divyamani1

Thank you for your interest in helping with this issue. There are several things that will be useful to note:

1. There are breaking changes in the upgrade to wazimap 2.1. The upgrade directions in the Wazimap docs will be useful for that. I am not sure what the implications of data table management being done through the Django admin interface are. We would want to try not to expose an admin interface on the production server if possible. If an admin interface is needed, we will need credentials to be kept secure, so you would not commit any credentials as part of the updates.
2. NepalMap on the production and test servers runs in a docker container. The base container defined in the web container Dockerfile will need to be updated.
3. The docker-compose setup that we have works by rebuilding the tables on startup. This allows us to provide updated data with each build. See the start-web bash script. Changes will need to be compatible with this or a credible alternative will need to be offered. You may or may not have to alter the SQL files that are used to populate tables if the table definitions have changed.
4. The best test for verifying the changes will be to run the application in the docker-compose setup, like we do on the servers. The directions for running this on a development machine are in the README.
5. in order to keep the change focused, please only touch things that are directly related to the upgrade. For example, only upgrade packages that are required to be updated for the change to Wazimap 2.1 and Python 3. If you notice other things (like the django-sass-processor), please create separate issues for them. This way we can isolate problems or issues related to the upgrade and keep this change focused.

Thank you for being willing to take on this issue and please let me know if you have any further questions.

[cliftonmcintosh]

@divyamani1

Please note

• The project has already been upgraded to wazimap 1.2.1. The step needed now is upgrading to wazimap 2.1.
• The main branch is not master but dev (for historical reasons that were not good reasons). You will need to submit changes based on that branch, not master.

[divyamani1]

All right, I'll look into it.

[divyamani1]

Wazimap v2.1.0 still uses dict.itervalues() to iterate through the values in dictionary (like FIELD_TABLES) instead of dict.values() as required by Python 3. This is causing problems to upgrade tables. Editing itervalues() to values() shows that the code is working fine. so, Am I missing something?

[cliftonmcintosh]

@divyamani1

I don't know. I have not explored that yet. I would suggest opening an issue on the wazimap project, asking there about it. They are good about responding.

https://github.com/OpenUpSA/wazimap

[cliftonmcintosh]

@divyamani1

I have opened https://github.com/OpenUpSA/wazimap/pull/157 to replace the use of .itervalues() with .values().

What happens if you add six to the dependencies for nepalmap_federal? Will that allow .itervalues() to work?

If not, can you import my branch that has the chance and use it for the wazimap dependency temporarily while you work on upgrading nepalmap_federal?

[cliftonmcintosh]

@divyamani1

If you see the Travis CI output for my wazimap branch (which failed to build until six was added), you'll see six is included in the testing. Please try using six with nepalmap_federal. This will allow you to proceed while with the upgrade without having to wait for a new release of wazimap.

[divyamani1]

@cliftonmcintosh I have six installed on my machine and still, the project refuses to work. I'll proceed with your branch.

[divyamani1]

@cliftonmcintosh While trying to use our current location, the following error is observed. Are we supposed update any json file to new format?

The function get_locations_from_coords (line 286) in wazimap/geo.py expects geometry features in following format:

feature = {'properties': values,
                'shape': values}

while currently it receives:

feature = {'NP': {'properties': values,
                  'shape': values}}

and the above error occurs.

I edited the code in geo.py (line 286-291) as follows and it works fine.

       for feature in features.values():
           code_value = list(feature.keys())[0]
           if feature['code_value']['shape'] and feature['code_value']['shape'].contains(p):
               geo = self.get_geography(feature['code_value']['properties']['code'],
                                        feature['code_value']['properties']['level'],
                                        version)

[cliftonmcintosh]

@divyamani1

I don’t know the answer to your question. Do the upgrade directions in the documentation provide any help?

If not, you might ask a question as an issue in wazimap.

[divyamani1]

With the latest update, most of the problems with Wazimap 2 seems to be fixed and everything is looking good locally. I'm looking into fixing issues with docker. For some reason, I couldn't connect with the postgresql container from nepal map container (port 5432 wasn't open to nepalmap) but I got around it using a quick fix for now (which might not be necessary, I'll look into it in detail). I'll report if any major problem occurs.

[divyamani1]

@cliftonmcintosh I forgot about this. Is anyone working on this? I have upgraded Nepalmap to Python 3.7 and Wazimap 2.1.2 (along with a few dependency upgrades). I have tested the docker builds (dev/stage/prod) and everything seems to be working fine (except for the "use my location" feature, but it doesn't seem to work on the current nepalmap.org as well). Should I submit a PR? Are there more tests I can use to validate if everything is working correctly?

[cliftonmcintosh]

@divyamani1

Hello. I am no longer active on this project, and I am not sure what the appropriate steps are for proceeding. There may be someone in C4N who is working on the project and can provide advice. You might try the C4N Slack for getting advice.

[pratimakandel]

@divyamani1 great, thanks for working on this. yes, can you submit a PR for this issue? I can review it. Since the "use my location" feature is not working on the current nepalmap.org too, we can create a diff issue for that and track/work on it. Not sure, if you've already joined C4N slack channel.. if not you can find the link here. We can further discuss regarding the use my location and the update wazimap issue there.