add field for maps urls to entity model

[pwalsh]

Docs on entities: http://open-budget.readthedocs.org/en/latest/guide/features/entities.html

Source data: https://docs.google.com/spreadsheet/ccc?key=0AoJzAmQXH28mdG5vUUlpT3RPSWF5eUxhb2sxeWhnNVE#gid=2

Task

We recently added the "MAP URL" column to the entities sheet linked above, as we'll be showing a map on the page of each municipality.

So, this task involves:

1. Added a new field for the map url to the entity model: https://github.com/hasadna/omuni-budget/blob/develop/openbudget/apps/entities/models.py - call it map_url

Django has several field types, so look up the Django docs on this.

1. Exporting the above sheet for entities as csv, renaming it 3_entities.csv and replacing the file of the same name in the openbudget/fixtures directory. These files are automatically loaded when the database is created. The script that does this importing is a quick and dirty one, here: https://github.com/hasadna/omuni-budget/blob/develop/openbudget/apps/transport/incoming/importers/initial.py

So, at this step, you'll reset your whole environment with:

python manage.py devstrap -m -t

This is our own commend that wraps up some other django commands - we basically kill the existing db, create a new one, and run some tests.

1. If everything went well, you'll now have a new field on the Entity model, with data in the database for that field.
2. We want to show the map on the muni page. There is no design yet, so you can just write a simple piece of html and put it somewhere on the page. Here is the template for the muni pages:

https://github.com/hasadna/omuni-budget/blob/develop/openbudget/apps/entities/templates/entities/entity_detail.html

You'll need to look at google maps to make a snippet of html that can load these urls to display a map.

[pwalsh]

Kobi, did you get to do any of this?

[kobiluria]

Hi Paul , I hope you are feeling better..

i Got to do some. i add a map field and looked in the field types. must of Wednesday open-budget meeting wed night was to get the dev env working for me and the env of other newbie's to the team.

i need some time to understand how the whole virtual env works. after i shut down on Wednesday , i couldn't get the system to work again on Thursday. the bug was importing open-budget settings and i can't figure it out.

i will work on the import settings env problem and continue on sun.

I do have a few questions :

1.i should manually download the csv from the google data. and rename it . the database script will add the field data from the weird named csv? i will also look into the script file on sun.

1. the Google map URL should just be clipped into the web template not manipulated , i.e , get coordinates for future . for many entities for a future bigger map ?

Thank You.

kobi.

On Fri, Jun 7, 2013 at 10:57 AM, Paul Walsh notifications@github.comwrote:

Kobi, did you get to do any of this?

— Reply to this email directly or view it on GitHubhttps://github.com/hasadna/omuni-budget/issues/172#issuecomment-19093739 .

[pwalsh]

Hi Kobi,

the env problems some of you had were due to PyPi, the Python package server, becoming intermittently unavailable. It happens sometimes. We have updated the docs to take precautions against this, using the --use-mirrors flag to the pip install command - see the updated docs.

As for your questions:

1. Yes. Michael and Adi are working on a set of features to allow us to pull data directly from google docs. Until then, manually export that file as CSV, rename it to 3_entities.csv, and replace the current file of the same name in the codebase. You'll then run:

python manage.py devstrap -m -t

and you'll quickly find out if there is a problem with the new file, or not. I'll try to be avail. on wednesday night via chat if poss.

1. yeah, these are URLs to simple maps where google already knows to highlight the muni region automatically. just need to throw it on the page. You can see this UX mockup of the page:

https://docs.google.com/file/d/0B5DhLTlHzJGdOEFJMUhyYzNZWGM/edit

Which is the direction things are aiming in, but it will be a couple of weeks before we start implementing design there.

[kobiluria]

Hi Paul ,

so o.k , i got my system up and running , and env well installed. managed to debug the InitImporter using PyCharm ( i know its not open etc. but i got used to it in phpstorm in a previous dev ) and see how the data is updated and built up. took a while to understand the fixtures part , and the initial start up- i do think a got the basics.

i think there might me a bug in a special condition in the init importer esp look like a translation issue :

if model == 'entities':

got keywords, need IDs

            item['division'] =

Division.objects.get(name_he=item['division'])

might be a problem since the 'division' is declared in Entiteis.object.get and not in | Division |

i will try to comment out this part for my development and see if the devstrap works.

i will try to finish this task on Wednesday night , might play a little bit with it tonight (studying for a test today and tom. )

kobi.

On Sun, Jun 9, 2013 at 10:48 PM, Paul Walsh notifications@github.comwrote:

Hi Kobi,

the env problems some of you had were due to PyPi, the Python package server, becoming intermittently unavailable. It happens sometimes. We have updated the docs to take precautions against this, using the --use-mirrors flag to the pip install command - see the updated docs.

As for your questions:

1. Yes. Michael and Adi are working on a set of features to allow us to pull data directly from google docs. Until then, manually export that file as CSV, rename it to 3_entities.csv, and replace the current file of the same name in the codebase. You'll then run:

python manage.py devstrap -m -t

and you'll quickly find out if there is a problem with the new file, or not. I'll try to be avail. on wednesday night via chat if poss.

1. yeah, these are URLs to simple maps where google already knows to highlight the muni region automatically. just need to throw it on the page. You can see this UX mockup of the page:

https://docs.google.com/file/d/0B5DhLTlHzJGdOEFJMUhyYzNZWGM/edit

Which is the direction things are aiming in, but it will be a couple of weeks before we start implementing design there.

— Reply to this email directly or view it on GitHubhttps://github.com/hasadna/omuni-budget/issues/172#issuecomment-19172239 .

[pwalsh]

Good to hear.

that doesn't look like a bug to me, but for sure, especially in that InitImporter, nothing is sacred, I wrote it specifically just to get the data in, and at some point, we'd like to clean it up so it can accept different types of data, etc, to load on init. I have a task for cleaning it up: https://github.com/hasadna/omuni-budget/issues/144 so, feel free, if you want to fix something in there, or change, just go for it and make a suggestion.

[pwalsh]

Also Kobi, I just updated the quickstart, you might want to go over it again:

http://open-budget.readthedocs.org/en/latest/quickstart.html

[pwalsh]

Hi Kobi,

Mor told me yesterday you didn't understand why we are not doing this via the Google Maps API directly.

First, you are absolutely free to implement it as you like, as long as you can maintain the following goals:

• Every map must show the borders of the municipality - this is absolutely critical
• If changes to maps are needed in the future, it must be editable by content editors, not programmers.
• If you are going to do it programmatically, please provide one or two tests for the feature

Also, not that the gdata Python client is part of the project dependencies, so I believe you have a Python interface to the Maps API ready to go.

Do you have any other questions? Please let me know if you need anything else, I'm here to help/answer questions.

[kobiluria]

Hi Paul ,

Yes I didn't completely understated, but wanted to investigate myself before I ask a silly question.

I started by fixing a small import bug on my machine at least (the csv files were not a sorted list) I think you fixed this bug in the meanwhile. I added a text field for the url and got them into the db using the devstrap.

The problem was that the map url wasn't enough for rendering a Google map. Google maps need a latitude and longitude.

I wrote a small geocode function in JavaScript and managed to get a nice map for each location using just the municipality name. Then I saw a nice python app on pypi called (django-easy-maps)

Using the name of the municipality and one line of template code I got a nice Google map on each page.

---

Then I noticed that almost all of the pages have the municipality boundaries on their Google map url --> and figured out that was the important piece of the puzzle.

The problem is: Google maps api doesn't give you those polygons. Some programmers think they come from a twitter api. but you just can't get these shapes on a Google maps. http://stackoverflow.com/questions/9688682/how-to-retrieve-a-static-map-showing-city-or-municipality-boundaries

I then tried to regex the url to an embedded html snip which Google supplies  the problem was that not all map_url had all the info an embedded html needed, After embedding a single map using the html Google provided: no boundaries were presented in the html snip.

---

I think we have a few possibilities:

1. Use open maps -> I think google maps just works better! And google API is just great so that’s not a good option , plus we like google.
2. Present an image of the map for each page with a link to the google map url ( we might need to hard code the images..)
3. What I think is a challenging and interesting idea :

Use a creative and exciting method. If we are able to get gis info from either Hebrew U , or open source or from contributors we can update a map table using Google fusion tables, The API for this option is well documented and we will be able to have a much more “open” and reusable code for presenting polygons on Google maps for all Hasdana projects.

Google has a very nice example : https://developers.google.com/fusiontables/docs/samples/adv_fusiontables

We could import these polygons using some of the open source tools for fusion tables: https://developers.google.com/fusiontables/docs/tools

I think this approach will probably consume more time to understand how these tables work but since we work mostly with csv files, and some GIS info is available online for Israel (as opposed to all municipalities info for USA is open source GIS info)

This option also gives the possibility to present budget data on a large map of all of Israel municipalities and just zooming in on each municipality page. plus doing some cool visual budget data for each division etc'

I’m sorry I wasn’t that responsive in the last week I just have a test in optoelectronics devices on Sunday.

What do you think about this issue?

[pwalsh]

Hey Kobi,

Overall, sounds awesome.

I tested myself (on a few examples) that those URLs could work with the embeddable snippet Google provide, and also show the muni boundaries. And, because this is a very minor part of the project at this stage, I considered it good enough.

The boundaries of the munis are critical - we have 255 munis, and we are expecting to show the boundaries on most/all, and if that is a matter of manually grabbing a URL once, that is the solution we need.

Also note that we'll be wanting to color the maps, something like the attached image here.

This is possible through some editor that google provide (designers checked, not me). We need to know that any option you do will allow us to control the map colors.

But, I am really glad with the direction you are going in, so if you can definitely meet our current minimal needs, but also potentially give us a platform to present data on maps, for instance, then I am happy for you to just run with with it!

Let me know if I can help with anything.

[kobiluria]

Thnx . :-)

i will look into it on Wednesday night. it does sound cool. i might try to find a friend to work with me on this issue.

could you show me an embedded html of a map url including the boundaries. since i did try a few on html online editors and could not get any of them for example : http://jsfiddle.net/ksykh/ http://jsfiddle.net/ksykh/embedded/result/

the url does work but once embedded the boundaries are gone.. maybe i was doing something wrong.

[pwalsh]

Nope, you are right, I can't get it now either. I also did a fiddle, I am sure it was working.... but yes, I can't reproduce now.

[kobiluria]

Hey so i manged to import some GIS data and create a Google fusion table using some data i found. I might need some help with matching each area to the muni. basically hard code each area to its real name. but some are already set. these tables are good since you have a really nice API for adding selecting and having a lot of data in them. so that is nice.

I wrote a small javascript page for a example : http://jsbin.com/ekawik/1/

you can click once on any muni or city and get data from my tables.. which will be great for us or any project. you can also edit the HTML for that tag , so we can make a nice one.

I will learn and read the API and see we can make a nice system for exporting and adding to our table for the project.

i added a short geocoding service in the javascript in order to center to Jerusalem but we will be able center to any muni using {entity.name}

I think the options are great but need some work , since we can really use this method for a lot of projects.

here is the current table : https://www.google.com/fusiontables/DataSource?snapid=S1030402l9b9

I going into test mode in the next week so i will probably not be available to code on next Wednesday. :-(

but looks like we are on our way. :-)

[pwalsh]

Hey Kobi:

Man, hats off to you :).

This is an awesome direction.

Note that our entities table has munis, sub-districts, and districts - which are all administrative levels of the government system in Israel. You can also map those.

You are building here a great base for our project, and hopefully for other HaSadna projects.

Nothing much else to say except - excellent work, and just keep going! Let us know if you need anything.

BTW, we are working on a standalone screenshot app, so, if necessary, we could also take screenshots of the maps you output, for use here and/or elsewhere.

Also, you may not know yet, but there will be a hackathon on July19th, with the Open Muni Budgets project a significant portion of it. If you can have this ready by then, not only do we have maps per muni, but, like you said earlier, we have a basis for people to hack on budgets + maps together. too cool.

[idoivri]

Kudos, this looks fantastic!

[pwalsh]

Hey @kobiluria you still around? Are you coming to the Hackathon?

[kobiluria]

Hey Paul Yes!! Still here and coding !! tried to meet the hack deadline I managed to graph and pull the Geo data for 180 munis . and a changed the entity model a bit .

I will send a complete description tonight late . and a pull request for the budget stuff also tonight

I will try to come tom morning or afternoon but because of shabbat its a bit hard for me.

Is there a possibility you can run my code. Before a pull request Like push to my fork and then you can try it?

I will send another email before tom with description don't worry didn't go anywhere :-) On Jul 18, 2013 10:30 PM, "Paul Walsh" notifications@github.com wrote:

Hey @kobiluria https://github.com/kobiluria you still around? Are you coming to the Hackathon?

— Reply to this email directly or view it on GitHubhttps://github.com/hasadna/omuni-budget/issues/172#issuecomment-21207937 .

[kobiluria]

Hey so i updated my fork which has the entity model changes and a new entity csv file which contains all the info i need for connecting to fusion map. and setting the map to the center of the entity. and rebased as of tonight at 2 am.

a small description :

1. in the entity detail template page : a map with only the "this" entity boarders ( i could present all of the map info including all entities but iT would be a shame since we really wanted the page to have only the entity wanted.
2. in the entity list template page : a full map of all entities in the system currently. pressing one of them gives you a more button which should redirect you to the entity detail template of the entity you choose from the map. this only works if the stub url is the same as the entites name therfor only short one word entity work in this fashion.

future work :

1. fix 2 feature : if this is cool and you want this feature. add db connection between entity code --> entity url_stub
2. add Heatmap by budget. i can select the color of each region by a by a compare between all entities .
3. add a map for division -> already exists just didn't know
4. write more docs. ( sorry had about 4 test in the last two weeks. )
5. add feature to search by budget compare keyword.

lots more :-)

I need some help with the 70 other entity which i couldn't find their geodata :

https://docs.google.com/spreadsheet/ccc?key=0AlgVSI9c46EQdHdtLUs2bGIwSUEwc1NGTG5tM1h5NlE#gid=0

I will try to explain tomorrow morning what i need.

here are some samples of maps which i did with the 180 entites i got . ( i like how the whole system works in my fork which is more advanced . but this is an example of the data i have in the system currently)

you can also try click on each entity.

all 180 entities i got their geodata

http://jsbin.com/ikuqov/1/edit

division map :

https://www.google.com/fusiontables/data?docid=1EMWGXIJGxHn2bETrN6255b9_O-5RYFP4xRPhdN0#map:id=3

will you be at the meeting tomorrow morning?

Talk tomorrow

kobi.

On Fri, Jul 19, 2013 at 1:41 AM, kobi Luria kobi@luria.net wrote:

Hey Paul Yes!! Still here and coding !! tried to meet the hack deadline I managed to graph and pull the Geo data for 180 munis . and a changed the entity model a bit .

I will send a complete description tonight late . and a pull request for the budget stuff also tonight

I will try to come tom morning or afternoon but because of shabbat its a bit hard for me.

Is there a possibility you can run my code. Before a pull request Like push to my fork and then you can try it?

I will send another email before tom with description don't worry didn't go anywhere :-) On Jul 18, 2013 10:30 PM, "Paul Walsh" notifications@github.com wrote:

Hey @kobiluria https://github.com/kobiluria you still around? Are you coming to the Hackathon?

— Reply to this email directly or view it on GitHubhttps://github.com/hasadna/omuni-budget/issues/172#issuecomment-21207937 .

[kobiluria]

i forgot to add the link to all the map data merged with entity file.

https://www.google.com/fusiontables/DataSource?docid=1xG5u4i2Qs_PSViDKJlM1vcXOg8nSXRMMlJOhNnA#rows:id=1

On Fri, Jul 19, 2013 at 3:09 AM, kobi Luria kobi@luria.net wrote:

Hey so i updated my fork which has the entity model changes and a new entity csv file which contains all the info i need for connecting to fusion map. and setting the map to the center of the entity. and rebased as of tonight at 2 am.

a small description :

1. in the entity detail template page : a map with only the "this" entity boarders ( i could present all of the map info including all entities but iT would be a shame since we really wanted the page to have only the entity wanted.
2. in the entity list template page : a full map of all entities in the system currently. pressing one of them gives you a more button which should redirect you to the entity detail template of the entity you choose from the map. this only works if the stub url is the same as the entites name therfor only short one word entity work in this fashion.

future work :

1. fix 2 feature : if this is cool and you want this feature. add db connection between entity code --> entity url_stub
2. add Heatmap by budget. i can select the color of each region by a by a compare between all entities .
3. add a map for division -> already exists just didn't know
4. write more docs. ( sorry had about 4 test in the last two weeks. )
5. add feature to search by budget compare keyword.

lots more :-)

I need some help with the 70 other entity which i couldn't find their geodata :

https://docs.google.com/spreadsheet/ccc?key=0AlgVSI9c46EQdHdtLUs2bGIwSUEwc1NGTG5tM1h5NlE#gid=0

I will try to explain tomorrow morning what i need.

here are some samples of maps which i did with the 180 entites i got . ( i like how the whole system works in my fork which is more advanced . but this is an example of the data i have in the system currently)

you can also try click on each entity.

all 180 entities i got their geodata

http://jsbin.com/ikuqov/1/edit

division map :

https://www.google.com/fusiontables/data?docid=1EMWGXIJGxHn2bETrN6255b9_O-5RYFP4xRPhdN0#map:id=3

will you be at the meeting tomorrow morning?

Talk tomorrow

kobi.

On Fri, Jul 19, 2013 at 1:41 AM, kobi Luria kobi@luria.net wrote:

Hey Paul Yes!! Still here and coding !! tried to meet the hack deadline I managed to graph and pull the Geo data for 180 munis . and a changed the entity model a bit .

I will send a complete description tonight late . and a pull request for the budget stuff also tonight

I will try to come tom morning or afternoon but because of shabbat its a bit hard for me.

Is there a possibility you can run my code. Before a pull request Like push to my fork and then you can try it?

I will send another email before tom with description don't worry didn't go anywhere :-) On Jul 18, 2013 10:30 PM, "Paul Walsh" notifications@github.com wrote:

Hey @kobiluria https://github.com/kobiluria you still around? Are you coming to the Hackathon?

— Reply to this email directly or view it on GitHubhttps://github.com/hasadna/omuni-budget/issues/172#issuecomment-21207937 .

[pwalsh]

Hey Kobi,

I'm going to close this ticket now, because the original request is no longer relevant. Perhaps we should start up a new repo for your work, and/or, a new branch of open muni budgets with your work in it. Let me know, when you progress.... I'm here to help.