Explore geojson convertor => cityIO module?

[RELNO]

There are two main options:

1. geojson convertor becomes a real-time cityIO module that spits the grid geojson
2. since this script should only run once per table, we can (?) avoid having this as a real-time module running in the cloud. Instead, this can be well documented and used when deploying a new table

[andredaa]

Question concerning 2: Right now including information per cell such as type and rotation. This information is passed from cityio. If we run this only once at table deploy, this information would not be needed, right? The only purpose of this script would be to serve cell_id and coordinates per cell.

The front end would then need then need to update the properties of each cell after each user interaction. Lets say the user changes the building type in cell 3. Then the front-end logic would have to append/update the properties of cell 3 in the geojson.

[RELNO]

If we run this only once at table deploy, this information would not be needed, right? The only purpose of this script would be to serve cell_id and coordinates per cell.

Yes, exactly -- the purpose of your script is to create a geojson of the entire gridded area (whatever we agree upon, project based) and ship it as post req to a cityio field. So in practice your output might be used once, as long as the server keeps your field alive.

Can you please add @doorleyr to this conv.? (add him to the issue) -- Ronan did similar work apparently for another project and you guys can share notes. See: https://cityio.media.mit.edu/api/table/mocho/site_geojson

[doorleyr]

@andredaa I was indeed working on something similar as part of my mobility simualtion repo. You can find the relevant script here: https://github.com/CityScope/CS_Mobility_Service/blob/master/python/simulation/abm.py I've tested it for multiple sites and it works fine.

Basically, when the simulation starts, it calls createGrid() which checks the basic spatial information of the grid and then computes the top-left coordinate of each grid cell. It also creates a road network around all the grid cells and joins this to the larger road network (from OSM).

There is also another function, get_grid_geojson() which takes the coordinates of every cell as well as the land use information and creates a geojson of all the cells with the property "usage".

Looks like there's a lot of crossover with what you've done so if there's any functionality here which your module doesn't yet have let's combine efforts in your repo.

[andredaa]

Hej Ronan I would prefer to use my script - as it is already standalone, well documented and custom build to the CityScope context. All necessary parameters can be set in the config. I would suggest to add the OSM road network in a combined repo? Best Regards Andre

[doorleyr]

Hi Andre, My geojson creator is also a standalone module built for CityScope context. You can find it here: https://github.com/doorleyr/grid_geojson

Of course i'm biased but I think this grid_geojson repo should be the basis for a CS module for the following reasons:

• I tried running your using the example in the repo and it tool about 10-15 mins to run. grid_geojson can run for the same example instantaneously.
• grid_geojson is much simpler. The entire module is 75 lines of python 3 code and there are no external libraries required. city_io_to_geojson has multiple scripts and requires 11 libraries which makes it difficult to maintain. It's also in python 2 which will be fully deprecated next year.
• I looked into the code and the way you calculate the coordinates is assuming that lat, lon coordinates can be treated as planar spatial coordinates. In fact, lat, lon are angles and in order to calculate a coordinate at some distance and angle from another coordinate, you must use the Haversine formula. The difference is small but accumulates with each grid cell away from the origin.

Having said that, @RELNO mentioned that city_io_to_geojson has some features which grid_geojson does not. I suggest that we identify those features and incorporate them into grid_geojson. That can then become a Cityscope repo.

[RELNO]

@andredaa @doorleyr thanks both -- @andredaa, I remember you've implemented a cityIO auto-send of the created network, this might be one feature to plug into @doorleyr's code. Another feature (which might be in both, not sure) is the pre-definition of cells features (such as colors, height) etc.

[andredaa]

I looked at @doorleyr 's feedback:

• long processing time: fixed. This was a dependency issue. Locally I had pyproj 1.9.6 installed which reprojected much faster than the pyproj2.2.1 version. I adapted the code to work with the new pyproj and it computes within 5 seconds even for large grids (44x78cells) now. Since this get's only executed once upon table creation - I think response time in ms. is not necessary.

• grid_geojson is much simpler: It is in python2 due to a downwards compatitiblily issue for a project here at CSL. I could easily port it to python3. I updated the requirements.txt and there is only pyproj (same as grid_geojson) and shapely and the configparser as dependencies. I think the code is well structured and flexible to feature requests like https://github.com/andredaa/city_io_to_geojson/issues/2#issuecomment-518585641 or the addition of the margins.

• suspected mistreating of lat, lon coordinates:

1. I asked our cartographer to manually create a shp file with the edge points of the entire grid in GIS. My grid matches those points by a offset of max. 0.5meters.
2. Using shapely to calculate the area of all squares shows that the all squares have exactly the same area.
3. All the edges of all squares exactly match with my script.

I find @doorleyr 's code hard to interprete for others and difficult to adapt for new features like adding the margins or predefinition of different features (colors, height, ..) per cell. Also I generated a test grid with @doorleyr 's code and looked at it in QGIS:

• the genereated grid does not match the reference points generated by our cartographer (more than 33meters off)
• the corners of each cell do not match

@doorleyr : maybe I did something wrong with your script on https://github.com/doorleyr/grid_geojson . Could you generate a grid for these specs of the grasbrook table: ? TABLE_ROTATION = 326° CELL_SIZE = 16 ORIGIN_LONGITUDE = 10.00677491086256 ORIGIN_LATITUDE = 53.53789434597681 TABLE_ROWS = 44 TABLE_COLS = 78

You find the reference grid of our cartographer here: https://github.com/andredaa/city_io_to_geojson/tree/testing/resulting_jsons/perfect_gis_grid_frank

[andredaa]

I quickly adapted the code to work in python3 on this branch: https://github.com/andredaa/city_io_to_geojson/tree/testing (no type hinting or anything yet) Also I put a file with @doorleyr 's script and the reference points from the cartographer on the branch. In case you want to play around a bit.

[doorleyr]

@andredaa the reason for the slight difference in our results is simple. You use the following line to compute the position of each cell: corner_x = self.get_origin().x + distance * math.sin(math.radians(bearing)) This assumes that the bearing or angle is specified on city_io with respect to the axes of the projected coordinate system. If your cartographer used the same assumption, they would get the same result. The x axis of a projected coordinate system will not usually line up with the WGS84 parallel because of the curvature of the earth. As a mentioned before, the method I used is not the same. My method is assuming that the angle is with respect to the WGS84 parallel. This is an intentional decision because this is our base coordinate system and CityScope users should not have to know what projections will be used at the backend or to calculate their desired rotation angle in that coordinate system. This is why I had to use the Haversine method to calculate the corners of the grid.

As for the complexity, it's hard for either of us to objectively say how easy our own code is to understand. Your project is indeed well-structured but in my opinion, it's overkill for such a simple task. Creating multiple hierarchical classes can be helpful to break down a complex project but there is a readability cost associated with having to switch between multiple files and understand many methods while trying to follow the logic. Again, mine is 75 lines and new features can easily be added as needed with additional methods. As for height color etc., they can be specified as properties. See the example.

As for processing times, I already use my module for both static and real-time use-cases and ms response time is essential for the latter.

[andredaa]

I had discussed this with our cartographer and he told me he would set the longitude of the project side as meridian to avoid these problems. I will try to pull him into this discussion tomorrow, when he is back.

[RELNO]

Thanks @andredaa and @doorleyr for the deepest geojson conv. I've followed in some time 🚀

[andredaa]

thank you @RELNO :p ...the nitty gritty of grid making ... So, we discussed this with cartographers and geoinformatics here. Turns out there is no perfect way to do things. You either get inacurrarcies on distances or on angles. I would vote for keeping the distances true.

We tested the outcomes of the different scripts. It turns out that @doorleyr 's first script (with projection) and this repo produce grids with the same dimensions and true length's (each grid cell is 16meter). Our angles are different, as for the reason that @doorleyr explained above. While our geoinformatics' approach would be to specify origin and angle already in local projection - I understand that this might seem odd for the user. So we could go with the angle relative to the WGS84 system. In terms of distances, the new script without projection generates results in which the cell sizes are slightly off.

If I run this repo with the angle respective to the WGS84 , as calculated by @doorleyr , both scripts produce the same results :) (centimeters difference).

I would suggest to take this repo - while taking the angle respective to WGS84 - because it generates a grid in which all the edges of the cells exactly match, without overlap. Overlapping borders could lead to problems in the calcuation models. ( https://github.com/andredaa/city_io_to_geojson/blob/testing/example_results/ronan_border.png screenshot of a result produced with ronan's script)

Best Andre

[andredaa]

I don't think we can/have to be right on the centimeter. But if half a degree of difference accounts for more than 20meters offset over the entire grid, then we should know which angle we are talking. Thanking for pointing that out @doorleyr - I would have never thought about the different respective axis without this discussion. I just think it's important we all use the same grid - and we all know what space is covered by the project side :)

[doorleyr]

Ok great, thanks for exploring further. I think we're in agreement. So just to make sure I understand, the plan is to use your script but to calculate the angle using the method from the initial commit of my project? If so, I'm good with that.

[RELNO]

Considering recent CityScoPy implementation of geojson-grid, this is resolved.