Cells and Border for real-scale world-map?

[addu390]

Hi 👋🏼 do you have cells.csv and border.csv handly, for a real-sized world map, in other words, a cartogram with size being the scaling parameter instead of the population?

I plan to extend this project to generate the cells and border algorithmically. I would appreciate any tips and recommendations 😇

[mattdzugan]

Hello @addu390

This repo only contains data for generating population-based cartograms sorry about that.

Unfortunately a lot of manual-effort goes into making it "look right" so it's tricky to expand for other variables.

I can't quite understand exactly what you're looking for though. Are you looking for a layout where each country is 1 square? Or if you're looking for a layout where each country is sized to its normal size.... Forgive the silly question, but isn't that just a "normal map"? 😅

Going to close the issue since this repo is population-only. But I'm happy to help you try and find what you're looking for

[addu390]

Hi @mattdzugan Thank you for the response.

The goal is to generate a population-based world map algorithmically.

Here's what I had in mind:

• Taking an example, let's say there are 15000 cells in a size-scaled world map, i.e., each country has a specific number of cells representing the total area of the country such that the value of all cells is the same (Normal Map).
• Now, taking the population as the measure instead of size changes the number of cells for every country (total number of cells remains to be 15000), thereby bringing the need to increase/decrease the number of cells.
• Lastly, increasing/decreasing the number of cells in each country while retaining the shape generates a new world map that is relatively scaled—also known as Diffusion Method (DIF), where cells move from one country to another until a balanced distribution is reached (A country cannot grow into the sea unless another country compensates for it).

[mattdzugan]

@addu390

Got it. That description helps a lot thanks.

I haven't tried anything like that but it's a neat idea.

The first thing that jumps to my mind, for your "step 1" would be something like this

https://sedac.ciesin.columbia.edu/data/set/gpw-v4-national-identifier-grid-rev11

This grid is pretty high resolution so I suspect you might want/need to down sample a bit. But it's essentially a square by square grid that tells you which pixel maps to which country/water

You should be able to start with that and "diffuse" from there.

[mattdzugan]

Good luck! And if you come up with any good outputs let me know!

[addu390]

Thanks again. This might be a very basic question, how do I derive the cells and borders from this? I downloaded the files. Is it the .shp and .tif files that I have to convert? (For the first step - the end result is to generate cells.csv and borders.csv like in your codebase).

[mattdzugan]

There should be an ASCII file that you can download (extension is probably .asc or .txt)

Sorry I'm not at my computer at the moment to check.

If you open in a text editor, you'll see that it's just a giant matrix of numbers almost like an excel spreadsheet.

Then all you'll have to do is just write a small script (just a couple FOR loops) to convert it from the grid format to a format where you label each row and column with its corresponding index.

Sounds complicated but if you can find that text file that is just the big matrix, it should make sense.

[addu390]

Thank you so much, makes a lot more sense now. It's the .asc file.

[mattdzugan]

Happy to help,

That should help you get something similar to the cells.csv file

Then, you can use that to create the borders

In fact, once you have the cells you could steal some code from here which takes the cells and computes the borders https://github.com/mattdzugan/World-Population-Cartogram/blob/master/src/computeBoundaries.py

[addu390]

So far, what I have derived is: the dataset I downloaded has 180 x 360 number of cells, mapping each item (ignoring the 32767, which probably represents the ocean); the XY coordinates are the row-column pairs and the cell value is the country-code? (that's what I presumed).

Sweet, I'll try the code to generate the border after generating the cells.csv

[addu390]

Forked your repo

• cells.csv generated from grid.asc
• Script for cells.csv generation.

I'm trying to figure out what's happening with the computeBoundaries.py file.

[addu390]

Hi, do you mind sharing the python source code for generating the different maps in the GIF of the README?

Also, this conversation pretty much took down hours of work to minutes. Thank you so much.

[mattdzugan]

Unfortunately I didn't do those in python

I did them in R here https://github.com/mattdzugan/World-Population-Cartogram/blob/master/src/demo.R

[addu390]

Perfect! I got the cells.csv and borders.csv generation done, now I'm left with cell density calculation followed by diffusion.

[mattdzugan]

Nice @addu390

FYI that same site that had the country grids also has population

https://sedac.ciesin.columbia.edu/data/set/gpw-v4-population-count-rev11

https://sedac.ciesin.columbia.edu/data/set/gpw-v4-population-density-rev11

[addu390]

I was just checking this, looks like, this one has the population density and counts as the cell value and not the country-code. Unlike: https://sedac.ciesin.columbia.edu/data/set/gpw-v4-national-identifier-grid-rev11

But since the resolution is the same, I'm trying to figure out how I can use it.

[mattdzugan]

Yep that was my thinking.

Although I suppose maybe all you really need is just the single numerical population for each country?

[addu390]

For the diffusion method, yes. I will have to create a mapping of country-code to the number of cells taking size as a metric. Then another mapping takes the population as a metric.

Post which, moving cells such that between countries is the tricky part, to do it while retaining the shape.

Other methods: https://www2.cs.arizona.edu/~mjalam/cartogram/main.pdf

[addu390]

Also, how did you manage to scale the map? was it manual? - If yes, how did you manage to ensure a country looks like a country even after scaling it?

[mattdzugan]

All totally manual.

As I mention in the readme, it's based off this map which was created 100% manually https://ourworldindata.org/world-population-cartogram

[addu390]

Ahh! that must have been meticulous. Also, there are not many open source projects solving this use case.

[addu390]

Hi again! For the cartogram that you have constructed, do you by any chance have the population dataset based on which this cartogram was constructed (i.e., the population count of each country)? While it does say 2018, I figured it's better not to make assumptions and co-relate with the original population dataset.

[mattdzugan]

@addu390 i didn't do any of that work. You can read the citations on this page https://ourworldindata.org/world-population-cartogram

[addu390]

Got it, thank you.

I was trying to transform a size-scaled map into a population map, which gave very ugly results, it seemed like it was almost impossible to retain the shape and required very high resolution (at least 600 cells in height). So, I figured I'll try using the population-scaled world map as a reference to illustrate the map for successive years.

[mattdzugan]

Yeah makes sense.

I don't want to discourage you, you might get something to work. But i've tried a few different algos (and so has the academic literature), and at the end of the day, nothing seems to be better than "some manual love".

So my reason for creating this repo was to basically take somebody else's "really good manual work" - and make it available programmatically.

[addu390]

I wouldn't disagree. In fact, the way I have planned this project - is to algorithmically generate a population-scaled world map and then plot a grid where one can manually retweak to make it look better and save the result as a CSV; this way, plotting the world map for successive years would be easier.

[mattdzugan]

nice idea!

you might take some inspiration from this project https://pitchinteractiveinc.github.io/tilegrams/

[addu390]

Woah! that's a cool reference, exactly what I was talking about. Thanks you :)

[addu390]

Hey! Tilegrams is not maintained anymore and do not handle higher resolutions. So I built a lightweight version here: https://www.pyblog.xyz/population-cartogram/

It's generated algorithmically; you can play around with the years 1950 to 2100 and the hex radius.