Refactor Seshat variable loading on world map for speed

[edwardchalstrey1]

Role

Developer

Desired Feature

Opening the world map, the variables dropdown is available faster

Benefit

Avoid long wait time for variables to be fully loaded

Acceptance Criteria

All data variables used to colour the World map should be loaded in under one minute

Dependencies

No response

Technical Notes

• Commit 2045a98 on the AWS EC2 set up at CSH (seshatdata.com) it currently loads in 47s on Turing wifi ~85mbps download and upload
• However, one of the RAs mentioned that on her connection this took 290s with a speed of 12.70 Mbps
• If I want to make it 5X faster, that would mean getting the loading time down to 9.4s

TODO

Need to figure out if the time the variables functions take to run, or the size of the data in Mb is causing the problem. Almost certainly it will be the latter, otherwise it would also be slower on faster internet connections or when running locally.

• [x] Find out the size of the shapes data being passed via the view function to the front end before and after the variables are added
• Could one way to refactor this be that there is a new view function that gets called each time the variables dropdown is changed, and this gets a data dict linking each shape id to the variable info, which updates shapesData in the world map template? This would be an enormous refactor, see the alternative below that has resulted in a significant speed up

[edwardchalstrey1]

Time before changes implemented (commit 3f173e7).

shapesData	Mb	Python time (local)	Template time (local)	Python time (Turing VM)	Template time (Turing VM)	Template time (CSH VM)
Load of all shapes	111.69	0.52	6.10	3.72	20.42	34.43, 122.74, 32.52
Load of all shapes with vars	187.89	5.87	15.29	8.70	31.72	70.05, 182.29, 120.71

As of commit 60f2ba:

shapesData	Mb	Python time (local)	Template time (local)	Python time (Turing VM)	Template time (Turing VM)	Template time (CSH VM)
Load of all shapes	111.68	0.54	6.01	3.67	20.12	65.85
Load of all shapes with vars	80.12	5.03	9.06	6.63	12.94	39.60

Where VMs are connected to on internet speed of: 13Mbps download (note: in reality this varies a lot)

This suggests the following:

1. Locally, loading the vars view takes about 3.5X the time of loading without vars
2. Once you get to the site, the python time is however irrelevant since the vars view is only 2X the time, suggesting the main thing here is just the time it takes to load the shapes all again
3. So the problems are that the loading 111-188 mb is slow and that we are doing so twice

TODO:

• [x] Double check that Python speed is not an issue on VM by checking the Turing VM
• [x] Make it so we don't load the shapes all over again in the vars view
  • Could assign_variables_to_shapes and the categorical func too work with the "shapes" sans geometries? Then once this gets passed through to the template, merge the shapes dict that includes variables with the shapes dict that includes geometries
  • For that to work, you'd still need to load or somehow get from before, a version of "shapes" that does not include the geometries.
• Consider if there is anything else we can do to reduce the Mb of the shapes data I tried using the GZipMiddleware but this seemed to make it slower not faster

[edwardchalstrey1]

Even after merging #97 this can still take almost 3 minutes