Further efficiency increases

[edwardchalstrey1]

Branches

• adjust-loading just does some of the below, see in progress below

• 159 actually has significant speedups by other means than the below, simply reducing for loopage etc

TODO 6th June

• [x] Finish #159
• Finish adjust-loading
  • [x] Hide loading indicators judiciously
  • [x] If the slider moves, run map_view_one_year again
    • can we update this such that if the displayYear != enteredYear whilst the world_map_all fetch is in progress, we exit that fetch to run the map_view_one_year again before running the world_map_all fetch again
      • In progress - adjust-loading sort of does this, but it doesn't quite work as expected, keep fiddling and see if it could work, bit don't spend long and otherwise
        • [x] remove this idea and merge progress on this branch from before b3c4732a2bd69776fe7ce288d8fef836c5fbe92d into #159

TODO

• [ ] Write structured doc on this idea including the below
• [ ] Time things
  • [ ] Model -> view
  • [x] View completion #159
  • [ ] View -> JS
• [ ] Should we sequentially update seshat_id_page_id and allCapitalsInfo too?
• [ ] Set the initial polityBorderWeight in Python, not JS, to avoid looping though dict before page load
• [ ] Is it faster on a GPU machine?
• [ ] Can we load points faster than shapes?
• [ ] Could we store the results of the full map view as a temporary JSON file which gets loaded if it exists already instead of using the view? Edit: perhaps try this on a separate branch from the stuff above
  • If so, this might be what we want to use parquet/arrow or something fast for
• [x] Speed up assign_variables_to_shapes Update: this has already been sped up via refactoring so further speed increases may not be needed
  • [ ] Alternatively load these as a separate thing to the shapes and suppress at first that's pointless because the issue is running the function, it shouldn't slow down the Python=>JS transition - yes but you could have a separate view that runs this and updates the shapes?
  • [ ] Try just not doing it for the first year and suppress the variable changer until all shapes loaded
    • Problem is that the shapes loaded for the initial year don't have the colour info, so we'll need to make sure that gets added in
  • [ ] Speed up categorical variables function

  • 154

• [ ] Time the loading of the non-simplified shapes on the VM, is it any slower?

[edwardchalstrey1]

Bounding box loading idea

What if the initial map view was zoomed in and we load the shapes in the viewed bounding box first (first one year, then across all years), then allow zooming out after it has loaded

[edwardchalstrey1]

Could we use GeoArrow in Django?

My Q:

Would it be better to/ is it possible to somehow use shape data directly from file, rather than the db. Perhaps the logic in the view should be moved to the data loading command script, creating a new file. Could the new file be in more efficient "GeoArrow" instead of GeoJSON?

Copilot answer:

Apache Arrow (GeoArrow) is a binary format and is not directly readable in a HTML template. You would need to convert it to a format that can be read in the template, such as GeoJSON. Remember, this approach is not recommended for larger files, as it can lead to performance issues. It's generally better to use a view to handle the file and pass only the necessary data to the template.

For using GeoJSON, basically you would go via a view as we are