edwardchalstrey1 commented 6 months ago

Cache shapes ideas

See use of memcache on cache-everything (and below)
Use of indexedDB on copilot-speedup branch

Ideas 16th Jan 2024

[ ] Refactor code, including not making an entire JS object for each shapes list/dict from Django, loading them sequentially instead
- [x] Start by seeing how fast it is for just the polities from a specific year (e.g. 0) with no basemap country/polity shapes
- [ ] Try to implement async view which can be run with and ASGI server e.g. uvicorn
  - Asynchronous views are still experimental
  - Seshat already had an asgi.py
[ ] Still may want to use caching after first load for each shape. Perhaps just use localStorage.setItem rather than anything new, as was done for colours.
- This has memory limitations, but perhaps if the stored shapes are separate objects that's not an issue?

edwardchalstrey1 commented 6 months ago

Updated on the cache-everything branch, but doesn't seem to make page loading any faster

edwardchalstrey1 commented 6 months ago

Notes 14th December

I can't save the whole shape data with memcached because it is too large (needs to be < 1MB)
- Can we use separate cache variables instead?
May be worth taking a different approach to speedup - I've assumed caching is what's needed, but worth starting another new branch and just seeing what GPT suggests?
It may be worth using Ajax instead (perhaps make a new branch), see suggestion below from ChatGPT

edwardchalstrey1 commented 6 months ago

Certainly! Here are the fully updated versions of your Django view and template code with the changes to implement asynchronous loading of shapes using JavaScript and AJAX.

Django View (views.py):

from django.http import JsonResponse
from django.shortcuts import render
from django.db import connection
from django.contrib.gis.geos import GEOSGeometry
from .models import VideoShapefile

def get_provinces(selected_base_map_gadm='province', simplification_tolerance=0.01):
    # ... existing code

def get_shapes(request):
    shapes = VideoShapefile.objects.all()

    shapes_data = []
    for shape in shapes:
        shapes_data.append({
            'startYear': shape.start_year,
            'endYear': shape.end_year,
            'polityStartYear': shape.polity_start_year,
            'polityEndYear': shape.polity_end_year,
            'geometry': shape.geom.geojson,
            'name': shape.name,
            'colour': shape.colour,
            'seshatId': shape.seshat_id,
        })

    return JsonResponse({'shapes': shapes_data})

def map_view(request):
    # ... existing code

    return render(request, 'core/spatial_map.html', content)

Django URLs (urls.py):

from django.urls import path
from .views import map_view, get_provinces, get_shapes

urlpatterns = [
    path('spatial_map/', map_view, name='spatial_map'),
    path('get_provinces/', get_provinces, name='get_provinces'),
    path('get_shapes/', get_shapes, name='get_shapes'),
    # ... other paths
]
Updated JavaScript in Template (spatial_map.html):
html
Copy code
{% extends "core/seshat-base.html" %}
{% load leaflet_tags %}

<head>
    {% leaflet_js %}
    {% leaflet_css %}
</head>

<body>
    {% block content %}
    <script>
        document.addEventListener("DOMContentLoaded", function () {
            // Fetch shapes asynchronously
            fetch('/get_shapes/')
                .then(response => response.json())
                .then(data => {
                    var shapesData = data.shapes;

                    // Plot polities after fetching shapes
                    plotPolities();
                })
                .catch(error => console.error('Error fetching shapes:', error));
        });

        // ... existing JavaScript code

    </script>
    {% endblock %}
</body>

These changes incorporate the asynchronous loading of shapes using JavaScript and AJAX. The get_shapes endpoint is added to the Django view to provide the shapes data in JSON format, and the template is updated to fetch shapes asynchronously when the page loads. Make sure to adjust the code based on your specific requirements and additional functionalities.

edwardchalstrey1 commented 6 months ago

https://docs.djangoproject.com/en/5.0/topics/cache/#setting-up-the-cache

edwardchalstrey1 commented 5 months ago

On m Mac:

32s to load
No difference if not converting the django view objects to JS dicts
22s to load just polities, no background country/provinces
2s without loading any shapes from django

edwardchalstrey1 / seshat

Make the map view load faster #33

Cache shapes ideas

Ideas 16th Jan 2024

Notes 14th December