search

submarcos / django-vectortiles

Mapbox VectorTiles for django, with PostGIS or Python
https://django-vectortiles.readthedocs.io
MIT License
38 stars 11 forks source link

Streaming responses #37

Open lbam opened 1 year ago

lbam commented 1 year ago

In this line...

https://github.com/submarcos/django-vectortiles/blob/86f00879a27a0c5aaf1eebad6448ca27993a3d13/vectortiles/postgis/mixins.py#L58

the entire MVT response from PostGIS is converted to a bytes object. That means it has to be in memory twice, once as a memoryview and once as a bytes. I'm wondering if this is necessary, since Django performs the same conversion. If the conversion were not there, a caller could intercept the memoryview and make a StreamingHttpResponse out of it, so that it only needs to be in memory once.

StefanBrand commented 1 year ago

Good point, it looks as if the tobytes call could possibly be removed. I quickly ran the tests with this change and they run fine. Are you interested in contributing a PR?

lbam commented 1 year ago

I could discuss a PR with my employer, but I'd first need to have #36 resolved.

lbam commented 1 year ago

I've also done some more research and found that this is not the bottleneck, memory-wise. The tile is sent over the wire by PostgreSQL as ASCII hex. psycopg reads the entire tile into memory in that format, then decodes it into a separate buffer, so at that point the memory use is 3x the size of the tile. Fixing this issue would therefore not reduce the peak memory use. For that, the ST_AsMVT query would somehow have to chunk the resulting bytea in PostgreSQL so this library can receive it piecemeal and sent the chunks to the client in a streaming response.

submarcos commented 1 year ago

I could discuss a PR with my employer, but I'd first need to have #36 resolved.

This point is resolved. I'm working on a 1.0 refactor and release. If you contribute a PR I can implement it in 0.2.1 and 1.0 versions