AtlasOfLivingAustralia / image-service

Image repository and tiling services
https://images.ala.org.au
0 stars 17 forks source link

exportDataset endpoint is slow to send first bytes #158

Closed sbearcsiro closed 3 years ago

sbearcsiro commented 3 years ago

This endpoint should start streaming results instantly, instead it seems to wait until the entire result set has been buffered on the postgres server before beginning to write the response to the client. This initial waiting period for a large dataset can cause HTTP clients such as the pipelines ImageSync to time out.

Initial investigation suggests that the issue is using a stored procedure to return tuples. Because the stored procedures require an input parameter they're not good view candidates so the queries themselves may need to be moved into the application. The function could instead return a cursor but this would complicate the calling logic as the postgres JDBC driver needs additional steps to use the cursor with a fetch size.