High resource usage

[frafra]

I am experiencing a high resource usage: I needed to allocate 4 GB of RAM for showing ~1250 points on the map, ~20 seconds to be loaded, and 132 MB transferred to fetch the JSON data (compressed to 38 thanks to gzip compression over HTTP).

That is because my tags.json includes a multiple choice selector with ~2000 possible values.

Some possible solutions:

• Use JSONL instead of JSON (just stream every single note as a single JSON file)
• Use a different API endpoint when loading data for the map, which contain less data, and then add a note-specific endpoint to retrieve details when needed
• Store multiple choices in a different table which can be referred to as an ID
• Avoid storing the multiple choices entirely, as these are used only as client side suggestion and validation system

I am using GSS 3.2 from https://hub.docker.com/r/frafra/gss.

Related: https://github.com/geopaparazzi/smash/issues/129

[frafra]

Analysis

There are two main problems on the server side. Addre

SMASH stores data in a peculiar way: each entry has a copy of the form that generated it. This amplifies an existing issue: when the map loads, every note is sent to the client, including the whole form. There is no need to preload such large amount of information into the server memory nor to send it to the user.

High memory usage is also linked to GSS using JSON instead of JSONL to return a list of element. https://github.com/geopaparazzi/GSS/blob/674270629be66439cc2454b7aa47fc7f86bbca1f/flutter_server_backbone/src/main/java/com/hydrologis/kukuratus/gss/GssDatabaseUtilities.java#L183-L188

Proposal

Adding two functions to GSS:

1. An alternative getNotes, which returns only the information needed to display the pins on the map (latitude, longitude, icon, label), using JSONL preferably
2. A new function to get the form values when a pin is clicked

The old function would then be deprecated and removed in a later version of GSS, making the memory usage on the server more predictable.

@moovida What is your opinion on that?

[moovida]

Hi @frafra , what you write makes good sense. I as living in the assumption that the notes already where passed with the minimum necessary information, so I will first have to check back into that code to be sure and give a statement :-)