allow download of model data

[ricardogsilva]

From the requirements:

Improve the mode of downloading time series on point and map (Download data). Give the ability to change fields without going back to the map. Allow to change the fields "Indicator", "Model and scenario", "Period" without the need to go back to the map. Implement a massive download of data, allowing the ability to download multiple fields, i.e. multiple Netcdfs, at the same time.

Note that, in spite of the dynamic download requests, we likely need to have some limit in place (and potentially some caching strategy) to avoid having the user making very large data download requests and risk degrading performance of the system.

[ricardogsilva]

My initial plan was to gather all the requested inputs, ask THREDDS NCSS for the relevant data (which would potentially imply doing a temporal and spatial subset) and then collect the generated results into a zip file which would get returned to the user. However, upon further thinking about this, it does not seem like such a good idea.

Packing the files inside a zip would introduce some issues:

• There would have to be some sort of limit to how many things the user would be able to request, in order to prevent abusing the system's processing and IO capabilities. This is even more of a problem in this case, as the actual processing is to be done by the THREDDS NCSS, which is a service that is not part of the ARPAV-PPCV system. - these things are always hard to implement and tune;
• The UI would need to enforce this limiting scheme somehow too, potentially making it confusing for the user;
• There would be less opportunities for caching data.

We should instead strive for having:

• A clear and easy UI for the downloads
• Since preparing the data is an intensive task that hits the THREDDS server, cache as much as possible

---

I think a better strategy for the design of model data downloads is:

• The backend only accepts requests for downloading a single file each time - this makes caching easier;
• Let the user specify the spatial bounds they are interested in, if any - additionally, in order to leverage caching, implement a spatial subsetting strategy that works on fixed intervals, using a 10 km grid, for example;
• Let the user specify the temporal bounds they are interested in, if any - additionally, in order to leverage caching implement a temporal subsetting strategy that works with fixed intervals, for example only allow choosing years that are multiple of five;
• Instead of a single download button, the UI presents a list of links with all possible model variables, so that the user can easily request multiple downloads;
• The URL for each download link contains the spatial and temporal subsetting parameters;
• Upon receiving a download request, the backend forwards it to THREDDS NCSS, then captures the generated result and caches it before returning to the user;
• Next time that a user requests the same NetCDF file with the same spatial and temporal subsetting the backend does not need to ask THREDDS NCSS to generate it, but rather sends directly from its cache
• The backend has an additional process which takes care of cleaning the cache every x amount of days

[ricardogsilva]

In the below mock-up we keep some configuration parameters in dynamic HTML controls, allowing the user to modify them without having to navigate away from the download section. Other parameters are not selectable and we include a list of possible download links and let the user select which they want.

This seems to provide a nice balance of presenting a list with quick access to downloads and not make it excessively big (in order to not disorientate the user).

We just need to select which parameters are more important to be set by the user and which ones can be shown in the links. geospatial extent and temporal extent will always need to be dynamically set by the user, but in addition we may put (as in the mock-up below):

• climatological variable (TAS, PR, etc)
• aggregation period (30 years, yearly) - this could use a radio instead of a combo box
• forecast model (ensemble, etc)
• scenario (RPC 2.6, etc)

@sirmmo @francbartoli your thoughts on this?