fititnt / spatial-data-maching

https://sdm.etica.ai/v/
GNU Affero General Public License v3.0
0 stars 0 forks source link

Implement initialization of "data import (into memory)" step via URL parameters #4

Closed fititnt closed 1 month ago

fititnt commented 1 month ago

Currently, we have 3 steps represented by tabs: "Data import", "Real time filter" and "Export".

The "Real time filter" implies user interaction (so maybe is something not worth be exposed on URL parameters or as initialization configuration). However, "Data import" might be something that user may try several times, or be something user clicks to load external data.

I believe the idea here is simply add option to initialize the first step totally by URL parameters. This also may need less documentation (or the documentation be focused in the "Real time filter".

About the last tab, "Export", maybe if "Data import" already is defined by URL parameters, then some hint also by URL parameters to force download the result could be done.

fititnt commented 1 month ago

This feature already is working. For now it only accepts a direct list of remote data files. Maybe will not implement every future feature from the configuration file (which also soon will also be possible to load from URL parameters. Some comments for future (this behave differently from files loaded from the disk into the app).

  1. The parser from CSV/TSVs seems to complain something about data range "papaparse.min.js:7 Refused to get unsafe header "Content-Range". But this maybe is based on where the remote file is hosted. However, the data seems to work, only this warning.
  2. The GeoJSON text sequences (because use a streaming process more optimized than read all at once) was not rewriten to work with remote fetch requests. Maybe not hard to fix, but I did not done today. The workaround in this case (which actually is not a bad idea, because big files is better already user cache locally) is just download the file.

HOW IT WORKS: load data files with data[]=

The syntax to configure files is the data[]=(URL) and the data[]= can be repeated multiple times. So, for load 3 files (in sequential order) would be as example the following at the end of the URL:

?data[]=http://example.org/file-1.geojson&data[]=http://example.org/file-1.geojson

One example with the beta version with some test files would be for example this

(later, maybe will change) load the conf file with conf= or conf[]=

This requires #5 to be finished. However for sure will have some form to allow all the instructions (or at least the list of files) be in a configuration file (which either could be passed by URL or by upload).

The presence of [] means a hint that actually is a list of values, not only one. So maybe there will be more than one because I'm unsure if it would be a good idea to allow users to "merge" multiple configuration files.

One rationale for reusing parts of configuration files is if they start to contain not only a list of files , but maybe also dictionaries or complex instructions that are reusable.