Implement datasets tab of the plugin dock

[ricardogsilva]

The plugin dock has a Datasets tab. This is meant to be the primary place where the user manages and interacts with Trends.Earth data.

As depicted by the above mockup, the datasets tab can be split into three areas:

1. An actionable list of known datasets.

   This list shall be scrollable, in order to accomodate a large number of entries.

   Each entry in the list represents a known dataset. It shall feature the following:

   • Dataset name
   • Creation date
   • Source of the dataset
   • Current state (available|being generated|downloading|unavailable)
   • UI controls to manage the dataset: load, details, delete

2. Some UI controls to operate on the list as a whole

   These allow filtering and ordering the list and also performing a manual refresh.

   In this context, refreshing means performing both:

   • A scan of the base_dir in order to discover any new datasets
   • A poll of the remote server in order to get updates on datasets that are being generated

3. Additional UI controls to import new datasets

   These shall allow the user to introduce new datasets to Trends.Earth by either:

   • Downloading some sample data from the Trends.Earth servers
   • Importing local datasets

known datasets

Known datasets are those whose metadata can be extracted by inspection of the Trends.Earth base_dir (as configured in the settings dialogue).

These datasets are generated by either:

1. Running one of the available Trends.Earth algorithms;
2. Downloading from the Trends.Earth sample data repositories
3. Importing from some (local) location

Each known dataset is described by an auxiliary .json file that carries all relevant metadata about it. These json files must be located inside the base dir in order for Trends.Earth to be able to find the respective dataset.

Datasets can be located everywhere on the user's machine, just as long as their respective .json file is under the Trends.Earth base dir. This means that inside the json file there must be a property that informs the plugin of the full path to the dataset. The current implementation features a file property, which we can continue using, but with a full path instead of just a filename.

Trends.Earth base dir

The structure of the base dir shall be:

> {trends-earth-base-dir}/
|   v downloaded-sample-datasets/
|   |    dataset_1.json
|   |    dataset_2.json
|   |    ...
|   |    dataset_n.json
|   v imported-datasets/
|   |    dataset_1.json
|   |    dataset_2.json
|   |    ...
|   |   dataset_n.json
|   v in-transit/
|   |   dataset_1.json
|   |   dataset_2.json
|   |   ...
|   |   dataset_n.json
|   v outputs/
|   |   v {algorithm-group-1}/
|   |   |   v {algorithm-name1}/
|   |   |   |   v {execution-date-1}/
|   |   |   |   |   dataset_1.json
|   |   |   |   |   dataset_2.json
|   |   |   |   |   ...
|   |   |   |   |   dataset_n.json
|   |   |   |   > {execution-date-2}/
|   |   |   |   > ...
|   |   |   |   > {execution-date-n}/
|   |   |   > {algorithm-name-2}/
|   |   |   > ...
|   |   |   > {algorithm-name-n}/
|   |   > {algorithm-group-2}/
|   |   > ...
|   |   > {algorithm-group-n}/

• downloaded-sample-datasets - This directory is used to put all json files and all datasets that are downloaded from Trends.Earth data servers
• imported-datasets - This directory is used to put all json files of datasets that are imported. It may optionally also be used to copy the imported files (but we'll likely provide a UI control for the user to decide if the dataset shall be copied or not)
• in-transit-datasets - This directory is used to put all json files of datasets that are still being generated on a remote server. The contents of this directory are therefore volatile.
• outputs - All datasets and json files that are generated by a Trends.Earth algorithm shall be placed under this directory. It is further discretized according to each algorithm's group and name and execution date (as in year, month, day)

Objective

Implement an initial version of the datasets tab that has the following:

• dataset list, with existing datasets being scanned from the base dir, as mentioned above. for now it is OK to only add:

  • dataset name
  • creation date
  • source
  • Button to load the dataset onto QGIS canvas

  We shall add the remaining features later. Polling the remote server is alos not in scope for the current issue, we shall implement it later

• dataset import section, with two buttons (Import and download sample data), as shown in the mockup. These buttons shall reuse the existing dialogues

• list controls section. Implement only the refresh button for now. We will be adding list sorting and filtering later

[azvoleff]

@ricardogsilva - I worry a bit about this:

Datasets can be located everywhere on the user's machine, just as long as their respective .json file is under the Trends.Earth base dir. This means that inside the json file there must be a property that informs the plugin of the full path to the dataset. The current implementation features a file property, which we can continue using, but with a full path instead of just a filename.

That would mean that if a user for example gets dataset and imports it into trends.earth, it would lead to generation of a .json being generated for that dataset under the Trends.Earth base dir. That makes sense. That json would then point to the location of the relevant tif if it were elsewhere on their machine, correct? But then if the user moved that tif the file location would no longer be known to the plugin, so they would have to re-import it. For a trends.earth generated file this wouldn't be big deal, as each tif already has its own json file with information on the dataset type and algorithm used to generate it. But for other types of data a user might import (land cover, for example) import can take some time due to the need for a user to assign classes to the codes within the file, perhaps resample it, etc.

Long way of saying - we'll need a way for users to link datasets within the "datasets" tab back to their respective tif files (or vrt file if the file is tiled) if the data file is moved from the location specified in the json under the trends.earth main directory.

[luipir]

@ricardogsilva - I worry a bit about this:

Datasets can be located everywhere on the user's machine, just as long as their respective .json file is under the Trends.Earth base dir. This means that inside the json file there must be a property that informs the plugin of the full path to the dataset. The current implementation features a file property, which we can continue using, but with a full path instead of just a filename.

That would mean that if a user for example gets dataset and imports it into trends.earth, it would lead to generation of a .json being generated for that dataset under the Trends.Earth base dir. That makes sense. That json would then point to the location of the relevant tif if it were elsewhere on their machine, correct? But then if the user moved that tif the file location would no longer be known to the plugin, so they would have to re-import it. For a trends.earth generated file this wouldn't be big deal, as each tif already has its own json file with information on the dataset type and algorithm used to generate it. But for other types of data a user might import (land cover, for example) import can take some time due to the need for a user to assign classes to the codes within the file, perhaps resample it, etc.

Long way of saying - we'll need a way for users to link datasets within the "datasets" tab back to their respective tif files (or vrt file if the file is tiled) if the data file is moved from the location specified in the json under the trends.earth main directory.

a multi file solution has not solution about consistency! nor using json, vrt, absolute or relative paths... the two way to reduce this problems are:

1) using a a DB (a geopackage container) ti maintain images... but do not remove the problem to import/export 2) embed trends.earth json as HEX string in a TIFF custom tag so the image would remain always linked. Drawback is tiff modification (equivalent to a import)

obviously remain the solution to live with a suboptimal solution with some drawback that is maintain a trends.earth data repo/folder-tree and link external data with wrt of json or whatever.

[luipir]

@ricardogsilva it't not completely clear to me the possible values for Source of a dataset. e.g. what kind of data type... I'm trying to create a model and would be better to create it more generic as possible and well typed. Possible values for source a a Algorithm datatype or something that says that the alg is generated from 'Default data'?... can you explain?

[ricardogsilva]

@luipir a dataset's source refers to what originated it. It is what is shown in the above mockup in the Generated by part of the dataset list. It can be one of:

• The identifier of the algorithm (i.e. the same that is shown in the algorithms tab as being the name of the algorithm) that created the dataset. Example: Land Productivity (sub-indicator1) Algorithm
• Default Trends.Earth data when the dataset has simply been downloaded from the remote Trends.Earth sample data server. This shall be a fixed value, like Downloaded from sample dataset server (or something similar to this)
• Imported from the user's local filesystem. This shall be a fixed value, like Imported from local files (or something similar to this)

[luipir]

@ricardogsilva @Samweli what about this first draft of the list?

I added a progress bar and a graphic element (to be shown alternatively basing on status)

[luipir]

I contionued to use a tree model instead of a list view to have more flexibility in case we need to add some sub-element to the Dataset item. BTW it is represented as a list view

[ricardogsilva]

Looking good @luipir !

I'm not sure what that Generated by: ????? means, but I guess this is work in progress, right? Would also be nice to have some icons in the buttons instead of text, but I think that can come later on, once we get the basics working. Keep it up :smile_cat:

[luipir]

@ricardogsilva "???" is a placeholder waiting for your answer in https://github.com/ConservationInternational/trends.earth/issues/371#issuecomment-804826913 BTW this demostrate that the value is dinamyc. About icons... I would completly remove any button string substituting with icons to reduce localization problems,.

[luipir]

a dataset's source refers to what originated it

so it's a string... not necessary to pass the entire AlgorigthmDescriptor instance

[luipir]

Better... I used qgis icons (copied into plugin icon folder to avoid dependency). BTW We should have a style directive. I'm not the best person to choose stylish icons and UX :)

[luipir]

This is a first draft of integration

[ricardogsilva]

Looking good, nice work! A few remarks:

1. That white area on the left looks a bit weird. Maybe it is an artifact of the tree widget? If so, maybe replacing with a list widget might make it go away?
2. Each individual dataset seems to have too much margin space
3. Can't see by this screenshot, is this a scrollable widget for when there are more datasets available?
4. The action icons need some tweaking:
   1. They should appear below the text, not to the side of it
   2. The Delete icon should be the last one and we should probably move it to the far right and the others to the far left (with a spacer) in order to make it harder for the user to accidentally press delete. We should also show an additional modal dialogue to ask for confirmation when the user presses delete
   3. As for the icons themselves:
      1. Since Trends.Earth deals with raster data, I think I'd use the usual Add Raster layer icon instead of the generic plus icon
      2. The details icon would be better with the generic information icon, as used also in the QGIS layer properties Information section
5. The filter line edit should also have some placeholder text, in a similar spirit to the QGIS Processing Toolbox
6. I'm not too fond of the icons being used for both the Import and the Download raw data buttons. The Import one has a cloud, which seems to imply that the import consists in downloading some resource from the web, which is not the case. The Download raw data seems to bee too generic and not really convey the concept of a download.

I am happy if we work on these additional remarks as further issues in the future. They do not need to be included in scope for this issue. Unless you feel like it is faster to just go ahead and do them now. I'll leave it for you to decide. Just keep in mind that most of these can also be done in parallel with @Samweli's help if done later.

[luipir]

Asi' mejor

solo dos notas: 3) Scroll bar appear only if necessary in authomatic form 6) I did't changed this icons waiting a more clear indication... I do not like too... but at the same time I prefer to focus on code logic than estetic

[luipir]

I prepare the PR