Flexible History/Log Register

[nyalldawson]

QGIS Enhancement: Flexible History/Log Register

Date 2018/08/10

Author Nyall Dawson (@nyalldawson)

Contact nyall dot dawson at gmail dot com

maintainer @nyalldawson

Version QGIS 3.4

Summary

Currently within QGIS, the Processing plugin records a log of any algorithms run by users and the parameters used for this algorithm. The history dialog gives a list of these algorithms, with the option to copy the algorithm run as a python string or re-run the algorithm using the same parameters:

Additionally, Processing adds a separate "Results Viewer" dock, which shows a temporary (lost on QGIS close) list of HTML outputs from algorithms:

These features are used by Processing only, yet there is a strong demand for a universal "history" log, allowing other areas of QGIS to record log entries and provide user actions relating to these. Additionally, there is a need to allow certain log entries to be stored inside and associated with a particular QGIS project, allowing these entries to persist across installs and work within multi-user scenarios.

This is a highly desirable feature, given the ongoing trend toward open, accountable data processes and the need for QGIS to allow audit trails for the lifetime of a particular project.

Potential uses

• Replacing the existing Processing history handling, allowing the history of algorithms executed while a project is open to be stored within that project as well as locally. Additionally, the full log contents of the algorithm can be stored for future reference.
• Recording exports from print layouts and atlases, and their location. Stored locally and within projects, with actions to open the outputs directly and open the target folder
• Potentially: A project layer log, which stores interactions such as layer creation, layer removal, and layer edits inside a project's history only (not locally)

Proposed Solution

The following history classes will all reside within the GUI library. This is done for two reasons:

• The types of entries recorded in the history are all directly related to user actions, and the history log is not intended to record actions performed by e.g. plugins or scripts.

• History providers include gui related API, so keeping all the classes within the GUI library avoids GUI related API leaking into core, or requiring a more complex API with two separate registries and sets of classes for GUI/non-GUI.

QgsHistoryProviderRegistry

A new registry, QgsHistoryProviderRegistry, will be created. A global instance of this registry will be available via QgsGui::historyProviderRegistry().

QgsHistoryProviderRegistry has a method to register history providers (see below):

/**
 * Adds a \a provider to the registry. Ownership of the provider is 
 * transferred to the registry.
 * Returns true if the provider was successfully added.
 */
bool addProvider( QgsAbstractHistoryProvider* provider SIP_TRANSFER )

Additionally, all the usual methods for looking up providers by id, removing providers, etc will be present in the registry.

Individual history entries would be added to the log by calling a addEntry in QgsHistoryProviderRegistry. Adding an entry requires both the associated provider's ID, and a QVariant representing the provider-specific content for that entry. The entry content is deliberately kept free-form, to allow individual providers the flexibility to store any required content without restriction. (However, it is anticipated that most entries will consist of QVariantMaps.)

/**
 * Adds an \a entry to the history logs.
 * The \a providerId specifies the history provider responsible for this entry.
 * Entry options are specified via the \a options argument.
 */  
addEntry( const QString& providerId, const QVariant& entry, const HistoryEntryOptions& options );

The HistoryEntryOptions is a simple struct, used to control how the entry will be handled. Initially the options will be:

struct HistoryEntryOptions
{
  //! Whether the entry should be stored within the local history database
  bool storeLocally;

  //! Whether the entry should be stored inside the current project's history
  bool storeInProject;
};

When an entry is added to the registry, it is recorded to the corresponding databases (local and project, depending on the entry options specified) alongside the provider id, current timestamp, the QGIS version, the logged on user name and account name, and the entry's variant. The entry's variant is stored as XML, using QgsXmlUtils to convert the variant to XML.

It is the responsibility of individual providers to register entries whenever applicable. E.g. the processing provider will add entries to the registry whenever an algorithm is executed through the processing toolbox, a layout export provider will add entries from the corresponding methods in QgsLayoutDesignerDialog, etc.

QgsHistoryProviderRegistry will also have methods for querying the history, which will return a list of corresponding entries. Querying via data time ranges, search strings, and the storage backend (either local database or active project) will be supported.

QgsAbstractHistoryProvider

A new abstract class QgsAbstractHistoryProvider will be created. QgsAbstractHistoryProvider subclasses will be created by each individual area of QGIS which wants to record entries in the history log (e.g. QgsProcessingHistoryProvider, QgsLayoutExportHistoryProvider, QgsProjectLayerInteractionsHistoryProvider).

The QgsAbstractHistoryProvider class will have members:

/**
 * Returns the provider's unique id, which is used to associate existing history entries with the provider. 
 */
virtual QString id() const = 0;

/**
 * Creates a new history node for the given \a entry.
 */
QgsHistoryEntryNode* createNodeForEntry( const QVariant& entry ) = 0 SIP_FACTORY;

QgsHistoryWidget

A reusable widget for displaying history content will be created, QgsHistoryWidget. The widget will have a method to set the associated backend, i.e. either local history database or active project. History entries will always be filtered to the matching backend.

The history widget will consist of a tree view of history items, along with a filter box allowing users to enter a search string to filter to history.

Creating the history widget tree

When a user views the history log, the registry will call createNodeForEntry on the associated provider for every entry with a matching provider ID. The provider will then return a new QgsHistoryEntryNode for inclusion in a tree view of the log. QgsHistoryEntryNode can have children, allowing providers to return a structured tree for each entry. This allows flexibility in how individual providers will present history items - e.g. a provider algorithm execution entry may have child nodes for the "algorithm log", the "python command", and nodes for inputs/outputs containing child nodes for each layer used as an input or created as an output by the algorithm.

QgsHistoryEntryNode

QgsHistoryEntryNode has a number of virtual members (in addition to members for adding/retrieving the node's children):

/**
 * Returns a HTML formatted text string which should be shown to a user when
 * selecting the node.
 *
 * Subclasses should implement this method or createWidget(), but not both.
 */
virtual QString html() const;

/**
 * Returns a new widget which should be shown to users when selecting the node.
 * 
 * If a nullptr is returned, the node's html() method will be called instead to
 * create the node's content.
 */
virtual QWidget* createWidget() SIP_FACTORY;

/**
 * Returns a list of actions which users can trigger to interact with the history
 * entry. Buttons corresponding to each action will be automatically created and
 * shown to users.
 *
 * Actions should be parented to the specified \a parent widget.
 */
virtual QList< QAction* > actions( QWidget* parent );

/**
 * Returns true if the node matches the specified \a searchString, and
 * should be shown in filtered results with that search string.
 */
virtual bool matchesString( const QString& searchString );

Application UI

The local history will be visible through a new entry in the Settings menu. Triggering this action will open a dialog showing all local history entries.

The project history will be visible through a new tab in the Project Properties dialog. Additionally, this tab will have an option to "clear" the project's history.

Storage of history

The local history will be stored in a sqlite database within the user's active profile folder.

Project history will be stored directly within the project's XML (possibly project auxiliary storage could be used instead, although the api for that does not yet exist).

Affected Files

Mostly new classes, with changes to processing to replace the existing history, log and results window with the new implementations.

Performance Implications

N/A

Backwards Compatibility

N/A

Votes

(required)

[pcav]

IMHO it is important to keep the possibility of running past algs with a simple action (e.g. double click). If this is done, +1 for me.

[nyalldawson]

IMHO it is important to keep the possibility of running past algs with a simple action (e.g. double click). If this is done, +1 for me.

That will definitely remain - but it will also be extended with the ability to open input/output datasets in QGIS/file explorer, view the full algorithm execution log, etc...

[nyalldawson]

Any objections/suggestions? @m-kuhn @wonder-sk @elpaso ?

[elpaso]

No objections (even if I don't have a clear picture of the use cases for this enhancement).

Only a minor note about putting this into settings: wouldn't be the project menu a better place, at least for the project-storage history/logs?

Generally speaking, I don't feel like all of this is belonging to settings, but I've no better recommendation.

[nyalldawson]

@elpaso

Generally speaking, I don't feel like all of this is belonging to settings, but I've no better recommendation.

Yeah, I can't come up with a better place - it's no longer Processing specific, so that doesn't fit. The View menu would be an option, but that's already very full. Project History does fit in the Project menu, so that's fine, but the local history,... not sure.

No objections (even if I don't have a clear picture of the use cases for this enhancement).

The main driver initially here is part of the Processing GUI refactoring grant. The history panel needs reworking and made more robust, but it seems a shame to just replace like-for-like.

The big benefit in adding a project history is in being able to justify decisions made at a later stage. E.g. if something comes up and you need to check the history of a project you worked on 6+ months ago, it'll mean that many of the actions taken while working on that project are automatically logged and can be reviewed.

It's also intended as a helper for multi-user environments. I often work collaboratively on projects, and would love to know what's been done on a project since I last worked on it. (AKA who the heck screwed up this project???!).

[Gustry]

The big benefit in adding a project history is in being able to justify decisions made at a later stage. E.g. if something comes up and you need to check the history of a project you worked on 6+ months ago, it'll mean that many of the actions taken while working on that project are automatically logged and can be reviewed.

This reminds me the "History" tab in Metadata panel (both layers and project properties). Do you plan to have a kind of bridge between these 2?

[nyalldawson]

@Gustry

Do you plan to have a kind of bridge between these 2?

It's not in scope for this project, but I'd like to see Processing retain the existing metadata and append to the history log whenever a new algorithm is run on the data. I don't think that specifically relates to the work proposed here, though.

[ar-siddiqui]

@nyalldawson one suggestion is to change the processing.run to processing.execAlgorithmDialog This would make the behavior of both double-clicking the history item and running the command from the python console similar. Currently, the former loads the algorithm with pre-populated parameters, and the latter directly run it. Loading the parameters is more useful because that allows the user to change parameters before running. A lot of the time history is used to fix a mistake in parameters in the previous run.

[nyalldawson]

@ar-siddiqui

one suggestion is to change the processing.run

The intention here though is for building (non-interactive) Python scripts