smart-data-lake / sdl-visualization

Visualization for SDLB config
GNU General Public License v3.0
4 stars 1 forks source link

Overview

SDL Visualizer is a single page web application to show SDLB configuration and workflows' runs as a browsable, nice web page.

Usage

Prerequisites

Developer server

To run the app in a local developer server, it suffices to clone the project, install the prerequisites and run "yarn install" and then "yarn start" command. You can then see your changes updated live as any React dev project.

IMPORTANT: if you have not setup your backend, you will not be able to visualize statefiles. Please read the chapter "Backend" to learn about the different setup options.

Build and deployement

The deployment and release of the project has been automated in github using a build.yml file. In essence, a new merge or build in the master branch will trigger the creation of a new release and its corresponding .zip and .tar files.

Alternatively, the createRelease.sh script can be used to manually build the project and then create a .tar file with the necessary elements. Since this script uses the yarn build command, one should install yarn (for example with sudo apt-get install yarn) and run yarn install before executing the script.

Production server

The single page app can be deployed as static webapp on any cloud provider. Care must be taken about the CRSF problematic for the API calls to get config and state. Usually you need to configure a basic API with your cloud provider that handles this requests.

A simple solution on Linux is using lighttpd web server to serve the web app, config files and state files. See https://github.com/smart-data-lake/sdl-visualization/tree/develop/public for details.

Backend

We defined the data format and how the data are fetched in frontend in the chapter "Frontend > Internal logic > Workflows explorer > API and connectors". We now explain in greater detail different way of setting up a source that can be used by the frontend.

Local setup

Prerequisites:

The easiest way to quickly get started is to directly provide the config and state files in the directory public/config and public/state respectively. Once that's done, we can generate an index that will aggregate the statefiles and provide pointers to them. The frontend will simply have to read the preprocessed information in order to display them in the "workflow's" pages and help fetching the state file directly to display it in the "run" page.

Once your files are placed in the correct directory, run:

$ ./build_index.sh <path-to-statefiles> <path-to-configfiles>"

IMPORTANT: the current script doesn't allow for only providing <path-to-configfiles>. If only one argument is provided it will be considered as a <path-to-statefiles>)

This will generate an index for each of the file sources. The outputs are JSON-Lines files named index and stored in public/state and public/config respectively.

Finally you have to make sure you are using the default connector for local statefiles. Change the field backendConfig to local in public/manifest.json if it is not already the case.

You should now be able to browse to localhost:3000/ and should be greeted by the home page.

REST API

Perequisites:

In order to use the provided connector, it is necessary that your REST endpoints must be defined as follow:

Where <BASE_URL> can be any string.

You should then change the field backendConfig to rest;<BASE_URL> in public/manifest.json.

You should now be able to browse to localhost:3000/ and should be greeted by the home page.

Other settings

In public/manifest.json you can set various configurations for the UI: