Connect data layers

[moreirathomas]

We must persist the user's test when the process exits.

Which datastore ?

• SQL database
• document database
• in memory key/value store

[moreirathomas]

Updated flow:

• runner sends a run's report to the API server
• the server stores the raw report in a no-sql database (key-value store like redis, or some gcp hosted solution like firestore, firebase (?), ...)
• the server immediately enqueues the raw report processing (queue + worker, look for gcp hosted solutions again, like cloud function, ideally something with integrated queue system, @pagbrl said something for this exists)
• the worker processes the raw report and stores in a sql database the meaningful stats (look for something like gcp's sql cloud)
• once stored this data is available for the front-end, the raw data may be kept for details on the request made (dataviz, graphs and individual detail?)

    +--------+
    |        |
    | runner |
    |        |
    +--------+
      |
      |   +-----------------------------+
      |   |                             |
      v   |                             v
    +--------+     /----------\     +--------+
    |        |     | no-sql   |     |        |
    | server | --> | key-value| --> | worker |
    |        |     | store    |     |        |
    +--------+     \----------/     +--------+
      |   ^                             |
      |   |                             |
      |   |                             v
      |   |                        /----------\
      |   |                        | sql      |
      |   +----------------------- | database |
      |                            |          |
      v                            \----------/
    +--------+
    |        |
    | webapp |
    |        |
    +--------+