Add query history page

[edongashi]

Closes #4889.

Some caveats:

• If at the time of opening the page or filtering, if a query is in progress, it will keep showing in progress unless you reload the query history page. Should we join the query channel and listen for updates?
• The search phrase is an ILIKE instead of complex text search. It looks up the query note and statement.
• The sidebar for filtering shows all data sources and all possible query states, even if the user does not have queries fitting in those categories. I think narrowing down to only what you have makes it less obvious what is going on. For example, if you filter by data source "banking" and check on "Failed" queries, you will see "no results", which makes it clear that you have no such queries in that data source. If we remove options which the user has no queries in, you cannot ask that question but instead must implicitly infer the fact that "banking" has no "failed" queries because that combination disappeared from the sidebar.
• Local testing may fail because I have upgraded phoenix liveview. You need to delete your deps folder because webpack does not pick up the changes...

[aircloak-robot]

air_test job errored 😢

You can see the full build log by running: ci/production.sh build_log pr 4940 air_test

You can restart the build by running: ci/production.sh force_build pr 4940 air_test

You can start the remote console by running: ci/production.sh remote_console pr 4940 air

Log tail:

    {Credo.Check.Warning.LazyLogging, false},

Please report incorrect results: https://github.com/rrrene/credo/issues

Analysis took 5.9 seconds (0.4s to load, 5.4s running 46 checks on 182 files)
1953 mods/funs, found no issues.

=> 9 sec

aircloak_ci: `/home/ci/.aircloak_ci/data/cache/builds/pr-4940/src/air/ci/container.sh run_in_container aircloak_ci_q3SH8cCi-BBN8XOXuuqimg make check-format`
mix format --check-formatted
=> 2 sec

aircloak_ci: `/home/ci/.aircloak_ci/data/cache/builds/pr-4940/src/air/ci/container.sh run_in_container aircloak_ci_q3SH8cCi-BBN8XOXuuqimg mix bom --elixir deps --node assets/node_modules /tmp`
Gathering package data...
Processing 911 packages...
Bill of Materials written to /tmp/bom.json
Packaging dependency sources...
Dependency sources written to /tmp/dependencies.zip
=> 46 sec

aircloak_ci: `/home/ci/.aircloak_ci/data/cache/builds/pr-4940/src/air/ci/container.sh run_in_container aircloak_ci_q3SH8cCi-BBN8XOXuuqimg MIX_HOME=_build make dialyze`
mix dialyze_retry
Finding applications for analysis

14:05:49.012 [info]  Compiling auto-completions
Finding suitable PLTs
Looking up modules in dialyze_erlang-23.1_elixir-1.11.1_deps-dev.plt
Finding applications for dialyze_erlang-23.1_elixir-1.11.1_deps-dev.plt
Finding modules for dialyze_erlang-23.1_elixir-1.11.1_deps-dev.plt
Checking 1334 modules in dialyze_erlang-23.1_elixir-1.11.1_deps-dev.plt
Finding applications for analysis
Looking up modules in dialyze_erlang-23.1_elixir-1.11.1_deps-dev.plt
Finding modules for analysis
Analysing 198 modules with dialyze_erlang-23.1_elixir-1.11.1_deps-dev.plt
lib/air_web/live/admin/audit_log_live/index.ex:155: The pattern 
          #{'__changed__' := _@33} can never match the type 
          #{'event' := _,
            'id' := _,
            'metadata' := _,
            'query_params' := _,
            'socket' := _,
            'timestamp' := _,
            'user_id' := _,
            'user_name' := _}

lib/air_web/live/admin/audit_log_live/index.ex:155: The pattern 
          #{'__changed__' := _@41} can never match the type 
          #{'event' := _,
            'id' := _,
            'metadata' := _,
            'query_params' := _,
            'socket' := _,
            'timestamp' := _,
            'user_id' := _,
            'user_name' := _}

lib/air_web/live/admin/audit_log_live/index.ex:155: The pattern 
          #{'__changed__' := _@52} can never match the type 
          #{'event' := _,
            'id' := _,
            'metadata' := _,
            'query_params' := _,
            'socket' := _,
            'timestamp' := _,
            'user_id' := _,
            'user_name' := _}

lib/air_web/live/admin/audit_log_live/index.ex:155: The pattern 
          #{'__changed__' := _@2} can never match the type 
          #{'event' := _,
            'id' := _,
            'metadata' := _,
            'query_params' := _,
            'socket' := _,
            'timestamp' := _,
            'user_id' := _,
            'user_name' := _}

lib/air_web/live/admin/audit_log_live/index.ex:155: The pattern 
          #{'__changed__' := _@21} can never match the type 
          #{'event' := _,
            'id' := _,
            'metadata' := _,
            'query_params' := _,
            'socket' := _,
            'timestamp' := _,
            'user_id' := _,
            'user_name' := _}

** (Mix) Dialyzer reported 5 warnings
make: *** [Makefile:41: dialyze] Error 1
=> 25 sec

aircloak_ci: error: 
error running `/home/ci/.aircloak_ci/data/cache/builds/pr-4940/src/air/ci/container.sh run_in_container aircloak_ci_q3SH8cCi-BBN8XOXuuqimg MIX_HOME=_build make dialyze`
aircloak_ci: result: `error`
aircloak_ci: finished in 5:29 min

[aircloak-robot]

Standard tests have passed 💯

[sebastian]

Upon entering the queries page without having run any queries (with an empty database), I get the following JavaScript error:

TypeError: this.handleEvent is not a function. (In 'this.handleEvent("queries", function (data) { reactExports.setResults(data.queries); })', 'this.handleEvent' is undefined)

[sebastian]

And the page never seems to fully load. The loading bar continues to go forever. No errors in the elixir console though:

[edongashi]

TypeError: this.handleEvent is not a function. (In 'this.handleEvent("queries", function (data) { reactExports.setResults(data.queries); })', 'this.handleEvent' is undefined)

Our current liveview version does not support server-sent events. The updated version fixes this. You need to delete your deps and _build directory because webpack is stubborn and does not pick up the new file, probably because we're using a weird file dependency in package json:

https://github.com/Aircloak/aircloak/blob/376fa12bb9b90e96dfe61b052ecbba306573d2e7/air/assets/package.json#L21

[sebastian]

Our current liveview version does not support server-sent events. The updated version fixes this. You need to delete your deps and _build directory because webpack is stubborn and does not pick up the new file, probably because we're using a weird file dependency in package json:

Ah, I only deleted deps. Will try _build too.

[sebastian]

Looks good to me. One minor thing is that the padding is off:

vs

[edongashi]

Thanks for the feedback. All comments have been addressed.

What do you think about adding the data source next to the query date?

[aircloak-robot]

Standard tests have passed 😊

[sebastian]

What do you think about adding the data source next to the query date?

Yes please! I was going to suggest this! 👍🏻

[aircloak-robot]

Pull request can be merged 👏

[edongashi]

Added a clickable link for data sources next to the time.

Meanwhile I found a bug which we probably need to address: If you remove access to a data source for a user, they can no longer filter by it, even if they have queries on that DS. Should we instead show data sources where the user has queries?

[aircloak-robot]

Standard tests have passed 👍

[aircloak-robot]

Standard tests have passed 👏

[aircloak-robot]

Standard tests have passed 🎉

[sebastian]

Meanwhile I found a bug which we probably need to address: If you remove access to a data source for a user, they can no longer filter by it, even if they have queries on that DS. Should we instead show data sources where the user has queries?

Yes, that's a good point. Probably good to show all the data sources the user has queries for. Although isn't that potentially going to be a rather expensive query?

[edongashi]

Probably good to show all the data sources the user has queries for.

Done in 2f9a003 by reusing existing services.

Although isn't that potentially going to be a rather expensive query?

You first filter based on from/to, etc. From that result we gather the data sources in those queries. If it's a couple hundred records it's no big deal.

If you set a large window in say attack where a user has millions of queries, then we may get the same crash as audit log...

[aircloak-robot]

Pull request can be merged 👏