search

Aircloak / aircloak

This repository contains the Aircloak Air frontend as well as the code for our Cloak query and anonymization platform
2 stars 0 forks source link

Batch log DB insertions #4955

Closed edongashi closed 3 years ago

edongashi commented 3 years ago

The batch insertion worked amazingly well! The server's mailbox is always empty! Fixes #4953. Fixes #4954.

Debug messages < threshold are not discarded. If this threshold is exceeded, future debug messages are dropped until there's no high load for 5 seconds.

aircloak-robot commented 3 years ago

Standard tests have passed 👏

aircloak-robot commented 3 years ago

Pull request can be merged 👍

aircloak-robot commented 3 years ago

Standard tests have passed 🎉

aircloak-robot commented 3 years ago

Pull request can be merged 🎉

aircloak-robot commented 3 years ago

air_test job errored 💔

You can see the full build log by running: ci/production.sh build_log pr 4955 air_test

You can restart the build by running: ci/production.sh force_build pr 4955 air_test

You can start the remote console by running: ci/production.sh remote_console pr 4955 air

Log tail:

SELECT g0."id", g0."name", g0."source", g0."ldap_dn", g0."admin", g0."system", g0."inserted_at", g0."updated_at" FROM "groups" AS g0 []

10:31:56.371 [info]  == Migrated 20200710105907 in 0.0s

10:31:56.376 [info]  == Running 20201019140409 Air.Repo.Migrations.AddIndexForFasterLoadOfLastDataSource.change/0 forward

10:31:56.376 [info]  create index queries_user_id_context_index

10:31:56.378 [info]  == Migrated 20201019140409 in 0.0s

10:31:56.382 [info]  == Running 20201110161633 Air.Repo.Migrations.AddSoftDeleteToExplorerAnalysis.change/0 forward

10:31:56.382 [info]  alter table explorer_analyses

10:31:56.383 [info]  == Migrated 20201110161633 in 0.0s

10:31:56.387 [info]  == Running 20201209140829 Air.Repo.Migrations.MaterializeTotalTime.change/0 forward

10:31:56.388 [info]  alter table queries

10:31:56.389 [info]  execute "UPDATE queries SET total_time = (SELECT SUM(value::int) FROM jsonb_each_text(time_spent))"

10:31:56.390 [info]  == Migrated 20201209140829 in 0.0s

10:31:56.394 [info]  == Running 20201217142653 Air.Repo.Migrations.AddTypeCheckingToSettings.change/0 forward

10:31:56.394 [info]  alter table settings

10:31:56.395 [info]  == Migrated 20201217142653 in 0.0s

10:31:56.399 [info]  == Running 20210621131817 Air.Repo.Migrations.AddNoteToQueries.change/0 forward

10:31:56.399 [info]  alter table queries

10:31:56.401 [info]  == Migrated 20210621131817 in 0.0s

10:31:56.405 [info]  == Running 20210623210531 Air.Repo.Migrations.CreateLogs.change/0 forward

10:31:56.405 [info]  execute "CREATE TYPE public.log_source AS ENUM ('air', 'cloak', 'system')"

10:31:56.406 [info]  create table logs

10:31:56.410 [info]  create index logs_timestamp_source_index

10:31:56.411 [info]  == Migrated 20210623210531 in 0.0s

10:31:56.415 [info]  == Running 20210701203210 Air.Repo.Migrations.AddLevelToLogs.change/0 forward

10:31:56.415 [info]  execute "CREATE TYPE public.log_level AS ENUM ('debug', 'info', 'warn', 'error')"

10:31:56.417 [info]  alter table logs

10:31:56.418 [info]  == Migrated 20210701203210 in 0.0s

10:31:56.422 [info]  == Running 20210702083629 Air.Repo.Migrations.RefactorLogsIndexes.change/0 forward

10:31:56.422 [info]  drop index logs_timestamp_source_index

10:31:56.423 [info]  create index logs_timestamp_index

10:31:56.425 [info]  == Migrated 20210702083629 in 0.0s

10:31:56.429 [info]  == Running 20210705100146 Air.Repo.Migrations.DropLicenses.up/0 forward

10:31:56.429 [info]  drop table licenses

10:31:56.431 [info]  == Migrated 20210705100146 in 0.0s
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
[os_mon] memory supervisor port (memsup): Erlang has closed
mix test

10:32:00.193 [info]  Compiling auto-completions
Excluding tags: [:disabled, :pending]

..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

  1) test removes logs older than retention days (Air.Service.Cleanup.Test)
     test/air/service/cleanup_test.exs:65
     ** (ErlangError) Erlang error: :timeout_value
     code: :timer.sleep(:timer.seconds(1.5))
     stacktrace:
       (stdlib 3.13.2) timer.erl:152: :timer.sleep/1
       test/air/service/cleanup_test.exs:70: (test)

.......

Finished in 240.8 seconds
666 tests, 1 failure

Randomized with seed 326783
[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
make: *** [Makefile:21: test] Error 1
=> 283 sec

aircloak_ci: error: 
error running `/home/ci/.aircloak_ci/data/cache/builds/pr-4955/src/air/ci/container.sh run_in_container aircloak_ci_c6pChhJmgDvaAAvMrDPlnw make test`
aircloak_ci: result: `error`
aircloak_ci: finished in 5:24 min
aircloak-robot commented 3 years ago

Standard tests have passed 👍

aircloak-robot commented 3 years ago

Pull request can be merged 🎉

sebastian commented 3 years ago

@edongashi could you please deploy to demo so we have the latest version there?

edongashi commented 3 years ago

Sure, deploying now. Will ping once it's ready.