The Gitlab container is definitely outputting a variety of logs and the formatting is pretty consistent, which is nice. I'm not sure exactly how to filter on just logs coming from the exceptions_json.log file.
It seems that the starting point is a line that explains what kind of log is being presented, then the actual log message(s) appear afterwards.
There were some small changes to allow the logs to show up in the Error Logs panel, but the filtering isn't quite there. Had to add the instance label to the filtering/selector part and then update the cluster label's allValues to be .+. Doing this, all logs from the gitlab container show up in the panel.
If there's a path for filtering things that make sense I would love help understanding how to make it happen.
Log output from the container:
==> /var/log/gitlab/gitlab-rails/exceptions_json.log <==
{"severity":"ERROR","time":"2024-08-20T15:34:24.435Z","correlation_id":"6adf1737cb08e723bd20a6c1f539d434","exception.class":"Gitlab::Git::PreReceiveError","exception.message":"Internal API unreachable","exception.backtrace":["lib/gitlab/gitaly_client/operation_service.rb:418:in `user_commit_files'","lib/gitlab/git/repository.rb:924:in `block in multi_action'","lib/gitlab/git/wraps_gitaly_errors.rb:7:in `wrapped_gitaly_errors'","lib/gitlab/git/repository.rb:923:in `multi_action'","app/models/repository.rb:853:in `block in multi_action'","app/models/repository.rb:836:in `with_cache_hooks'","app/models/repository.rb:853:in `multi_action'","app/models/repository.rb:811:in `create_file'","app/services/files/create_service.rb:16:in `create_transformed_commit'","app/services/files/create_service.rb:10:in `create_commit!'","app/services/commits/create_service.rb:30:in `execute'","app/services/projects/create_service.rb:180:in `create_readme'","app/services/projects/create_service.rb:126:in `after_create_actions'","app/services/projects/create_service.rb:70:in `block in execute'","lib/gitlab/application_context.rb:110:in `block in use'","lib/gitlab/application_context.rb:110:in `use'","lib/gitlab/application_context.rb:52:in `with_context'","app/services/projects/create_service.rb:69:in `execute'","app/services/concerns/measurable.rb:35:in `execute'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:59:in `create_project'","app/models/concerns/stepable.rb:14:in `call'","app/models/concerns/stepable.rb:14:in `block in execute_steps'","app/models/concerns/stepable.rb:13:in `each'","app/models/concerns/stepable.rb:13:in `inject'","app/models/concerns/stepable.rb:13:in `execute_steps'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:27:in `execute'","(eval):3:in `block (2 levels) in run_file'","lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:115:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:184:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:111:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'","lib/gitlab/database.rb:331:in `block in transaction'","lib/gitlab/database.rb:330:in `transaction'","lib/tasks/gitlab/db.rake:104:in `block (3 levels) in \u003ctop (required)\u003e'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":"6adf1737cb08e723bd20a6c1f539d434"}
{"severity":"ERROR","time":"2024-08-20T15:34:24.461Z","correlation_id":"6adf1737cb08e723bd20a6c1f539d434","exception.class":"Sidekiq::Worker::EnqueueFromTransactionError","exception.message":"Pages::InvalidateDomainCacheWorker.perform_async cannot be enqueued inside a transaction as this can lead to\nrace conditions when the worker runs before the transaction is committed and\ntries to access a model that has not been saved yet.\n\nUse an `after_commit` hook, or include `AfterCommitQueue` and use a `run_after_commit` block instead.\n","exception.backtrace":["config/initializers/forbid_sidekiq_in_transactions.rb:28:in `raise_inside_transaction_exception'","config/initializers/forbid_sidekiq_in_transactions.rb:43:in `block (2 levels) in \u003cmodule:NoEnqueueingFromTransactions\u003e'","lib/gitlab/event_store/subscription.rb:16:in `consume_event'","lib/gitlab/event_store/store.rb:33:in `block in publish'","lib/gitlab/event_store/store.rb:32:in `each'","lib/gitlab/event_store/store.rb:32:in `publish'","lib/gitlab/event_store.rb:17:in `publish'","app/services/projects/create_service.rb:303:in `publish_event'","app/services/projects/create_service.rb:129:in `after_create_actions'","app/services/projects/create_service.rb:70:in `block in execute'","lib/gitlab/application_context.rb:110:in `block in use'","lib/gitlab/application_context.rb:110:in `use'","lib/gitlab/application_context.rb:52:in `with_context'","app/services/projects/create_service.rb:69:in `execute'","app/services/concerns/measurable.rb:35:in `execute'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:59:in `create_project'","app/models/concerns/stepable.rb:14:in `call'","app/models/concerns/stepable.rb:14:in `block in execute_steps'","app/models/concerns/stepable.rb:13:in `each'","app/models/concerns/stepable.rb:13:in `inject'","app/models/concerns/stepable.rb:13:in `execute_steps'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:27:in `execute'","(eval):3:in `block (2 levels) in run_file'","lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:115:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:184:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:111:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'","lib/gitlab/database.rb:331:in `block in transaction'","lib/gitlab/database.rb:330:in `transaction'","lib/tasks/gitlab/db.rake:104:in `block (3 levels) in \u003ctop (required)\u003e'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":"6adf1737cb08e723bd20a6c1f539d434"}
{"severity":"ERROR","time":"2024-08-20T15:39:26.501Z","correlation_id":"02c6b03f807133feb663d8c5f4c37938","exception.class":"Sidekiq::Worker::EnqueueFromTransactionError","exception.message":"AuthorizedProjectUpdate::ProjectRecalculateWorker.perform_async cannot be enqueued inside a transaction as this can lead to\nrace conditions when the worker runs before the transaction is committed and\ntries to access a model that has not been saved yet.\n\nUse an `after_commit` hook, or include `AfterCommitQueue` and use a `run_after_commit` block instead.\n","exception.backtrace":["config/initializers/forbid_sidekiq_in_transactions.rb:28:in `raise_inside_transaction_exception'","config/initializers/forbid_sidekiq_in_transactions.rb:43:in `block (2 levels) in \u003cmodule:NoEnqueueingFromTransactions\u003e'","app/services/projects/create_service.rb:150:in `setup_authorizations'","app/services/projects/create_service.rb:120:in `after_create_actions'","app/services/projects/create_service.rb:70:in `block in execute'","lib/gitlab/application_context.rb:110:in `block in use'","lib/gitlab/application_context.rb:110:in `use'","lib/gitlab/application_context.rb:52:in `with_context'","app/services/projects/create_service.rb:69:in `execute'","app/services/concerns/measurable.rb:35:in `execute'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:59:in `create_project'","app/models/concerns/stepable.rb:14:in `call'","app/models/concerns/stepable.rb:14:in `block in execute_steps'","app/models/concerns/stepable.rb:13:in `each'","app/models/concerns/stepable.rb:13:in `inject'","app/models/concerns/stepable.rb:13:in `execute_steps'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:27:in `execute'","(eval):3:in `block (2 levels) in run_file'","lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:115:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:184:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:111:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'","lib/gitlab/database.rb:331:in `block in transaction'","lib/gitlab/database.rb:330:in `transaction'","lib/tasks/gitlab/db.rake:104:in `block (3 levels) in \u003ctop (required)\u003e'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":"02c6b03f807133feb663d8c5f4c37938"}
{"severity":"ERROR","time":"2024-08-20T15:39:26.524Z","correlation_id":"02c6b03f807133feb663d8c5f4c37938","exception.class":"Sidekiq::Worker::EnqueueFromTransactionError","exception.message":"Projects::PostCreationWorker.perform_async cannot be enqueued inside a transaction as this can lead to\nrace conditions when the worker runs before the transaction is committed and\ntries to access a model that has not been saved yet.\n\nUse an `after_commit` hook, or include `AfterCommitQueue` and use a `run_after_commit` block instead.\n","exception.backtrace":["config/initializers/forbid_sidekiq_in_transactions.rb:28:in `raise_inside_transaction_exception'","config/initializers/forbid_sidekiq_in_transactions.rb:43:in `block (2 levels) in \u003cmodule:NoEnqueueingFromTransactions\u003e'","app/services/projects/create_service.rb:124:in `after_create_actions'","app/services/projects/create_service.rb:70:in `block in execute'","lib/gitlab/application_context.rb:110:in `block in use'","lib/gitlab/application_context.rb:110:in `use'","lib/gitlab/application_context.rb:52:in `with_context'","app/services/projects/create_service.rb:69:in `execute'","app/services/concerns/measurable.rb:35:in `execute'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:59:in `create_project'","app/models/concerns/stepable.rb:14:in `call'","app/models/concerns/stepable.rb:14:in `block in execute_steps'","app/models/concerns/stepable.rb:13:in `each'","app/models/concerns/stepable.rb:13:in `inject'","app/models/concerns/stepable.rb:13:in `execute_steps'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:27:in `execute'","(eval):3:in `block (2 levels) in run_file'","lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:115:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:184:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:111:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'","lib/gitlab/database.rb:331:in `block in transaction'","lib/gitlab/database.rb:330:in `transaction'","lib/tasks/gitlab/db.rake:104:in `block (3 levels) in \u003ctop (required)\u003e'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":"02c6b03f807133feb663d8c5f4c37938"}
{"severity":"ERROR","time":"2024-08-20T15:39:27.196Z","correlation_id":"02c6b03f807133feb663d8c5f4c37938","exception.class":"Gitlab::Git::PreReceiveError","exception.message":"Internal API unreachable","exception.backtrace":["lib/gitlab/gitaly_client/operation_service.rb:418:in `user_commit_files'","lib/gitlab/git/repository.rb:924:in `block in multi_action'","lib/gitlab/git/wraps_gitaly_errors.rb:7:in `wrapped_gitaly_errors'","lib/gitlab/git/repository.rb:923:in `multi_action'","app/models/repository.rb:853:in `block in multi_action'","app/models/repository.rb:836:in `with_cache_hooks'","app/models/repository.rb:853:in `multi_action'","app/models/repository.rb:811:in `create_file'","app/services/files/create_service.rb:16:in `create_transformed_commit'","app/services/files/create_service.rb:10:in `create_commit!'","app/services/commits/create_service.rb:30:in `execute'","app/services/projects/create_service.rb:180:in `create_readme'","app/services/projects/create_service.rb:126:in `after_create_actions'","app/services/projects/create_service.rb:70:in `block in execute'","lib/gitlab/application_context.rb:110:in `block in use'","lib/gitlab/application_context.rb:110:in `use'","lib/gitlab/application_context.rb:52:in `with_context'","app/services/projects/create_service.rb:69:in `execute'","app/services/concerns/measurable.rb:35:in `execute'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:59:in `create_project'","app/models/concerns/stepable.rb:14:in `call'","app/models/concerns/stepable.rb:14:in `block in execute_steps'","app/models/concerns/stepable.rb:13:in `each'","app/models/concerns/stepable.rb:13:in `inject'","app/models/concerns/stepable.rb:13:in `execute_steps'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:27:in `execute'","(eval):3:in `block (2 levels) in run_file'","lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:115:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:184:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:111:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'","lib/gitlab/database.rb:331:in `block in transaction'","lib/gitlab/database.rb:330:in `transaction'","lib/tasks/gitlab/db.rake:104:in `block (3 levels) in \u003ctop (required)\u003e'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":"02c6b03f807133feb663d8c5f4c37938"}
{"severity":"ERROR","time":"2024-08-20T15:39:27.223Z","correlation_id":"02c6b03f807133feb663d8c5f4c37938","exception.class":"Sidekiq::Worker::EnqueueFromTransactionError","exception.message":"Pages::InvalidateDomainCacheWorker.perform_async cannot be enqueued inside a transaction as this can lead to\nrace conditions when the worker runs before the transaction is committed and\ntries to access a model that has not been saved yet.\n\nUse an `after_commit` hook, or include `AfterCommitQueue` and use a `run_after_commit` block instead.\n","exception.backtrace":["config/initializers/forbid_sidekiq_in_transactions.rb:28:in `raise_inside_transaction_exception'","config/initializers/forbid_sidekiq_in_transactions.rb:43:in `block (2 levels) in \u003cmodule:NoEnqueueingFromTransactions\u003e'","lib/gitlab/event_store/subscription.rb:16:in `consume_event'","lib/gitlab/event_store/store.rb:33:in `block in publish'","lib/gitlab/event_store/store.rb:32:in `each'","lib/gitlab/event_store/store.rb:32:in `publish'","lib/gitlab/event_store.rb:17:in `publish'","app/services/projects/create_service.rb:303:in `publish_event'","app/services/projects/create_service.rb:129:in `after_create_actions'","app/services/projects/create_service.rb:70:in `block in execute'","lib/gitlab/application_context.rb:110:in `block in use'","lib/gitlab/application_context.rb:110:in `use'","lib/gitlab/application_context.rb:52:in `with_context'","app/services/projects/create_service.rb:69:in `execute'","app/services/concerns/measurable.rb:35:in `execute'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:59:in `create_project'","app/models/concerns/stepable.rb:14:in `call'","app/models/concerns/stepable.rb:14:in `block in execute_steps'","app/models/concerns/stepable.rb:13:in `each'","app/models/concerns/stepable.rb:13:in `inject'","app/models/concerns/stepable.rb:13:in `execute_steps'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:27:in `execute'","(eval):3:in `block (2 levels) in run_file'","lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:115:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:184:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:111:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'","lib/gitlab/database.rb:331:in `block in transaction'","lib/gitlab/database.rb:330:in `transaction'","lib/tasks/gitlab/db.rake:104:in `block (3 levels) in \u003ctop (required)\u003e'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":"02c6b03f807133feb663d8c5f4c37938"}
{"severity":"ERROR","time":"2024-08-20T16:32:17.523Z","correlation_id":"da83d2a1454918882bf10d1fc6f81942","exception.class":"Sidekiq::Worker::EnqueueFromTransactionError","exception.message":"AuthorizedProjectUpdate::ProjectRecalculateWorker.perform_async cannot be enqueued inside a transaction as this can lead to\nrace conditions when the worker runs before the transaction is committed and\ntries to access a model that has not been saved yet.\n\nUse an `after_commit` hook, or include `AfterCommitQueue` and use a `run_after_commit` block instead.\n","exception.backtrace":["config/initializers/forbid_sidekiq_in_transactions.rb:28:in `raise_inside_transaction_exception'","config/initializers/forbid_sidekiq_in_transactions.rb:43:in `block (2 levels) in \u003cmodule:NoEnqueueingFromTransactions\u003e'","app/services/projects/create_service.rb:150:in `setup_authorizations'","app/services/projects/create_service.rb:120:in `after_create_actions'","app/services/projects/create_service.rb:70:in `block in execute'","lib/gitlab/application_context.rb:110:in `block in use'","lib/gitlab/application_context.rb:110:in `use'","lib/gitlab/application_context.rb:52:in `with_context'","app/services/projects/create_service.rb:69:in `execute'","app/services/concerns/measurable.rb:35:in `execute'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:59:in `create_project'","app/models/concerns/stepable.rb:14:in `call'","app/models/concerns/stepable.rb:14:in `block in execute_steps'","app/models/concerns/stepable.rb:13:in `each'","app/models/concerns/stepable.rb:13:in `inject'","app/models/concerns/stepable.rb:13:in `execute_steps'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:27:in `execute'","(eval):3:in `block (2 levels) in run_file'","lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:115:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:184:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:111:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'","lib/gitlab/database.rb:331:in `block in transaction'","lib/gitlab/database.rb:330:in `transaction'","lib/tasks/gitlab/db.rake:104:in `block (3 levels) in \u003ctop (required)\u003e'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":"da83d2a1454918882bf10d1fc6f81942"}
{"severity":"ERROR","time":"2024-08-20T16:32:17.541Z","correlation_id":"da83d2a1454918882bf10d1fc6f81942","exception.class":"Sidekiq::Worker::EnqueueFromTransactionError","exception.message":"Projects::PostCreationWorker.perform_async cannot be enqueued inside a transaction as this can lead to\nrace conditions when the worker runs before the transaction is committed and\ntries to access a model that has not been saved yet.\n\nUse an `after_commit` hook, or include `AfterCommitQueue` and use a `run_after_commit` block instead.\n","exception.backtrace":["config/initializers/forbid_sidekiq_in_transactions.rb:28:in `raise_inside_transaction_exception'","config/initializers/forbid_sidekiq_in_transactions.rb:43:in `block (2 levels) in \u003cmodule:NoEnqueueingFromTransactions\u003e'","app/services/projects/create_service.rb:124:in `after_create_actions'","app/services/projects/create_service.rb:70:in `block in execute'","lib/gitlab/application_context.rb:110:in `block in use'","lib/gitlab/application_context.rb:110:in `use'","lib/gitlab/application_context.rb:52:in `with_context'","app/services/projects/create_service.rb:69:in `execute'","app/services/concerns/measurable.rb:35:in `execute'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:59:in `create_project'","app/models/concerns/stepable.rb:14:in `call'","app/models/concerns/stepable.rb:14:in `block in execute_steps'","app/models/concerns/stepable.rb:13:in `each'","app/models/concerns/stepable.rb:13:in `inject'","app/models/concerns/stepable.rb:13:in `execute_steps'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:27:in `execute'","(eval):3:in `block (2 levels) in run_file'","lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:115:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:184:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:111:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'","lib/gitlab/database.rb:331:in `block in transaction'","lib/gitlab/database.rb:330:in `transaction'","lib/tasks/gitlab/db.rake:104:in `block (3 levels) in \u003ctop (required)\u003e'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":"da83d2a1454918882bf10d1fc6f81942"}
{"severity":"ERROR","time":"2024-08-20T16:32:18.147Z","correlation_id":"da83d2a1454918882bf10d1fc6f81942","exception.class":"Gitlab::Git::PreReceiveError","exception.message":"Internal API unreachable","exception.backtrace":["lib/gitlab/gitaly_client/operation_service.rb:418:in `user_commit_files'","lib/gitlab/git/repository.rb:924:in `block in multi_action'","lib/gitlab/git/wraps_gitaly_errors.rb:7:in `wrapped_gitaly_errors'","lib/gitlab/git/repository.rb:923:in `multi_action'","app/models/repository.rb:853:in `block in multi_action'","app/models/repository.rb:836:in `with_cache_hooks'","app/models/repository.rb:853:in `multi_action'","app/models/repository.rb:811:in `create_file'","app/services/files/create_service.rb:16:in `create_transformed_commit'","app/services/files/create_service.rb:10:in `create_commit!'","app/services/commits/create_service.rb:30:in `execute'","app/services/projects/create_service.rb:180:in `create_readme'","app/services/projects/create_service.rb:126:in `after_create_actions'","app/services/projects/create_service.rb:70:in `block in execute'","lib/gitlab/application_context.rb:110:in `block in use'","lib/gitlab/application_context.rb:110:in `use'","lib/gitlab/application_context.rb:52:in `with_context'","app/services/projects/create_service.rb:69:in `execute'","app/services/concerns/measurable.rb:35:in `execute'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:59:in `create_project'","app/models/concerns/stepable.rb:14:in `call'","app/models/concerns/stepable.rb:14:in `block in execute_steps'","app/models/concerns/stepable.rb:13:in `each'","app/models/concerns/stepable.rb:13:in `inject'","app/models/concerns/stepable.rb:13:in `execute_steps'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:27:in `execute'","(eval):3:in `block (2 levels) in run_file'","lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:115:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:184:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:111:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'","lib/gitlab/database.rb:331:in `block in transaction'","lib/gitlab/database.rb:330:in `transaction'","lib/tasks/gitlab/db.rake:104:in `block (3 levels) in \u003ctop (required)\u003e'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":"da83d2a1454918882bf10d1fc6f81942"}
{"severity":"ERROR","time":"2024-08-20T16:32:18.162Z","correlation_id":"da83d2a1454918882bf10d1fc6f81942","exception.class":"Sidekiq::Worker::EnqueueFromTransactionError","exception.message":"Pages::InvalidateDomainCacheWorker.perform_async cannot be enqueued inside a transaction as this can lead to\nrace conditions when the worker runs before the transaction is committed and\ntries to access a model that has not been saved yet.\n\nUse an `after_commit` hook, or include `AfterCommitQueue` and use a `run_after_commit` block instead.\n","exception.backtrace":["config/initializers/forbid_sidekiq_in_transactions.rb:28:in `raise_inside_transaction_exception'","config/initializers/forbid_sidekiq_in_transactions.rb:43:in `block (2 levels) in \u003cmodule:NoEnqueueingFromTransactions\u003e'","lib/gitlab/event_store/subscription.rb:16:in `consume_event'","lib/gitlab/event_store/store.rb:33:in `block in publish'","lib/gitlab/event_store/store.rb:32:in `each'","lib/gitlab/event_store/store.rb:32:in `publish'","lib/gitlab/event_store.rb:17:in `publish'","app/services/projects/create_service.rb:303:in `publish_event'","app/services/projects/create_service.rb:129:in `after_create_actions'","app/services/projects/create_service.rb:70:in `block in execute'","lib/gitlab/application_context.rb:110:in `block in use'","lib/gitlab/application_context.rb:110:in `use'","lib/gitlab/application_context.rb:52:in `with_context'","app/services/projects/create_service.rb:69:in `execute'","app/services/concerns/measurable.rb:35:in `execute'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:59:in `create_project'","app/models/concerns/stepable.rb:14:in `call'","app/models/concerns/stepable.rb:14:in `block in execute_steps'","app/models/concerns/stepable.rb:13:in `each'","app/models/concerns/stepable.rb:13:in `inject'","app/models/concerns/stepable.rb:13:in `execute_steps'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:27:in `execute'","(eval):3:in `block (2 levels) in run_file'","lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:115:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:184:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:111:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'","lib/gitlab/database.rb:331:in `block in transaction'","lib/gitlab/database.rb:330:in `transaction'","lib/tasks/gitlab/db.rake:104:in `block (3 levels) in \u003ctop (required)\u003e'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":"da83d2a1454918882bf10d1fc6f81942"}
I was able to get a working configuration, here's a screenshot of the logs working in the dashboard
Description
The Gitlab container is definitely outputting a variety of logs and the formatting is pretty consistent, which is nice. I'm not sure exactly how to filter on just logs coming from the
exceptions_json.logfile.It seems that the starting point is a line that explains what kind of log is being presented, then the actual log message(s) appear afterwards.
There were some small changes to allow the logs to show up in the Error Logs panel, but the filtering isn't quite there. Had to add the
instancelabel to the filtering/selector part and then update theclusterlabel'sallValuesto be.+. Doing this, all logs from thegitlabcontainer show up in the panel.If there's a path for filtering things that make sense I would love help understanding how to make it happen.
Log output from the container:
I was able to get a working configuration, here's a screenshot of the logs working in the dashboard
Changes