search

cds-snc / notification-planning-core

Project planning for GC Notify Core Team
0 stars 0 forks source link

FluentBit memory/streaming configuration #171

Open jimleroyer opened 1 year ago

jimleroyer commented 1 year ago

Describe the bug

Because FluentBit has faulty multi-line configuration, we are at risk of accumulating chunks of logs and stop emission of logs to CloudWatch.

Bug Severity

See examples in the documentation

SEV-2 Major

To Reproduce

Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior

A clear and concise description of what you expected to happen.

Impact

Describe overall user/system impact to consider when prioritizing this issue.

If applicable

Impact on Notify users:

Impact on Recipients:

Impact on Notify team:

Additional context

Add any other context about the problem here.

ben851 commented 1 year ago

Working on creating prometheus stats for fluent bit so we can better troubleshoot

jimleroyer commented 1 year ago

Ben worked on enabling chunk metrics for fluent-bit for cloudwatch.

sastels commented 1 year ago

work progressing. issues getting stats into correct format for prometheus.

sastels commented 1 year ago
ben851 commented 1 year ago

PR merged to staging in review for testing.

ben851 commented 1 year ago

PR did not work, reverted on Friday. Back to the drawing board!

ben851 commented 10 months ago

Closing in favour of #221