rdpanek commented 2 years ago

Documentation https://canarytrace.com/docs/guides/listener
All thresholds in a rules are and all internal checks are automatically reported into c.listener.events-*
Higher score is better than lower score.

How it works

After starting the listener-agent, several processes are started one by one. Selftests so-called internal checks next step is evaluate stored data in a Elasticsearch from Canarytrace (synthetics) and RUM so-called rules and the last process are autonomous reports so-called overviews. This is a lifecycle each run of the listener-agent.

First process are an internal checks, which are helpful test health for the Canarytrace toolset and Elasticsearch. Bad results and warnings are reported by their score into via a different reporters.

Elasticsearch indices The listener-agent create and use these indices:

c.listener the latest date of the listener processes / uninteresting for you.
c.listener.events-* is index with results of evaluates from all processes. This index is the place of truth, is interesting for you a your additional tools ( e.g. zabbix ) can use data from this index for other things.
c.listener.queue-* contains current opened defects / errors / exceeded rules.

Internal checks

First process are an internal checks, which are helpful test health for the Canarytrace toolset and Elasticsearch. Bad results and warnings are reported by their score into a different reporters.

Elasticsearch

healthCheck evaluate state of Elasticsearch yellow or red and count of pending tasks.
nodesCheck evaluate non-functional metrics for all nodes heapPercent and diskUsedPercent

Canarytrace

canaryIsLive evaluate if exists some data in c.report-* index from Canarytrace in the last 5 min.
lifecycle evaluate if exist some error type ContainerError in c.report-* index.

Data Manager

dataManager evaluate if exist c.dm index. This index is used by Data Manager for save timestamp of the latest run. If this index missing, it means, that Data Manager not running or doesn't deployed and used.

Rules

The Listener agent contains three types of the rules in a files:

rules/internal.yaml rules for analyzing data from Canarytrace ( E2E testing and synthetics monitoring ). These rules are defined from us and we frequently release updates. We analyze a lot of data from all Elasticsearch indices.
rules/rum.yaml rules for analyzing data from RUM. These rules are defined from us, we frequently release updates and these rules analyze c.rum* indices..
rules/custom.yaml this file is a right place for your business rules. You can analyze any index in your Elasticsearch e.g. c.report-* for your rule / login into our system must be faster than 1s.

Overviews

Overviews are autonomous reports. The listener-agent autonomously evaluates the severity of problems and sends a reminder report or overview with more informations.

emergencyOverview sending notice in case, that internal-checks with score lower than 30 still failing. This notice is send on emergency e-mail. You must set environment variable EMAIL_EMERGENCY
stage1 sending notice in case, that index c.listener.queue-* contains errors with score lower than 30 and if is repetition.

Events

The listener-agent save results from all processes into index c.listener.events-*. Each item in this index contains additional informations.

Filtering You can filtering events by label rule e.g. internal-check, close-old-errors, contains, match and range or by label type e.g. slack-internal or events

Reporters

Important, how reporters works 1). Report is send immediately only in case, that score is lower than 31 and that this type of rule exceeded isn't in c.listener.queue 2). All rule exceeded are automatically stored into c.listener.events-* and into c.listener.queue-* Elasticsearch indices regardless of value the score. 3). If rule exceeded again, report isn't send but label of repetition exceeded rule stored in c.listener.queue-* is updated.

Types Results of the listener-agent processes are reported via so-called reporters. In this time are available these types:

events results are stored into c.listener.events-*
slack minireport is send via hook into your Slack channel.
slack-internal minireport is send via hook into your Slack channel. You can use different hook. This variant is for cricital events e.g. problem with Elasticsearch and should be watched.
email this reporter is used for sending an overview and for sending an emergency notices.

Reporters syntax

reporters:
    - type: slack
       message: Page is complete loaded."
    - type: slack-internal
       message: Page is complete loaded."
    - type: email
       message: "An error occurred while checking."
       recipients:
       - 'rdpanek@canarytrace.com'
       - 'verca@canarytrace.com'

Rule analyzator

Rule analyzator is process which each of rule transform into Elasticsearch query, send query into Elasticsearch and analyze response by rule.

rdpanek commented 1 year ago

Score table

Canarytrace toolkit use this score tables

Description	Score	Color
needs fix!	0-30	red
needs improvement!	31-70	orange
good job!	71-100	green

This score table is used in a rule definitions, for coloring a reporters and for overview reports. By score is controlled alerting too.
Score define impact on a real user from 0 = much poor to 100 percent = excellent.

rdpanek commented 1 year ago

Internal Rules

last rev. quay.io/canarytrace/listener-agent:1.41 source: rules/internal.yaml

Title	Index	Condition	Count / hour	Score
Load time needs improvement	c.performance-entries	> 3000ms	5	40
Load time is poor	c.performance-entries	> 5000ms	5	20
Higher response time.	c.performance-entries	> 3000ms	10	40
Performance Score is poor	c.audit	0 - 49	3	20
Performance Score needs improvement	c.audit	50 - 89	3	60
FCP needs improvement	c.audit	1800 - 3000	3	40
FCP is poor	c.audit	> 3000	3	20
LCP needs improvement (CoreWebVitals).	c.audit	> 2500ms	5	40
LCP is poor (CoreWebVitals).	c.audit	> 4000ms	5	40
CLS needs improvement (CoreWebVitals)	c.audit	> 0.1	5	40
CLS is poor (CoreWebVitals)	c.audit	> 0.25	5	30
TBT needs improvement	c.audit	200 - 600	3	40
TBT is poor	c.audit	> 600	3	20
TTI needs improvement	c.audit	> 3900ms	5	40
TTI is poor	c.audit	> 7300ms	5	20
Response with Javascript must contains gzip or brotli compression.	c.request-log	gzip or br missing in headers.content-encoding	10	40
Responses with CSS files must contains gzip or brotli compression.	c.request-log	gzip or br missing in headers.content-encoding	10	40
Response code 400	r.request-log	>= 400	5	80
Response code 500	r.request-log	>= 500	5	20
Failed check your page!	c.report	test step failed	2	10

rdpanek commented 1 year ago

RUM Rules

last rev. quay.io/canarytrace/listener-agent:1.41 source: rules/rum.yaml

Title	Index	Condition	Count / hour	Score
LCP needs improvement (RUM)	c.rum.metrics	2500ms - 4000ms	2	40
LCP is poor (RUM)	c.rum.metrics	>= 4000ms	2	20
CLS needs improvement (RUM)	c.rum.metrics	0.1 - 0.25	5	40
CLS is poor (RUM)	c.rum.metrics	>= 0.25	5	30
FID needs improvement (RUM)	c.rum.metrics	100 - 300	2	20
FID is poor (RUM)	c.rum.metrics	>= 300	5	30
FCP needs improvement (RUM)	c.rum.metrics	1800 - 3000	3	40
FCP is poor (RUM)	c.rum.metrics	>= 3000	3	20
TTFB needs improvement (RUM)	c.rum.metrics	400 - 800	3	40
TTFB is poor (RUM)	c.rum.metrics	>= 800	3	20

rdpanek commented 1 year ago

Custom Rules

last rev. quay.io/canarytrace/listener-agent:1.41 source: rules/custom.yaml

This file contains only example rule.

rdpanek commented 1 year ago

Internal checks

Reporters internal-slack Internal checks are run when Listener agent start and before rules check.

Title	Score
Elasticsearch health check: yellow	50
Elasticsearch health check: red	0
Elasticsearch nodes: java heap is higher than 80%	50
Elasticsearch nodes: java heap is higher than 85%	0
Elasticsearch nodes: disk used percentage is higher than 85%	30
Canarytrace health check: missing data	0
Canarytrace lifecycle errors	30
DataManager exist: Not found.	31

rdpanek commented 1 year ago

Rules syntax

The listener-agent load and parsing a rules from files rules/internal.yaml, rules/rum.yaml and rules/custom.yaml. You can use one from three type of rules. Each rule type has different syntax and different use case.
All these files with rules are inside into the Docker image with listener-agent. If you want change some value or some rule or if you want delete or add a new rule, you can override these files with your own set of rules. Just create a config-map and deploy them into Kubernetes.

Example of one rule definition

This rule is type range, searching items with a label loadEventEnd with value between 3000 and 5000 ms in c.performance-entries index.
If the listener-agent finds minimal 5 items, automatically save event into c.listener.events index and send report into slack.
For creating report body are used additional labels loadEventEnd and timestamp.

  - type: range
    title: "Load time needs improvement"
    index: c.performance-entries
    filter:
    - field: 'loadEventEnd'
      operator: 'gte'
      value: 3000
    - field: 'loadEventEnd'
      operator: 'lt'
      value: 5000
    min: 5
    score: 40
    reportLabels:
    - 'loadEventEnd'
    - 'timestamp'
    reporters:
    - type: slack
      message: "Page is complete loaded."

rdpanek commented 1 year ago

rule type `range`

Example rule

This rule evaluate if exists minimal 5 items in c.performance-entries index and if these items has label loadEventEnd with value between 3000 and 5000ms

rules:
  - type: range
    title: "Load time needs improvement"
    index: c.performance-entries
    filter:
    - field: 'loadEventEnd'
      operator: 'gte'
      value: 3000
    - field: 'loadEventEnd'
      operator: 'lt'
      value: 5000
    min: 5
    score: 40
    reportLabels:
    - 'loadEventEnd'
    - 'timestamp'
    reporters:
    - type: slack
      message: "Page is complete loaded."

Mandatory parts

title name of this rule definition
index Elasticsearch labels
score define score by score table https://github.com/canarytrace/documentation/issues/110#issuecomment-1288030522
reporters are list of reporters type, which are you want use for sending notice about the rule exceeded. You must use at least one reporter type and you can combine them. E.g. only slack or ( slack and email )

Optional parts

filter is array of filters. Each filter must contains:
- field name of field from Elasticsearch index
- operator can contains value lt, lte, gt and gte and
- value
- min this rule will be true only in case, that was found minimal min items in Elasticsearch index. If is not min used, will be the listener-agent report all findings.

rdpanek commented 1 year ago

rule type `contains`

Example rule

This rule evaluate if exists minimal 10 items in c.request-log index and if these items has field response.headers.content-type with value css plus field response.headers.content-encoding must not have values gzip and br

rules:
  - type: contains
    title: "Responses with CSS files must contains gzip or brotli compression."
    index: c.request-log
    field: response.headers.content-type
    value: 'css'
    expression:
      field: 'response.headers.content-encoding'
      operator: must_not
      values:
      - 'gzip'
      - 'br'
    min: 10
    score: 40
    reportLabels:
    - 'url'
    - 'timestamp'
    reporters:
    - type: slack
      message: "Use Brotli for plain text compression."

Mandatory parts

title name of this rule definition
index Elasticsearch labels
score define score by score table https://github.com/canarytrace/documentation/issues/110#issuecomment-1288030522
reporters are list of reporters type, which are you want use for sending notice about the rule exceeded. You must use at least one reporter type and you can combine them. E.g. only slack or ( slack and email )
field items must contains this field
value items with field must contains exactly value
expression is a subset of conditions
- operator can contains must or must_not
- values is an array with with strings. Each items are and.

Optional parts

filter is array of filters. Each filter must contains:
- field name of field from Elasticsearch index
- operator can contains value lt, lte, gt and gte and
- value
- min this rule will be true only in case, that was found minimal min items in Elasticsearch index. If is not min used, will be the listener-agent report all findings.

rdpanek commented 1 year ago

rule type `match`

Example rule

This rule evaluate if exists minimal 2 items in c.report-* index and if these items has field passed with value false.

rules:
  - type: match
    title: "Failed check your page!"
    index: c.report
    field: passed
    operator: must
    expected: false
    min: 2
    score: 10
    reportLabels:
    - 'fullTitle'
    - 'timestamp'
    reporters:
    - type: slack
      message: "An error occurred while checking."

Mandatory parts

title name of this rule definition
index Elasticsearch labels
score define score by score table https://github.com/canarytrace/documentation/issues/110#issuecomment-1288030522
reporters are list of reporters type, which are you want use for sending notice about the rule exceeded. You must use at least one reporter type and you can combine them. E.g. only slack or ( slack and email )
field items must contains this field
operator you can use must or must_not
value items with field must contains exactly value

Optional parts

min this rule will be true only in case, that was found minimal min items in Elasticsearch index. If is not min used, will be the listener-agent report all findings.

canarytrace / documentation

Rules, Internal Checks, Scores & Reporters #110

How it works

Internal checks

Rules

Overviews

Events

Reporters

Rule analyzator

Score table

Internal Rules

RUM Rules

Custom Rules

Internal checks

Rules syntax

rule type `range`

rule type `contains`

rule type `match`

canarytrace / documentation

Rules, Internal Checks, Scores & Reporters #110

How it works

Internal checks

Rules

Overviews

Events

Reporters

Rule analyzator

Score table

Internal Rules

RUM Rules

Custom Rules

Internal checks

Rules syntax

rule type range

rule type contains

rule type match

rule type `range`

rule type `contains`

rule type `match`