Closed rdpanek closed 1 year ago
Canarytrace toolkit use this score tables
Description | Score | Color |
---|---|---|
needs fix! | 0-30 | red |
needs improvement! | 31-70 | orange |
good job! | 71-100 | green |
last rev.
quay.io/canarytrace/listener-agent:1.41
source: rules/internal.yaml
Title | Index | Condition | Count / hour | Score |
---|---|---|---|---|
Load time needs improvement | c.performance-entries | > 3000ms | 5 | 40 |
Load time is poor | c.performance-entries | > 5000ms | 5 | 20 |
Higher response time. | c.performance-entries | > 3000ms | 10 | 40 |
Performance Score is poor | c.audit | 0 - 49 | 3 | 20 |
Performance Score needs improvement | c.audit | 50 - 89 | 3 | 60 |
FCP needs improvement | c.audit | 1800 - 3000 | 3 | 40 |
FCP is poor | c.audit | > 3000 | 3 | 20 |
LCP needs improvement (CoreWebVitals). | c.audit | > 2500ms | 5 | 40 |
LCP is poor (CoreWebVitals). | c.audit | > 4000ms | 5 | 40 |
CLS needs improvement (CoreWebVitals) | c.audit | > 0.1 | 5 | 40 |
CLS is poor (CoreWebVitals) | c.audit | > 0.25 | 5 | 30 |
TBT needs improvement | c.audit | 200 - 600 | 3 | 40 |
TBT is poor | c.audit | > 600 | 3 | 20 |
TTI needs improvement | c.audit | > 3900ms | 5 | 40 |
TTI is poor | c.audit | > 7300ms | 5 | 20 |
Response with Javascript must contains gzip or brotli compression. | c.request-log | gzip or br missing in headers.content-encoding | 10 | 40 |
Responses with CSS files must contains gzip or brotli compression. | c.request-log | gzip or br missing in headers.content-encoding | 10 | 40 |
Response code 400 | r.request-log | >= 400 | 5 | 80 |
Response code 500 | r.request-log | >= 500 | 5 | 20 |
Failed check your page! | c.report | test step failed | 2 | 10 |
last rev.
quay.io/canarytrace/listener-agent:1.41
source: rules/rum.yaml
Title | Index | Condition | Count / hour | Score |
---|---|---|---|---|
LCP needs improvement (RUM) | c.rum.metrics | 2500ms - 4000ms | 2 | 40 |
LCP is poor (RUM) | c.rum.metrics | >= 4000ms | 2 | 20 |
CLS needs improvement (RUM) | c.rum.metrics | 0.1 - 0.25 | 5 | 40 |
CLS is poor (RUM) | c.rum.metrics | >= 0.25 | 5 | 30 |
FID needs improvement (RUM) | c.rum.metrics | 100 - 300 | 2 | 20 |
FID is poor (RUM) | c.rum.metrics | >= 300 | 5 | 30 |
FCP needs improvement (RUM) | c.rum.metrics | 1800 - 3000 | 3 | 40 |
FCP is poor (RUM) | c.rum.metrics | >= 3000 | 3 | 20 |
TTFB needs improvement (RUM) | c.rum.metrics | 400 - 800 | 3 | 40 |
TTFB is poor (RUM) | c.rum.metrics | >= 800 | 3 | 20 |
last rev.
quay.io/canarytrace/listener-agent:1.41
source: rules/custom.yaml
This file contains only example rule.
Reporters
internal-slack
Internal checks are run when Listener agent start and before rules check.
Title | Score |
---|---|
Elasticsearch health check: yellow | 50 |
Elasticsearch health check: red | 0 |
Elasticsearch nodes: java heap is higher than 80% | 50 |
Elasticsearch nodes: java heap is higher than 85% | 0 |
Elasticsearch nodes: disk used percentage is higher than 85% | 30 |
Canarytrace health check: missing data | 0 |
Canarytrace lifecycle errors | 30 |
DataManager exist: Not found. | 31 |
rules/internal.yaml
, rules/rum.yaml
and rules/custom.yaml
. You can use one from three type of rules. Each rule type has different syntax and different use case. Example of one rule definition
range
, searching items with a label loadEventEnd
with value between 3000 and 5000 ms in c.performance-entries
index.c.listener.events
index and send report into slack.loadEventEnd
and timestamp
. - type: range
title: "Load time needs improvement"
index: c.performance-entries
filter:
- field: 'loadEventEnd'
operator: 'gte'
value: 3000
- field: 'loadEventEnd'
operator: 'lt'
value: 5000
min: 5
score: 40
reportLabels:
- 'loadEventEnd'
- 'timestamp'
reporters:
- type: slack
message: "Page is complete loaded."
range
Example rule
This rule evaluate if exists minimal 5 items in
c.performance-entries
index and if these items has labelloadEventEnd
with value between3000
and5000
ms
rules:
- type: range
title: "Load time needs improvement"
index: c.performance-entries
filter:
- field: 'loadEventEnd'
operator: 'gte'
value: 3000
- field: 'loadEventEnd'
operator: 'lt'
value: 5000
min: 5
score: 40
reportLabels:
- 'loadEventEnd'
- 'timestamp'
reporters:
- type: slack
message: "Page is complete loaded."
Mandatory parts
title
name of this rule definitionindex
Elasticsearch labelsscore
define score by score table https://github.com/canarytrace/documentation/issues/110#issuecomment-1288030522reporters
are list of reporters type, which are you want use for sending notice about the rule exceeded. You must use at least one reporter type and you can combine them. E.g. only slack
or ( slack
and email
)Optional parts
filter
is array of filters. Each filter must contains:
field
name of field from Elasticsearch indexoperator
can contains value lt
, lte
, gt
and gte
andvalue
min
this rule will be true only in case, that was found minimal min
items in Elasticsearch index. If is not min
used, will be the listener-agent report all findings.contains
Example rule
This rule evaluate if exists minimal 10 items in
c.request-log
index and if these items has fieldresponse.headers.content-type
with valuecss
plus fieldresponse.headers.content-encoding
must not have valuesgzip
andbr
rules:
- type: contains
title: "Responses with CSS files must contains gzip or brotli compression."
index: c.request-log
field: response.headers.content-type
value: 'css'
expression:
field: 'response.headers.content-encoding'
operator: must_not
values:
- 'gzip'
- 'br'
min: 10
score: 40
reportLabels:
- 'url'
- 'timestamp'
reporters:
- type: slack
message: "Use Brotli for plain text compression."
Mandatory parts
title
name of this rule definitionindex
Elasticsearch labelsscore
define score by score table https://github.com/canarytrace/documentation/issues/110#issuecomment-1288030522reporters
are list of reporters type, which are you want use for sending notice about the rule exceeded. You must use at least one reporter type and you can combine them. E.g. only slack
or ( slack
and email
)field
items must contains this fieldvalue
items with field
must contains exactly value expression
is a subset of conditions
operator
can contains must
or must_not
values
is an array with with strings. Each items are and
.Optional parts
filter
is array of filters. Each filter must contains:
field
name of field from Elasticsearch indexoperator
can contains value lt
, lte
, gt
and gte
andvalue
min
this rule will be true only in case, that was found minimal min
items in Elasticsearch index. If is not min
used, will be the listener-agent report all findings.match
Example rule
This rule evaluate if exists minimal 2 items in
c.report-*
index and if these items has fieldpassed
with valuefalse
.
rules:
- type: match
title: "Failed check your page!"
index: c.report
field: passed
operator: must
expected: false
min: 2
score: 10
reportLabels:
- 'fullTitle'
- 'timestamp'
reporters:
- type: slack
message: "An error occurred while checking."
Mandatory parts
title
name of this rule definitionindex
Elasticsearch labelsscore
define score by score table https://github.com/canarytrace/documentation/issues/110#issuecomment-1288030522reporters
are list of reporters type, which are you want use for sending notice about the rule exceeded. You must use at least one reporter type and you can combine them. E.g. only slack
or ( slack
and email
)field
items must contains this fieldoperator
you can use must
or must_not
value
items with field
must contains exactly value Optional parts
min
this rule will be true only in case, that was found minimal min
items in Elasticsearch index. If is not min
used, will be the listener-agent report all findings.
c.listener.events-*
How it works
After starting the listener-agent, several processes are started one by one. Selftests so-called
internal checks
next step is evaluate stored data in a Elasticsearch from Canarytrace (synthetics) and RUM so-calledrules
and the last process are autonomous reports so-calledoverviews
. This is a lifecycle each run of the listener-agent.First process are an internal checks, which are helpful test health for the Canarytrace toolset and Elasticsearch. Bad results and warnings are reported by their score into via a different reporters.
Elasticsearch indices The listener-agent create and use these indices:
c.listener
the latest date of the listener processes / uninteresting for you.c.listener.events-*
is index with results of evaluates from all processes. This index is the place of truth, is interesting for you a your additional tools ( e.g. zabbix ) can use data from this index for other things.c.listener.queue-*
contains current opened defects / errors / exceeded rules.Internal checks
First process are an internal checks, which are helpful test health for the Canarytrace toolset and Elasticsearch. Bad results and warnings are reported by their score into a different reporters.
Elasticsearch
healthCheck
evaluate state of Elasticsearchyellow
orred
and count of pending tasks.nodesCheck
evaluate non-functional metrics for all nodesheapPercent
anddiskUsedPercent
Canarytrace
canaryIsLive
evaluate if exists some data inc.report-*
index from Canarytrace in the last 5 min.lifecycle
evaluate if exist some error typeContainerError
inc.report-*
index.Data Manager
dataManager
evaluate if existc.dm
index. This index is used by Data Manager for save timestamp of the latest run. If this index missing, it means, that Data Manager not running or doesn't deployed and used.Rules
The Listener agent contains three types of the rules in a files:
rules/internal.yaml
rules for analyzing data from Canarytrace ( E2E testing and synthetics monitoring ). These rules are defined from us and we frequently release updates. We analyze a lot of data from all Elasticsearch indices.rules/rum.yaml
rules for analyzing data from RUM. These rules are defined from us, we frequently release updates and these rules analyzec.rum*
indices..rules/custom.yaml
this file is a right place for your business rules. You can analyze any index in your Elasticsearch e.g.c.report-*
for your rule / login into our system must be faster than 1s.Overviews
Overviews are autonomous reports. The listener-agent autonomously evaluates the severity of problems and sends a reminder report or overview with more informations.
emergencyOverview
sending notice in case, thatinternal-checks
with score lower than30
still failing. This notice is send on emergency e-mail. You must set environment variableEMAIL_EMERGENCY
stage1
sending notice in case, that indexc.listener.queue-*
contains errors with score lower than30
and if is repetition.Events
The listener-agent save results from all processes into index
c.listener.events-*
. Each item in this index contains additional informations.Filtering You can filtering events by label
rule
e.g.internal-check
,close-old-errors
,contains
,match
andrange
or by labeltype
e.g.slack-internal
orevents
Reporters
Important, how reporters works 1). Report is send immediately only in case, that score is lower than
31
and that this type of rule exceeded isn't inc.listener.queue
2). All rule exceeded are automatically stored intoc.listener.events-*
and intoc.listener.queue-*
Elasticsearch indices regardless of value the score. 3). If rule exceeded again, report isn't send but label ofrepetition
exceeded rule stored inc.listener.queue-*
is updated.Types Results of the listener-agent processes are reported via so-called
reporters
. In this time are available these types:events
results are stored intoc.listener.events-*
slack
minireport is send via hook into your Slack channel.slack-internal
minireport is send via hook into your Slack channel. You can use different hook. This variant is for cricital events e.g. problem with Elasticsearch and should be watched.email
this reporter is used for sending an overview and for sending an emergency notices.Reporters syntax
Rule analyzator
Rule analyzator is process which each of rule transform into Elasticsearch query, send query into Elasticsearch and analyze response by rule.