Creation of silence in one Alertmanager takes time to reflect the changes in other 2 Alertmanagers

[vivekbny]

What did you do? Created/Expired a silence in alertmanager1 and tried to view the silence in other Alert managers

What did you expect to see? All 3 Alertmanagers should be clustered and in Sync

What did you see instead? Under which circumstances? Creation of silence in one Alertmanager takes time to reflect the changes in other Alertmanagers, approximately 3 minutes to view the changes in all 3 Alertmanagers

Environment Preprod

• Alertmanager version:

  V0.14.0

• Prometheus version:

  V 2.0

• Alertmanager configuration file:

  
  A(r0000000a)------>Peers B(r0000000b) & C(r0000000c)
  B(r0000000b)------>Peers A(r0000000a) & C(r0000000c)
  C(r0000000c)------>Peers A(r0000000a) & B(r0000000b)

alertmanager: image: prom/alertmanager:0.14.0 ports:

• mapping: "9093:9093" name: alertmanager
• mapping: "8001:8001" name: meshport
  constraints:

• [ "hostname", "CLUSTER", "localhost.aaa.net" ]

  parameters:

• key: memory-swap value: "-1" healthChecks:
• protocol: "HTTP" path: "/" gracePeriodSeconds: 300 intervalSeconds: 60 portIndex: 0 timeoutSeconds: 20 maxConsecutiveFailures: 3 args:
• "--storage.path"
• "/alertmanager"
• "--config.file"
• "/etc/alertmanager/config.yml"
• "--mesh.nickname"
• r0000000a
• "--mesh.listen-address"
• ":8001"
• "--log.level"
• info
• "--mesh.peer"
• r0000000b
• "--mesh.peer"
• r0000000c

[brian-brazil]

To be clear, does the silence not show up or the effect of the silence?

[vivekbny]

I created a silence in Alertmanager1 Trying to view the silence in Alertmanager 2 and 3 Silence is not seen in Alertmanager 2 & 3 Silence Shows up after approximately 3 minutes in Alertmanager 2&3

Same things happens with the Expiring silence scenario

[stuartnelson3]

This is expected due to the gossip protocol and the nature of eventual consistency. Hopefully time to convergence drops with the migration to the new gossip library (currently available in v0.15.0.rc0)