elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.67k stars 24.66k forks source link

triggered_watches is missing in 7.17.3 #87493

Open TomonoriSoejima opened 2 years ago

TomonoriSoejima commented 2 years ago

Elasticsearch Version

7.17.3

Installed Plugins

No response

Java Version

bundled

OS Version

OS X

Problem Description

I just noticed after running a test watch and looking at the cluster log, I came across this error. This is a super fresh install and I did POST _watcher/_stop and POST _watcher/_start, but it did not help. I eventually executed PUT .triggered_watches to fix it.


[2022-06-08T14:07:03,776][INFO ][o.e.x.w.WatcherService   ] [surfer.local] reloading watcher, reason [new local watcher shard allocation ids], cancelled [0] queued tasks
[2022-06-08T14:07:07,171][INFO ][o.e.x.w.a.l.ExecutableLoggingAction] [surfer.local] 124.35.92.216

[2022-06-08T14:07:07,175][ERROR][o.e.x.w.Watcher          ] [surfer.local] triggered watches could not be deleted [status-check_97634a68-3aa3-46a0-b2ca-c51d294b9853-2022-06-08T05:07:07.072703Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]]
[2022-06-08T14:08:17,043][INFO ][o.e.x.w.a.l.ExecutableLoggingAction] [surfer.local] 124.35.92.216

[2022-06-08T14:08:17,044][ERROR][o.e.x.w.Watcher          ] [surfer.local] triggered watches could not be deleted [status-check_33aecb32-c92d-40ab-acda-3674deba8091-2022-06-08T05:08:17.028228Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]]
[2022-06-08T14:12:48,525][INFO ][o.e.x.w.WatcherService   ] [surfer.local] stopping watch service, reason [watcher manually marked to shutdown by cluster state update]
[2022-06-08T14:12:48,526][INFO ][o.e.x.w.WatcherLifeCycleService] [surfer.local] watcher has stopped
[2022-06-08T14:13:08,851][INFO ][o.e.x.w.a.l.ExecutableLoggingAction] [surfer.local] 124.35.92.216

[2022-06-08T14:13:08,854][ERROR][o.e.x.w.Watcher          ] [surfer.local] triggered watches could not be deleted [status-check_9e380978-d354-43cb-a769-e9263fa20c9b-2022-06-08T05:13:08.834507Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]]
[2022-06-08T14:13:53,797][INFO ][o.e.c.m.MetadataCreateIndexService] [surfer.local] [.triggered_watches] creating index, cause [api], templates [], shards [1]/[1]
[2022-06-08T14:13:53,799][INFO ][o.e.c.r.a.AllocationService] [surfer.local] updating number_of_replicas to [0] for indices [.triggered_watches]
[2022-06-08T14:13:56,995][INFO ][o.e.x.w.a.l.ExecutableLoggingAction] [surfer.local] 124.35.92.216

[2022-06-08T14:14:20,051][INFO ][o.e.x.w.a.l.ExecutableLoggingAction] [surfer.local] 124.35.92.216

[2022-06-08T14:14:38,424][INFO ][o.e.x.w.a.l.ExecutableLoggingAction] [surfer.local] 124.35.92.216

[2022-06-08T14:30:00,378][INFO ][o.e.x.w.a.l.ExecutableLoggingAction] [surfer.local] 124.35.92.216

[2022-06-08T14:30:00,432][INFO ][o.e.c.m.MetadataCreateIndexService] [surfer.local] [.watcher-history-13-2022.06.08] creating index, cause [auto(bulk api)], templates [.watch-history-13], shards [1]/[0]
[2022-06-08T14:30:00,731][INFO ][o.e.x.i.IndexLifecycleTransition] [surfer.local] moving index [.watcher-history-13-2022.06.08] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [watch-history-ilm-policy]
[2022-06-08T14:30:01,511][INFO ][o.e.c.m.MetadataMappingService] [surfer.local] [.watcher-history-13-2022.06.08/VgzVAy4-SgmWnmUPXqGBTQ] update_mapping [_doc]
[2022-06-08T14:31:30,127][INFO ][o.e.x.w.a.l.ExecutableLoggingAction] [surfer.local] 124.35.92.216````

### Logs (if relevant)

_No response_
elasticmachine commented 2 years ago

Pinging @elastic/es-data-management (Team:Data Management)