search

apache / skywalking

APM, Application Performance Monitoring System
https://skywalking.apache.org/
Apache License 2.0
23.93k stars 6.53k forks source link

table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later. #4938

Closed jiangxiaoqiang closed 4 years ago

jiangxiaoqiang commented 4 years ago

Please answer these questions before submitting your issue.

Question

Bug

2020-06-17 08:11:01,022 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -16539 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later. 2020-06-17 08:11:04,025 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -19542 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later. 2020-06-17 08:11:07,028 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -22545 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later. 2020-06-17 08:11:10,031 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -25548 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later. 2020-06-17 08:11:13,034 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -28551 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later. 2020-06-17 08:11:16,037 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -31554 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later. 2020-06-17 08:11:19,040 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -34557 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later. 2020-06-17 08:11:22,043 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -37560 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later. 2020-06-17 08:11:25,046 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -40563 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later. 2020-06-17 08:11:28,049 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -43566 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later. 2020-06-17 08:11:31,053 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -46570 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later. 2020-06-17 08:11:34,057 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -49574 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later. 2020-06-17 08:11:37,060 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -52577 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later. 2020-06-17 08:11:40,062 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -55579 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later. 2020-06-17 08:11:43,065 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -58582 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.

Requirement or improvement

wu-sheng commented 4 years ago

You need to run a standard startup or init-only startup once, then you could start in the on-init mode.

SxunS commented 3 years ago

@wu-sheng Thanks, it did help me.

john1337 commented 2 years ago

You need to run a standard startup or init-only startup once, then you could start in the on-init mode.

i also met this using skywalking-9.2.0,i started the whole skywalking successfully,but when the k8s cluster reboot,the oap service came up with such "table: alarm_record does not exist. OAP is running in 'no-init' mode" errors, how to fix this?

liuyujie-tx commented 2 years ago

You need to run a standard startup or init-only startup once, then you could start in the on-init mode.

i also met this using skywalking-9.2.0,i started the whole skywalking successfully,but when the k8s cluster reboot,the oap service came up with such "table: alarm_record does not exist. OAP is running in 'no-init' mode" errors, how to fix this?

I also have the same problem with skywalking-9.2.0

leon5233 commented 1 year ago

Why start oap service start and throw: table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.

leon5233 commented 1 year ago

Bug Which version of SkyWalking, OS and JRE? Skywalking 8.9.1 Elasticsearch 7.14

Which company or project?

What happen?

2020-06-17 08:11:01,022 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller -43 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.

xuyf1001 commented 1 year ago

I also have the same problem with skywalking-9.4.0.

leduy123lsdd commented 1 year ago

My steps to fix this issue if you using Helm to install skywalking

  1. Get skywalking-oap-init Job template
    • execute: helm template skywalking oci://registry-1.docker.io/apache/skywalking-helm -n skywalking > manifest.yaml
  2. Extract Job from manifest.yaml
    • in manifest.yaml, search for "kind: Job" and copy entire Job manifest to new file called: init-oap.yaml Screenshot 2023-06-28 at 09 10 00
  1. Lastly, restart oap service pods. And voila, sky walking is back.

This solution is just temporary fix and it will happen next time when k8s got restarted

kezhenxu94 commented 1 year ago

My steps to fix this issue if you using Helm to install skywalking

  1. Get skywalking-oap-init Job template
  • execute: helm template skywalking oci://registry-1.docker.io/apache/skywalking-helm -n skywalking > manifest.yaml
  1. Extract Job from manifest.yaml
  • in manifest.yaml, search for "kind: Job" and copy entire Job manifest to new file called: init-oap.yaml
Screenshot 2023-06-28 at 09 10 00

  • Delete old job if it existed: kubectl delete -f init-oap.yaml -n skywalking

  • Execute the job: kubectl create -f init-oap.yaml -n skywalking

  1. Lastly, restart oap service pods. And voila, sky walking is back.

This solution is just temporary fix and it will happen next time when k8s got restarted

No need to do so now. The init job will be executed on installation and upgrade now

leduy123lsdd commented 1 year ago

My steps to fix this issue if you using Helm to install skywalking

  1. Get skywalking-oap-init Job template
  • execute: helm template skywalking oci://registry-1.docker.io/apache/skywalking-helm -n skywalking > manifest.yaml
  1. Extract Job from manifest.yaml
  • in manifest.yaml, search for "kind: Job" and copy entire Job manifest to new file called: init-oap.yaml
Screenshot 2023-06-28 at 09 10 00
  • Delete old job if it existed: kubectl delete -f init-oap.yaml -n skywalking
  • Execute the job: kubectl create -f init-oap.yaml -n skywalking
  1. Lastly, restart oap service pods. And voila, sky walking is back.

This solution is just temporary fix and it will happen next time when k8s got restarted

No need to do so now. The init job will be executed on installation and upgrade now

the problem is when k8s restart it running into 'OAP is running in 'no-init' mode'

ghost commented 1 year ago

I just tested with the latest helm install, 4.4.0, and this is still an issue.

leduy123lsdd commented 1 year ago

@eric-futureself upgrade skywalking oap and skywalking UI to version 9.5.0 will fix this issue

anantvardhan04 commented 1 year ago

Workaround:

You can modify the chart to mimic the run of oap-init Job as an initContainer in UI deployment. This way every time the cluster reboots, when the UI pod comes up, it will create the required tables as part of the init container run, essentially performing the same task as the oap-init Job. This way OAP pods will come up since all the required tables would have been already created.

Changes:

  1. ui-deployment.yaml: Add below two lines under initContainers:

    apiVersion: apps/v1
kind: Deployment
.
.
{{- end }}
  initContainers:
{{- include "skywalking.containers.wait-for-storage" . | nindent 6 }}
{{- include "skywalking.containers.oap-init-job" . | nindent 6 }}

  2. _helpers.tpl: Add template for skywalking.containers.oap-init-job included above

    # Template for oap init job container 
{{- define "skywalking.containers.oap-init-job" -}}
- name: oap-init-job
image: {{ .Values.oap.image.repository }}:{{ required "oap.image.tag is required" .Values.oap.image.tag }}
imagePullPolicy: {{ .Values.oap.image.pullPolicy }}
env:
- name: JAVA_OPTS
value: "{{ .Values.oap.javaOpts }} -Dmode=init"
{{- include "skywalking.oap.envs.storage" . | nindent 2 }}
{{- range $key, $value :=  .Values.oap.env }}
- name: {{ $key }}
value: {{ $value | quote }}
{{- end }}
{{- end -}}
JangGyeongSik commented 1 year ago

I had the same problem, but it was solved thanks to @anantvardhan04 Thanks for the good information :)