kubeflow / metadata

Repository for assets related to Metadata.
Apache License 2.0
121 stars 67 forks source link

metadata-grpc-deployment pod is not working in kf 1.2.0 , it‘ s status is CrashLoopBackOff #259

Closed ssdst closed 3 years ago

ssdst commented 3 years ago

describe is:

Name:         metadata-grpc-deployment-577c67c96f-f5bn7
Namespace:    kubeflow
Priority:     0
Node:         k8s-node2/192.168.2.145
Start Time:   Thu, 03 Jun 2021 04:42:10 -0400
Labels:       component=grpc-server
              kustomize.component=metadata
              pod-template-hash=577c67c96f
Annotations:  cni.projectcalico.org/podIP: 172.16.3.38/32
              cni.projectcalico.org/podIPs: 172.16.3.38/32
              sidecar.istio.io/inject: false
Status:       Running
IP:           172.16.3.38
IPs:
  IP:           172.16.3.38
Controlled By:  ReplicaSet/metadata-grpc-deployment-577c67c96f
Containers:
  container:
    Container ID:  docker://3fdb311417d172faa56b78bd885b172fa0050ecd60cdaa765561eea47cb57653
    Image:         gcr.io/tfx-oss-public/ml_metadata_store_server:v0.21.1
    Image ID:      docker://sha256:c2d7972571213a9fb6e760bcd5230c7766aa54fe4189fffd3be80681c8265efc
    Port:          8080/TCP
    Host Port:     0/TCP
    Command:
      /bin/metadata_store_server
    Args:
      --grpc_port=$(METADATA_GRPC_SERVICE_PORT)
      --mysql_config_host=metadata-db
      --mysql_config_database=$(MYSQL_DATABASE)
      --mysql_config_port=$(MYSQL_PORT)
      --mysql_config_user=$(MYSQL_USER_NAME)
      --mysql_config_password=$(MYSQL_ROOT_PASSWORD)
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    139
      Started:      Thu, 03 Jun 2021 04:52:54 -0400
      Finished:     Thu, 03 Jun 2021 04:52:54 -0400
    Ready:          False
    Restart Count:  7
    Environment Variables from:
      metadata-db-parameters   ConfigMap  Optional: false
      metadata-db-secrets      Secret     Optional: false
      metadata-grpc-configmap  ConfigMap  Optional: false
    Environment:               <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-mkgpd (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-mkgpd:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-mkgpd
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  13m                   default-scheduler  Successfully assigned kubeflow/metadata-grpc-deployment-577c67c96f-f5bn7 to k8s-node2
  Normal   Pulled     12m (x5 over 13m)     kubelet            Container image "gcr.io/tfx-oss-public/ml_metadata_store_server:v0.21.1" already present on machine
  Normal   Created    12m (x5 over 13m)     kubelet            Created container container
  Normal   Started    12m (x5 over 13m)     kubelet            Started container container
  Warning  BackOff    3m26s (x47 over 13m)  kubelet            Back-off restarting failed container

log is:

2021-06-03 08:58:01.880783: F ml_metadata/metadata_store/metadata_store_server_main.cc:219] Non-OK-status: status status: Internal: mysql_real_connect failed: errno: 1130, erro
│ stream closed

so is there anlyone else know how can i solve the problem ,thanks a lot

ssdst commented 3 years ago

forgive my poor english