milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
29.67k stars 2.85k forks source link

[Bug]: Attu pod restarts when changing the database #36211

Open iurii-stepanov opened 2 weeks ago

iurii-stepanov commented 2 weeks ago

Is there an existing issue for this?

Environment

- Milvus version: 2.4.10
- Attu version: 2.4.7
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka): external kafka

Yesterday I updated Milvus from 2.3.1 to 2.4.10 by changing images for Milvus, Attu and config-tool.

Current Behavior

Sometimes when I change the DB in WebIU Attu pod restarts with message in the log bellow. The thing is that collection "beauty_filtered" really isn't in database "cv_search". This collection in the "default" DB.

Expected Behavior

When I change the DB Attu pod does not restart.

Steps To Reproduce

Login to Attu
Change the BD from default to any other

Milvus Log

 GET 200 /api/v1/collections 4.417 ms @ Thu, 12 Sep 2024 06:36:52 GMT ::ffff:10.97.47.24 from undefined Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36

/app/dist/src/crons/crons.service.js:74
                    throw new Error(error);
                          ^

Error: can't find collection[database=cv_search][collection=beauty_filtered]
    at CronsService.<anonymous> (/app/dist/src/crons/crons.service.js:74:27)
    at Generator.throw (<anonymous>)
    at rejected (/app/dist/src/crons/crons.service.js:6:65)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Node.js v18.20.4
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Anything else?

Attu deployment spec:

spec:
  replicas: 1
  selector:
    matchLabels:
      app: milvus-attu
      app.kubernetes.io/instance: platform-milvus
      app.kubernetes.io/name: milvus
      component: attu
      release: platform-milvus
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: milvus-attu
        app.kubernetes.io/instance: platform-milvus
        app.kubernetes.io/name: milvus
        component: attu
        release: platform-milvus
      annotations:
        checksum/config: 406cd24a25e78dcb25870e6653055e35f8942eece05dce0a02ade82ebbc6b6cb
        kubectl.kubernetes.io/restartedAt: '2024-09-12T08:37:35Z'
        platform/env: prod
        platform/logs: enabled
        platform/system: milvus-attu
    spec:
      containers:
        - name: attu
          image: >-
            docker-hosted.artifactory/platform/base-images/milvus-attu:v2.4.7
          ports:
            - name: attu
              containerPort: 3000
              protocol: TCP
          env:
            - name: COMPONENT
              value: attu
            - name: MILVUS_URL
              value: http://platform-milvus:19530
          resources:
            limits:
              cpu: 500m
              memory: 512Mi
            requests:
              cpu: 100m
              memory: 256Mi
          livenessProbe:
            httpGet:
              path: /api/v1/healthy
              port: attu
              scheme: HTTP
            initialDelaySeconds: 90
            timeoutSeconds: 5
            periodSeconds: 30
            successThreshold: 1
            failureThreshold: 5
          readinessProbe:
            httpGet:
              path: /api/v1/healthy
              port: attu
              scheme: HTTP
            initialDelaySeconds: 90
            timeoutSeconds: 5
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 5
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
          securityContext:
            capabilities:
              drop:
                - ALL
            runAsUser: 0
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      securityContext: {}
      imagePullSecrets:
        - name: artifactory-cred-secret
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 0
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600
yanliang567 commented 2 weeks ago

/assign @shanghaikid /unassign

shanghaikid commented 2 weeks ago

please open an ticket here : https://github.com/zilliztech/attu/issues

iurii-stepanov commented 1 week ago

Done: https://github.com/zilliztech/attu/issues/648