zilliztech / milvus-helm

Apache License 2.0
54 stars 41 forks source link

[milvus] Change kafka as default MQ #101

Closed dongjiang1989 closed 2 months ago

dongjiang1989 commented 2 months ago

What this PR does / why we need it:

Change kafka as default MQ.

Originally posted by https://milvus.io/tools/sizing

1 Million of vectors, 128 Dimensions with 512MB segment

Pulsar: image

kafka: image

Pulsar kafka
CPU 40Core 32Core
Memory 167GB 112GB
SSD 240GB 150GB
Disk 70GB 70GB

Checklist

[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

sre-ci-robot commented 2 months ago

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: dongjiang1989 To complete the pull request process, please assign zwd1208 after the PR has been reviewed. You can assign the PR to them by writing /assign @zwd1208 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files: - **[OWNERS](https://github.com/zilliztech/milvus-helm/blob/master/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
sre-ci-robot commented 2 months ago

Welcome @dongjiang1989! It looks like this is your first PR to zilliztech/milvus-helm 🎉

haorenfsa commented 2 months ago

Thank you @dongjiang1989 for bring this out. But it's unlikely we change the default MQ, because many people still uses pulsar and will otherwise be affected next time they do update.

haorenfsa commented 2 months ago

By the way, if you have more data or collections or instances, pulsar usually cost less in our practice.

dongjiang1989 commented 2 months ago

By the way, if you have more data or collections or instances, pulsar usually cost less in our practice.

Thanks. In my business scenario, using kafka lower usage cost (muti instances in one Kubernetes Cluster). Is there any chance to kafka as default MQ?

haorenfsa commented 2 months ago

@dongjiang1989 Sry, it's a no.

LoveEachDay commented 2 months ago

@dongjiang1989 You can manually set kafka as your preferences via simple --set or specify in values.yaml. We'd like to keep pulsar as the default option for milvus cluster.