search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30k stars 3.79k forks source link

kubernetes: Evaluate local persistent volume support in v1.10 #23706

Open a-robinson opened 6 years ago

a-robinson commented 6 years ago

v1.10 is expected to be released on March 21, and it notably includes the promotion of LocalPersistentVolume to beta, meaning it will be available by default on all new kubernetes clusters. It's a good opportunity to eliminate our dependency on network-attached volumes and run better on bare metal clusters, but would need thorough testing given how new it is.

cc @mberhault @bobvawter @kannanlakshmi

Jira issue: CRDB-5809

msau42 commented 6 years ago

BTW I'm the lead for the feature in Kubernetes, would be happy to help answer questions and always looking for feedback!

a-robinson commented 6 years ago

Thanks @msau42! We'll let you know if anything comes up once we get started.

victorggsimida commented 5 years ago

@a-robinson hi i also want to use k8s with local storage to deploy cockroach db.

I have some questions

  1. the cockroach db statefulset secure yaml file is suitable for k8s with local storage ?

  2. do i need to rewrite the deploy yaml before create -f

  3. any bugs or problem or performance descrese with k8s local storage?

a-robinson commented 5 years ago
  1. Yes, although you'll have to make the few changes necessary in order to tell Kubernetes to use local volumes - https://kubernetes.io/docs/concepts/storage/volumes/#local
  2. Yes, the changes outlined in the docs linked above.
  3. Not that I know of, but as you can probably tell from the lack of updates on this issue, we haven't tried to vet them yet. If you try them, we (and the upstream Kubernetes folks) would love to hear about your experiences with them.
victorggsimida commented 5 years ago

@a-robinson
appreciate your feedback.~~

I will try it and if has any result , I will report here

jnicholls commented 4 years ago

Coming from little knowledge on the CockroachDB's Helm setup (I need to dig into it next), I am interested in the ability for CockroachDB to properly run on any specified storage class, including one that is backed with local persistent volumes. I'm curious if CockroachDB already is architected to handle the case when an instance/pod is rescheduled to a fresh node with an empty data volume (as would be the case with instance storage/local volumes). CockroachDB would begin replicating data to the new instance? Does the cluster just consider this to be a new identity while the old instance is considered permanently dead, given its state is lost?

I'll dig into the code a bit, but if the answer to my questions are "yes" then local persistent volumes ought to work just fine for the cases where the number of nodes that have lost their persistent state and/or are not fully in-sync remains at N - 1, where N is the replication factor. I'm interested in local persistent volume support for deploying onto a bare metal k8s cluster without using network-attached storage or distributed file systems such as Ceph/Rook, but instead using direct-attached NVMe/SSD drives.

github-actions[bot] commented 3 years ago

We have marked this issue as stale because it has been inactive for 18 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to CockroachDB!

Bessonov commented 3 years ago

adding a comment will keep it active

Bessonov commented 3 years ago

Very interested in this one. But until now the most challenging part is the disk encryption.

github-actions[bot] commented 1 year ago

We have marked this issue as stale because it has been inactive for 18 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to CockroachDB!

Bessonov commented 1 year ago

a comment will keep it active