search

opensearch-project / helm-charts

:wheel_of_dharma: A community repository for Helm Charts of OpenSearch Project.
https://opensearch.org/docs/latest/opensearch/install/helm/
Apache License 2.0
171 stars 234 forks source link

How to move from Opendistro helm to Opensearch helm #53

Open divyanshuch1 opened 3 years ago

divyanshuch1 commented 3 years ago

Hello, Can I use exactly the same helm values to deploy Opensearch that I had used to deploy Opendistro? What differences should I keep in mind. Thanks

peterzhuamazon commented 3 years ago

Migrate to helm-charts repo as that is the standalone repo for charts now, we are cleaning up helm form devops repo. @DandyDeveloper Seems like you have experience with this, probably we should write a migration guide somewhere? Thanks.

DandyDeveloper commented 3 years ago

@divyanshuch1 In short: No. No assumption should be made there and a number of changes are needed to reflect the exact older OpenDistro helm chart.

As long as you keep you PVCs and replace the STS volumeClaimTemplate to match what the previous OpenDistro statefulset created. You should be fine and will not lose the cluster state / index data.

However, you cannot do a direct helm upgrade from old to new. Not without headache.

@TheAlgo We may need to make the volumeClaimTemplate more configurable:

https://github.com/opensearch-project/opensearch-devops/blob/a2a2d8be7438a8dc5cbb300a8530f85a3784fd20/Helm/opensearch/templates/statefulset.yaml#L30

For people migrating. This name: would be expected to represent the existing claim from the older release.

We don't yet have an official migration path and in some respects, that's probably for the best.

smlx commented 3 years ago

As long as you keep you PVCs and replace the STS volumeClaimTemplate to match what the previous OpenDistro statefulset created. You should be fine and will not lose the cluster state / index data.

This is not a safe procedure as there is no guarantee that the data is consistent. The only safe method I'm aware of is to snapshot the old cluster and restore into the new cluster.

You cannot back up an Elasticsearch cluster by simply copying the data directories of all of its nodes. Elasticsearch may be making changes to the contents of its data directories while it is running; copying its data directories cannot be expected to capture a consistent picture of their contents. If you try to restore a cluster from such a backup, it may fail and report corruption and/or missing files. Alternatively, it may appear to have succeeded though it silently lost some of its data. The only reliable way to back up a cluster is by using the snapshot and restore functionality.

https://www.elastic.co/guide/en/elasticsearch/reference/7.10/snapshot-restore.html#snapshot-restore

TheAlgo commented 3 years ago

@divyanshuch1 In short: No. No assumption should be made there and a number of changes are needed to reflect the exact older OpenDistro helm chart.

As long as you keep you PVCs and replace the STS volumeClaimTemplate to match what the previous OpenDistro statefulset created. You should be fine and will not lose the cluster state / index data.

However, you cannot do a direct helm upgrade from old to new. Not without headache.

@TheAlgo We may need to make the volumeClaimTemplate more configurable:

https://github.com/opensearch-project/opensearch-devops/blob/a2a2d8be7438a8dc5cbb300a8530f85a3784fd20/Helm/opensearch/templates/statefulset.yaml#L30

For people migrating. This name: would be expected to represent the existing claim from the older release.

We don't yet have an official migration path and in some respects, that's probably for the best.

@DandyDeveloper If people really want to migrate to OpenSearch via Helm, this tweak looks good to me as well and I remember this was raised when I had raised the initial PR. We can do this

This is not a safe procedure as there is no guarantee that the data is consistent. The only safe method I'm aware of is to snapshot the old cluster and restore into the new cluster.

@smlx We have an official blog for guidance w.r.t the migration written by @saratvemulapalli . Also IMO, snapshot is the best and safest path for migration.

divyanshuch1 commented 3 years ago

@smlx @DandyDeveloper @TheAlgo, Okay Thanks. And suppose I am not migrating a running cluster, I want to deploy the opensearch on a different environment and concern is about that can I use the same helm values that I had set ( for eg; certs, oidc configurations) in "opendistro's helm" to deploy "opensearch" ? Are the fields locations changed in values yaml and what are their impact? For eg; "securityConfigSecret" field in opensearch comes under the parent field of securityConfig.config here where as in opendsitro comes under the parent filed of elasticsearch.securityConfig.config here So is there a reference for these kind of changes so that migration on the helm package level can be easier?

DandyDeveloper commented 3 years ago

This is not a safe procedure as there is no guarantee that the data is consistent. The only safe method I'm aware of is to snapshot the old cluster and restore into the new cluster.

@smlx This is subjective to the individuals experience and capabilities.

It is better to snapshot and restore, but that isn't feasible for people with 10s or 100s of terabytes of data.

I was able to migrate a askari test cluster wth very little downtime. However, again, your need to know what you're doing with both helm and opensearch.

You cannot back up an Elasticsearch cluster by simply copying the data directories of all of its nodes.

I did not suggest copy or pasting. You should replace your nodes by consuming the same pvc the old node was consuming.

DandyDeveloper commented 3 years ago

@divyanshuch1 i will try and help you out in my morning wth some details on values. They are different but I think everything from the old should work in the new with some adjustments.

If you have a relatively small cluster snapshot after bringing down your data nodes and preventing incoming document ingestion is the way to go.

smlx commented 3 years ago

It is better to snapshot and restore, but that isn't feasible for people with 10s or 100s of terabytes of data.

I just migrated a cluster with over 10T data via S3 snapshots. The procedure was quick and simple.

My point is that trying to "cross-grade" the old opendistro helm chart to the new opensearch helm chart is both slower and way more complex. It also requires deep knowledge of opensearch, helm, and kubernetes and if anything goes wrong there is no rollback procedure other than restoring from snapshots. So why not just use snapshots to upgrade in the first place?

We have an official blog for guidance w.r.t the migration written by @saratvemulapalli .

Ok, but that does not deal with upgrading the helm chart which also requires considerable delicate kubernetes surgery, such as safely detaching the PVCs and reattaching them to the new nodes.

Also IMO, snapshot is the best and safest path for migration.

It is the only path that should be recommended for the helm chart IMO.

divyanshuch1 commented 3 years ago

@divyanshuch1 i will try and help you out in my morning wth some details on values. They are different but I think everything from the old should work in the new with some adjustments.

If you have a relatively small cluster snapshot after bringing down your data nodes and preventing incoming document ingestion is the way to go.

Hi @DandyDeveloper , if there is any update on this,please let me know, Thanks for your help