search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
929 stars 24.82k forks source link

upgrading a read-only index causes ClusterBlockException #11286

Closed jeremybmerrill closed 9 years ago

jeremybmerrill commented 9 years ago

I had been running Elasticsearch 1.3.2. A few of my indexes were set to read-only -- that is, blocks.read_only was set to true.

I upgraded to 1.5.2. Various end-points, like /_aliases and /_cat/indices all returned a ClusterBlockException like ClusterBlockException[blocked by: [FORBIDDEN/5/index read-only (api)];].

I was able to fix the problem by unlocking the indices individually. However, there was -- to my admittedly-limited knowledge -- no way to list all my indices. It was only because I kept a separate list of indices that I was able to determine which indices to unlock. Conceivably, a user could get into a problematic state where they can't determine which index to unlock.

My guess is that the upgrade process attempted to upgrade the read-only indexes, which failed because they were set to read-only. Two possible solutions: one is that upgrades disregard the read-only flag; the other is that the _aliases etc. endpoints don't fail when some indexes are upgraded to 1.5.2 and others are still an older version.

I should note that this was a one-box, one-cluster install of Elasticsearch. Both 1.3.2 and 1.5.2 were installed via the .deb. My elasticsearch.yml is quite boring (just a few special HTTP settings).

This was my production box, so I don't have it available for testing anymore. Let me know if you have questions about replicability, etc.

clintongormley commented 9 years ago

@tlrx I think these issues have already been solved by https://github.com/elastic/elasticsearch/pull/9203, no?

tlrx commented 9 years ago

@clintongormley @jeremybmerrill yes, this issue has been solved by #9203.

In 1.4 some cluster blocks were added to various endpoints (see #7740 and #7763) in order to have more control of what operation is allowed or rejected. Theses cluster blocks did not exist in 1.3.2 and that's why you saw a different behavior, the ClusterBlockException, in 1.5.2. Unfortunately some cluster blocks were not fine grain enough and block some endpoints like /_aliases, /_settings... #9203 add more precise cluster blocks level and fixed that.

I'm closing this issue since it will be resolved by the upcoming 1.6 release.