search

rethinkdb / rethinkdb-python

Python driver for RethinkDB
https://rethinkdb.com/api/python/
Apache License 2.0
64 stars 35 forks source link

Index-rebuild failure after upgrading from 2.0.4 to 2.2.0 #205

Open jilen opened 8 years ago

jilen commented 8 years ago

Every time while I am upgrading the web console always remind to run rethinkdb index-rebuild

But the release note just states that it was not necessary to do this, why ?

Anyway I perform the index rebuild but the python driver complains following

ReQL error during 'rename `aaa.bbb` index `idx_ccc`': The server(s) hosting table `weixin.$reql_temp_index$_idx_ccc` are currently unreachable. The secondary index was not renamed.
mlucy commented 8 years ago

@jilen -- Does the Web UI show any problems with your cluster when you go to it, such as unavailable servers? Or is the cluster otherwise healthy aside from the index rebuild failing?

jilen commented 8 years ago

@mlucy The Web UI shows the table replication is backfilling of one shard of that table. But I never change replication config.

mlucy commented 8 years ago

@jilen -- that's interesting. Does the backfilling look like it's going to complete, or is it stuck in that state?

jilen commented 8 years ago

@mlucy It stuck in that state and seems won't finish. But that table has 1.4M docs, maybe it just need some time.

mlucy commented 8 years ago

@jilen -- I'd also consider checking on the health of the server it's backfilling to, just to make sure it isn't out of RAM or hard drive space or something. (RethinkDB using up all the RAM and getting pushed into swap can sometimes cause very slow backfilling.)

jilen commented 8 years ago

@mlucy I reduce replicas per shard . And the backfill finished soon. But there are indexes like $reql_temp_index$_xxx should I drop the old index, and rename the temp index as the new index ?

mlucy commented 8 years ago

@jilen -- I think that should work (@danielmewes, @Tryneus, am I right?).

I would rename the old index and not drop it until you're sure the new index is working as intended, though.

danielmewes commented 8 years ago

@jilen Yes, renaming the $reql_temp_index$_... indexes to their proper names (without the $reql_temp_index$ prefix) should work and complete the index migration.

jilen commented 8 years ago

@mlucy @danielmewes After the backfilling finish, it could do rename properly

danielmewes commented 8 years ago

It's very likely that the original index rebuild failure was caused by the table still being backfilling.

I think we should put a table.wait() into the index rebuild script. Probably as a separate step with a message "Waiting for tables to become ready" or something, in case the wait takes a while? @Tryneus any thoughts on that?

danielmewes commented 8 years ago

We added the table.wait() to the import/restore scripts, but not to index-rebuild. I think we should do it in index-rebuild as well.