Ran into a situation on Rediff VSNL primary where the Master failed to come up
because it hit an exception trying to insert a new RangeServerClient object
into the ServerList container in the context object. Here's what happened
1. ht-vsnl-slave7 was inserted with IP address 10.50.60.125
2. At some point, the IP address for ht-vsnl-slave7 changed to 10.50.60.167,
but the corresponding MML entry was not changed
3. We added two new nodes (ht-vsnl-slave9 and ht-vsnl-slave10) and
ht-vsnl-slave9 coincidentally had the IP address 10.50.60.125. When the
Master tried to add ht-vsnl-slave9, it asserted because it already had an entry
for that IP. See Master log output below.
To fix this, whenever a RangeServer connects, we should check the IP it is
connecting with and if it differs from the RangeServerConnection object that
what was read from the MML, we should update the IP address in the
RangeServerConnection object and write out a new MML entry.
----------
1333054356 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/OperationSystemUpgrade.cc:85)
Entering SystemUpgrade-979552
1333054356 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/OperationSystemUpgrade.cc:99)
Leaving SystemUpgrade-979552
1333054356 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/Context.cc:47) Tried to insert
rs9 host=ht-vsnl-slave9 local=0.0.0.0:0 public=10.50.60.125:38060
1333054356 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/Context.cc:51) Contains rs1
host=ht-vsnl-slave1 local=0.0.0.0:0 public=10.50.30.175:38060
1333054356 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/Context.cc:51) Contains rs2
host=ht-vsnl-slave3 local=0.0.0.0:0 public=10.50.30.173:38060
1333054356 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/Context.cc:51) Contains rs3
host=ht-vsnl-slave5 local=0.0.0.0:0 public=10.50.30.171:38060
1333054356 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/Context.cc:51) Contains rs4
host=ht-vsnl-slave2 local=0.0.0.0:0 public=10.50.30.174:38060
1333054356 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/Context.cc:51) Contains rs5
host=ht-vsnl-slave6 local=0.0.0.0:0 public=10.50.30.243:38060
1333054356 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/Context.cc:51) Contains rs6
host=ht-vsnl-slave7 local=0.0.0.0:0 public=10.50.60.125:38060
1333054356 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/Context.cc:51) Contains rs7
host=ht-vsnl-slave8 local=0.0.0.0:0 public=10.50.90.46:38060
1333054356 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/Context.cc:51) Contains rs8
host=ht-vsnl-slave10 local=0.0.0.0:0 public=10.50.30.170:38060
1333054356 FATAL Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/Context.cc:53) failed
expectation: insert_result.second
Original issue reported on code.google.com by nuggetwh...@gmail.com on 29 Mar 2012 at 10:51
Original issue reported on code.google.com by
nuggetwh...@gmail.com
on 29 Mar 2012 at 10:51