MUPIP JOURNAL ROLLBACK or RECOVER can hang if processes are killed abnormally (kill -9) and database shared memory is not removed

[nars1]

Final Release Note

Description

A test hang in regular in-house testing exposed an issue.

A process in the test that was holding the database critical section and was in the midst of committing a transaction was killed in the following timing window. Inside multiple levels of macros.

t_end.c
   1430                 UPDATE_JRS_RSRV_FREEADDR(csa, jpc, jbp, jrs, jpl, jnl_fence_ctl, replication);  /* updates jbp->rsrv_freeaddr.
   1431                                                                                                  * Step CMT06
   1432                                                                                                  */

jnl.h
   1437 #define UPDATE_JRS_RSRV_FREEADDR(CSA, JPC, JBP, JRS, JPL, JNL_FENCE_CTL, REPLICATION)                   \
   1438 MBSTART {                                                                                               \
   1439         SET_JNLBUFF_PREV_JREC_TIME(JBP, jgbl.gbl_jrec_time, DO_GBL_JREC_TIME_CHECK_TRUE);               \
   1440                 /* Keep jb->prev_jrec_time up to date */                                                \
   1441         assert(JNL_ENABLED(CSA));                                                                       \
   1442         assert(JBP->rsrv_freeaddr >= JBP->freeaddr);                                                    \
   1443         assert(0 < JRS->tot_jrec_len);                                                                  \
   1444         UPDATE_JBP_RSRV_FREEADDR(CSA, JPC, JBP, JPL, JRS->tot_jrec_len, JRS->phase2_commit_index, TRUE, \
   1445                                         JNL_FENCE_CTL.token, JNL_FENCE_CTL.strm_seqno, REPLICATION);    \
   1446 } MBEND

jnl.h
   1268 #define UPDATE_JBP_RSRV_FREEADDR(CSA, JPC, JBP, JPL, RLEN, INDEX, IN_PHASE2, JNL_SEQNO, STRM_SEQNO, REPLICATION)        \
   1269 MBSTART {                                                                                                               \
   1270         uint4                   rsrv_freeaddr;                                                                          \
      .
      .
   1325         INDEX = endIndex;                                                                                               \
   1326         SET_JBP_RSRV_FREEADDR(JBP, rsrv_freeaddr);                                                                      \
   1327         SHM_WRITE_MEMORY_BARRIER; /* see corresponding SHM_READ_MEMORY_BARRIER in "jnl_phase2_cleanup" */               \
   1328         JBP->phase2_commit_index2 = nextIndex;                                                                          \
   1329 } MBEND

jnl.h
   1254 /* Sets jbp->rsrv_freeaddr & jbp->rsrv_free. They need to be kept in sync at all times */
   1255 #define SET_JBP_RSRV_FREEADDR(JBP, RSRV_FREEADDR)       \
   1256 {                                                       \
   1257         assert(JBP->freeaddr <= RSRV_FREEADDR);         \
   1258         JBP->rsrv_freeaddr = RSRV_FREEADDR;             \
   1259         JBP->rsrv_free = (RSRV_FREEADDR) % JBP->size;   \
   1260 }

In the failure, Line 1258 finished but line 1259 did not start. The process was issued a kill -9 before then. And the test did not remove database shared memory but instead did a MUPIP RUNDOWN and that hung indefinitely. If line 1328 (set of JBP->phase2_commit_index2) happened when the kill -9 happens, then no hang is seen.

MUPIP JOURNAL ROLLBACK or MUPIP JOURNAL RECOVER suffer from the same hang issue. This needs to be fixed so they do not hang irrespective of wherever the kill -9 occurs.

Draft Release Note

MUPIP JOURNAL ROLLBACK and MUPIP JOURNAL RECOVER BACKWARD correctly rundown database files after abnormal process terminations (e.g., resulting from a kill -9, or an OS out-of-memory process termination) that leave shared memory segments in place. Previously, there was a small window of a few instructions in the terminated process that could cause the subsequent MUPIP JOURNAL process to hang indefinitely. Note that YottaDB strongly recommends against using kill -9 to terminate processes.