Missing AmbigID's

[denislepage]

There are missing records in the ambig table that are in the runs table:

select distinct motusTagID from runs a where motusTagID < 0 and not exists (select ambigID from tagAmbig b where a.motusTagID = b.ambigID)

[jbrzusto]

specifically:

-58, -59, -63, -64, -67, -69, -70, -71, -72, -74, -76, -77, -78, -79, -80, -81, -82, -83, -86, -87, -93, -94, -95, -106, -221, -222, -223, -224, -225, -226, -227, -229, -230, -231, -232, -233, -234, -235, -236, -237, -238, -239, -240, -241, -242, -243, -244, -247, -249, -251, -252, -253, -254, -255, -256, -257, -258, -259, -260, -261, -262, -264, -265, -266, -267, -268, -269, -270, -271, -272, -273, -274, -275, -276, -277, -280, -281, -282, -283, -285, -286, -288, -289, -290, -291, -292, -293, -294, -295, -296, -297, -298, -299, -300, -301, -302, -304, -305, -306, -307, -309, -313, -314, -315, -321, -323, -324, -325, -327, -328, -329, -331, -339, -341, -343, -347, -355, -356, -357, -376, -383

[jbrzusto]

The atomic index block allocation scheme used to ensure that e.g. runIDs from a single receiver batch are contiguous in the transfer tables even when data from other receivers are being pushed at the same time does not work for the tagAmbig table. There, the goal is not disjoint index blocks, but rather a unique global map. With the current buggy scheme, we'd expect:

• duplicated ambiguity sets with different masterAmbigID when the same tags are listed in a different order within the record fields (e.g. (motusTagID1, motusTagID2) = (12345, 23456) vs (23456, 12345)
  • missing ambiguity sets when parallel runs of receiver data each generated a masterAmbigID for the same set of tags ordered in the same way. The attempt to add the second such record will cause a key violation, so that the corresponding negative motusTagID in the runs table has no referrent.

[jbrzusto]

(ignore accidental close) The initial issue is due to the second effect. And we find the first effect has also occurred:

mysql> select t1.ambigID,t2.ambigID,t1.motusTagID1,t1.motusTagID2  
from tagAmbig as t1 join tagAmbig as t2 
on t1.ambigID < t2.ambigID and t1.motusTagID1=t2.motusTagID2 
and t1.motusTagID2=t2.motusTagID1 
and t1.motusTagID3 is null and t2.motusTagID3 is null;
+---------+---------+-------------+-------------+
| ambigID | ambigID | motusTagID1 | motusTagID2 |
+---------+---------+-------------+-------------+
|    -220 |     -28 |       18468 |       18276 |
|    -178 |     -19 |       21507 |       22798 |
|    -217 |     -22 |       21748 |       22390 |
|    -120 |     -12 |       21755 |       22397 |
|    -139 |     -20 |       22786 |       22911 |
|    -215 |    -181 |       22792 |       21501 |
|    -214 |    -180 |       22795 |       21504 |
|    -212 |    -179 |       22797 |       21506 |
|    -211 |    -136 |       22799 |       21508 |
|    -210 |    -177 |       22800 |       21509 |
|    -209 |    -135 |       22801 |       21510 |
|    -208 |    -134 |       22802 |       21511 |
|    -205 |    -176 |       22807 |       21516 |
|    -198 |    -175 |       22809 |       21518 |
|    -199 |     -36 |       22824 |       22694 |
|    -197 |    -141 |       22899 |       22775 |
|    -196 |    -186 |       22900 |       22776 |
|    -195 |    -185 |       22901 |       22777 |
|    -201 |    -140 |       22902 |       22778 |
|    -193 |     -18 |       22904 |       22780 |
|    -119 |     -35 |       22906 |       22781 |
|    -192 |    -184 |       22907 |       22782 |
|    -189 |    -183 |       22910 |       22785 |
|    -188 |    -138 |       22914 |       22789 |
|    -187 |    -182 |       22915 |       22790 |
+---------+---------+-------------+-------------+