[robohash app] biased coin flip robust reconstruction fails to open "d" & "e" (BeaverMultiply)

[sbellem]

When running the robohash app with the biased coin flip, based on fixed point arithmetic, robust reconstruction fails. The error occurs in mpc.py as shown below:

hbmpc.peer0.io_1        | 2020-08-19 18:23:55,576:[mpc.py:cb:162]:[ERROR]: Robust reconstruction for share {6087826343110230357947059204667799136441592364043611791189230863245351977037} (id: 49) failed with errors: None!

Although the error consistently occurs at the above location, the preceding code, invoking the opening may differ for different runs of the same code.

Root cause (?)

Upon further inspection, the reconstruction error, in mpc.py appears to "consistently" occur after trying to open d and e in BeaverMultiply (line 19):

https://github.com/initc3/HoneyBadgerMPC/blob/f2a9ac58547bba2d4ed7d2f1a99a8aa97ef35b44/honeybadgermpc/progs/mixins/share_arithmetic.py#L9-L21

Reproducing the error

To reproduce the error, first checkout the branch troubleshoot/robohash/robust-reconstruction/biased-coin-flip, from the initc3 remote, e.g.:

# upstream points to git@github.com:initc3/HoneyBadgerMPC.git -- change if needed
git fetch upstream
git checkout troubleshoot/robohash/robust-reconstruction/biased-coin-flip

Go into robohash directory:

cd apps/robohash
make run

After running make run you should eventually see a terminal window with 7 panes, the 4 bottom ones are MPC servers where the logs from the failing robust reconstruction can be seen, as in the following screenshot:

Further troubleshooting

It's not clear (to me :smile:) how to proceed forward with finding the exact root cause of the problem. In case this can help here's a bit of information regarding what has been done to get the current information.

Multiple logging statements were added to try to pinpoint where the problem is occurring. The locations of interests are in:

• apps/robohash/biased_coin.py (from observations, the problem occurs after an instruction call in the function flip_biased_coin line 76, where a lt operation is invoked with a FixedPoint instance)
• honeybadgermpc/fixedpoint.py (many supporting functions are involved (trunc, trunc_pr, div2m, etc) -- from observations, the "root" has been the FixedPoint class methods __mul__() or lt()
• honeybadgermpc/progs/mixins/share_arithmetic.py (in BeaverMultiply, _prog function, opening of d and e)
• honeybadgermpc/reed_solomon.py (_optimistic_update, line 323)
• honeybadgermpc/mpc.py (inner function cb, line 162)

Example of logs for one MPC server (hbmpc.peer3.io):

Tmux tricks

When troubleshooting, it's useful to search the logs. To do so in tmux:

1. go to the pane you wish to search
2. enter in the copy mode with ctrl + b [
3. enter /
4. enter characters to search, e.g. (mpc.py)

For more information about key combinations in tmux see https://tmuxcheatsheet.com/.

[sbellem]

When running the dockerized app for asynchromix (apps/asynchromix2), the following error occasionally occurs:

hbmpc.peer1.io_1        | 2020-08-25 00:43:25,854:[reed_solomon.py:_optimistic_update:323]:[CRITICAL]: Optimistic decoding failed
hbmpc.peer1.io_1        | 2020-08-25 00:43:25,855:[reed_solomon.py:_optimistic_update:323]:[CRITICAL]: Optimistic decoding failed
hbmpc.peer1.io_1        | 2020-08-25 00:43:25,957:[reed_solomon.py:_optimistic_update:323]:[CRITICAL]: Optimistic decoding failed
hbmpc.peer1.io_1        | 2020-08-25 00:43:25,959:[reed_solomon.py:_optimistic_update:323]:[CRITICAL]: Optimistic decoding failed
hbmpc.peer1.io_1        | 2020-08-25 00:43:25,980:[batch_reconstruction.py:batch_reconstruct:184]:[ERROR]: [BatchReconstruct] P1 reconstruction failed!
hbmpc.peer1.io_1        | 2020-08-25 00:43:25,981:[batch_reconstruction.py:batch_reconstruct:184]:[ERROR]: [BatchReconstruct] P1 reconstruction failed!
hbmpc.peer1.io_1        | 2020-08-25 00:43:25,981:[mpc.py:cb:195]:[ERROR]: Batch reconstruction for share_array (id: 10) failed!
hbmpc.peer1.io_1        | 2020-08-25 00:43:25,982:[mpc.py:cb:195]:[ERROR]: Batch reconstruction for share_array (id: 11) failed!
hbmpc.peer1.io_1        | 2020-08-25 00:43:25,982:[misc.py:print_exception_callback:17]:[CRITICAL]:
hbmpc.peer1.io_1        | Exception:
hbmpc.peer1.io_1        | <Task finished coro=<AsyncMixin.__call__() done, defined at /usr/src/HoneyBadgerMPC/honeybadgermpc/progs/mixins/base.py:42> exception=HoneyBadgerMPCError('Batch reconstruction failed!')>
hbmpc.peer1.io_1        | Traceback (most recent call last):
hbmpc.peer1.io_1        |   File "/usr/src/HoneyBadgerMPC/honeybadgermpc/progs/mixins/base.py", line 49, in __call__
hbmpc.peer1.io_1        |     return await cls._prog(context, *args, **kwargs)
hbmpc.peer1.io_1        |   File "/usr/src/HoneyBadgerMPC/honeybadgermpc/progs/mixins/share_arithmetic.py", line 55, in _prog
hbmpc.peer1.io_1        |     f, g = await gather(*[(j - u).open(), (k - v).open()])
hbmpc.peer1.io_1        | honeybadgermpc.exceptions.HoneyBadgerMPCError: Batch reconstruction failed!
hbmpc.peer1.io_1        | )
hbmpc.peer1.io_1        | 2020-08-25 00:43:25,995:[misc.py:print_exception_callback:17]:[CRITICAL]:
hbmpc.peer1.io_1        | Exception:
hbmpc.peer1.io_1        | <Task finished coro=<MPCProgRunner._mpc_loop() done, defined at /usr/src/HoneyBadgerMPC/apps/asynchromix2/mpcprogrunner.py:117> exception=HoneyBadgerMPCError('Batch reconstruction failed!')>
hbmpc.peer1.io_1        | Traceback (most recent call last):
hbmpc.peer1.io_1        |   File "/usr/src/HoneyBadgerMPC/apps/asynchromix2/mpcprogrunner.py", line 198, in _mpc_loop
hbmpc.peer1.io_1        |     result = await ctx._run()
hbmpc.peer1.io_1        |   File "/usr/src/HoneyBadgerMPC/honeybadgermpc/mpc.py", line 254, in _run
hbmpc.peer1.io_1        |     return result.result()
hbmpc.peer1.io_1        |   File "<string>", line 6, in prog
hbmpc.peer1.io_1        |   File "/usr/src/HoneyBadgerMPC/apps/asynchromix/butterfly_network.py", line 51, in iterated_butterfly_network
hbmpc.peer1.io_1        |     result = await batch_switch(ctx, xs_, ys_, k)
hbmpc.peer1.io_1        |   File "/usr/src/HoneyBadgerMPC/apps/asynchromix/butterfly_network.py", line 15, in batch_switch
hbmpc.peer1.io_1        |     ms = (await (sbits * (xs - ys)))._shares
hbmpc.peer1.io_1        |   File "/usr/local/lib/python3.7/asyncio/events.py", line 88, in _run
hbmpc.peer1.io_1        |     self._context.run(self._callback, *self._args)
hbmpc.peer1.io_1        |   File "/usr/src/HoneyBadgerMPC/honeybadgermpc/utils/misc.py", line 18, in print_exception_callback
hbmpc.peer1.io_1        |     raise ex
hbmpc.peer1.io_1        |   File "/usr/src/HoneyBadgerMPC/honeybadgermpc/progs/mixins/base.py", line 49, in __call__
hbmpc.peer1.io_1        |     return await cls._prog(context, *args, **kwargs)
hbmpc.peer1.io_1        |   File "/usr/src/HoneyBadgerMPC/honeybadgermpc/progs/mixins/share_arithmetic.py", line 55, in _prog
hbmpc.peer1.io_1        |     f, g = await gather(*[(j - u).open(), (k - v).open()])
hbmpc.peer1.io_1        | honeybadgermpc.exceptions.HoneyBadgerMPCError: Batch reconstruction failed!
hbmpc.peer1.io_1        | )

Reproducing the error

The error may be reproduced by running make run under apps/asynchromix2.