The current appchain-powered PKI dev iteration does not require changes to katzenpost servers or clients, and rather focuses changes on a drop-in authority replacement (one belonging to, and running alongside, each node). Katzenpost servers and clients use MixPublishDeadline and PublishConsensusDeadline defined in katzenpost:authority/voting/server/state.go. So, that aspect of the pki epoch schedule should be preserved.
The current appchain-pki epoch schedule is rudimentary, and while it mostly works there are some observable edge cases where packets are dropped, or PKI document is not available at the expected time; likely from some schedule or pki doc mismatch.
[x] ensure appchain-pki schedule meets timing expectations servers and clients have for the voting-pki
[x] improve pki doc production with more delineated epoch schedule
follow voting-pki deadlines more closely where it makes sense
consider modeling voting-pki's state worker that sleeps then awakes to perform actions at more precise deadlines
[x] ensure all nodes get and use the same pki doc, move towards production of same doc with deterministic topology
logs: PKI error: client2: no PKI document for current epoch
Note: line numbers could be off from using local changes with more debug.
```
INFO walletshield:: Probe packet transmitted/received/loss = 18/18/0.0% | rtt min/avg/max = 1.02/1.91/2.73 s
23:33:59.507 DEBU client2/incomingConn: read length prefix
23:33:59.507 DEBU client2/incomingConn: length prefix is 292
23:33:59.507 DEBU client2/incomingConn: after blob read
23:33:59.507 DEBU client2/incomingConn: before Unmarshal
23:33:59.507 DEBU client2/incomingConn: after Unmarshal
23:33:59.507 DEBU client2/incomingConn: incomingConn received message
23:33:59.507 DEBU client2/incomingConn: recvRequest TCP
23:33:59.507 INFO client2/incomingConn: Received Request from peer application.
23:34:00.440 DEBU client2/pki: getDocument
23:34:00.441 DEBU client2/pki: Fetching PKI doc for epoch 1954067 from Provider.
23:34:00.441 DEBU client2/conn: getConsensus
23:34:00.441 DEBU client2/conn: Enqueued GetConsensus command for send.
23:34:00.442 DEBU client2/conn: Received Consensus: ErrorCode: 1, Payload 0 bytes
23:34:00.442 WARN client2/pki: Failed to fetch PKI for epoch 1954067: client/pki: consensus not ready yet
23:34:00.442 ERRO katzenpost/client2: WaitForCurrentDocument failed on updateDocument with err: client/pki: consensus not ready yet
panic: COMPOSE SPHINX PACKET FAIL client/conn: PKI error: client2: no PKI document for current epoch
goroutine 19 [running]:
github.com/katzenpost/katzenpost/client2.(*Client).SendCiphertext(0xc0000e6370, 0x44df30?)
github.com/katzenpost/katzenpost@v0.0.43/client2/send.go:119 +0x108
github.com/katzenpost/katzenpost/client2.(*Daemon).send(0xc0001ceea0, 0xc00010e7e0)
github.com/katzenpost/katzenpost@v0.0.43/client2/daemon.go:353 +0xb7
github.com/katzenpost/katzenpost/client2.(*Daemon).egressWorker(0xc0001ceea0)
github.com/katzenpost/katzenpost@v0.0.43/client2/daemon.go:234 +0x137
github.com/katzenpost/katzenpost/core/worker.(*Worker).Go.func1()
github.com/katzenpost/katzenpost@v0.0.43/core/worker/worker.go:39 +0x52
created by github.com/katzenpost/katzenpost/core/worker.(*Worker).Go in goroutine 1
github.com/katzenpost/katzenpost@v0.0.43/core/worker/worker.go:37 +0xaa
make: *** [Makefile.appchain:48: probe] Error 2
```
logs: dropped outgoing packet from node listed in future epoch
From katzenpost:server/internal/pki/pki.go:AuthenticateConnection, the node is listed in the pki document for the next epoch (now+1), and as an outgoing connection, an early connection is not allowed and it fails.
```
appchain-mixnet/mix1/katzenpost.log:601:05:14:22.193 DEBU outgoing:1: failed to authenticate connect via latest PKI doc
appchain-mixnet/mix1/katzenpost.log:1150:05:34:21.719 DEBU outgoing:5: failed to authenticate connect via latest PKI doc
```
The current appchain-powered PKI dev iteration does not require changes to katzenpost servers or clients, and rather focuses changes on a drop-in authority replacement (one belonging to, and running alongside, each node). Katzenpost servers and clients use
MixPublishDeadlineandPublishConsensusDeadlinedefined inkatzenpost:authority/voting/server/state.go. So, that aspect of the pki epoch schedule should be preserved.The current appchain-pki epoch schedule is rudimentary, and while it mostly works there are some observable edge cases where packets are dropped, or PKI document is not available at the expected time; likely from some schedule or pki doc mismatch.
logs: PKI error: client2: no PKI document for current epoch
Note: line numbers could be off from using local changes with more debug. ``` INFO walletshield:: Probe packet transmitted/received/loss = 18/18/0.0% | rtt min/avg/max = 1.02/1.91/2.73 s 23:33:59.507 DEBU client2/incomingConn: read length prefix 23:33:59.507 DEBU client2/incomingConn: length prefix is 292 23:33:59.507 DEBU client2/incomingConn: after blob read 23:33:59.507 DEBU client2/incomingConn: before Unmarshal 23:33:59.507 DEBU client2/incomingConn: after Unmarshal 23:33:59.507 DEBU client2/incomingConn: incomingConn received message 23:33:59.507 DEBU client2/incomingConn: recvRequest TCP 23:33:59.507 INFO client2/incomingConn: Received Request from peer application. 23:34:00.440 DEBU client2/pki: getDocument 23:34:00.441 DEBU client2/pki: Fetching PKI doc for epoch 1954067 from Provider. 23:34:00.441 DEBU client2/conn: getConsensus 23:34:00.441 DEBU client2/conn: Enqueued GetConsensus command for send. 23:34:00.442 DEBU client2/conn: Received Consensus: ErrorCode: 1, Payload 0 bytes 23:34:00.442 WARN client2/pki: Failed to fetch PKI for epoch 1954067: client/pki: consensus not ready yet 23:34:00.442 ERRO katzenpost/client2: WaitForCurrentDocument failed on updateDocument with err: client/pki: consensus not ready yet panic: COMPOSE SPHINX PACKET FAIL client/conn: PKI error: client2: no PKI document for current epoch goroutine 19 [running]: github.com/katzenpost/katzenpost/client2.(*Client).SendCiphertext(0xc0000e6370, 0x44df30?) github.com/katzenpost/katzenpost@v0.0.43/client2/send.go:119 +0x108 github.com/katzenpost/katzenpost/client2.(*Daemon).send(0xc0001ceea0, 0xc00010e7e0) github.com/katzenpost/katzenpost@v0.0.43/client2/daemon.go:353 +0xb7 github.com/katzenpost/katzenpost/client2.(*Daemon).egressWorker(0xc0001ceea0) github.com/katzenpost/katzenpost@v0.0.43/client2/daemon.go:234 +0x137 github.com/katzenpost/katzenpost/core/worker.(*Worker).Go.func1() github.com/katzenpost/katzenpost@v0.0.43/core/worker/worker.go:39 +0x52 created by github.com/katzenpost/katzenpost/core/worker.(*Worker).Go in goroutine 1 github.com/katzenpost/katzenpost@v0.0.43/core/worker/worker.go:37 +0xaa make: *** [Makefile.appchain:48: probe] Error 2 ```logs: dropped outgoing packet from node listed in future epoch
From katzenpost:server/internal/pki/pki.go:AuthenticateConnection, the node is listed in the pki document for the next epoch (now+1), and as an outgoing connection, an early connection is not allowed and it fails. ``` appchain-mixnet/mix1/katzenpost.log:601:05:14:22.193 DEBU outgoing:1: failed to authenticate connect via latest PKI doc appchain-mixnet/mix1/katzenpost.log:1150:05:34:21.719 DEBU outgoing:5: failed to authenticate connect via latest PKI doc ```