`-n` flag is not respected when running entry and build node in a combined process

Commit: cdc7893481aa10d51070d965ae9fc6eebfe91ac2

Problem Description

I am using the mstool to put a given microslice archive into a specified shared memory region to feed its content to a Flesnet entry node. (see Steps to reproduce) When using the -n flag, Flesnet does not produce the specified amount of timeslices and exits early when running entry and build node in a combined process. When running build and entry node in dedicated processes, Flesnet behaves as expected and exits after the configured amount of timeslices.

Steps to Reproduce

Producing a microslice archive file and providing it via shared memory using the mstool

./mstool -n 1000000 -p 0 -o ms_archive.msa
./mstool --input-archive ./ms_archive.msa --output-shm fles_in_shared_memory

When running build and entry node in a combined process:

./flesnet -n 15 -t zeromq -i 0 -I shm:/fles_in_shared_memory/0 -o 0 -O shm:/fles_out_shared_memory/0 --processor-instances 1 -e "./tsclient -i shm:%s -o file:timeslice_archive.tsa"
[09:51:36] INFO: this is input 0 (of 1)
[09:51:36] INFO: this is output 0 (of 1)
[09:51:36] INFO: timeslice size: 100 microslices
[09:51:36] INFO: number of timeslices: 15
[09:51:36] INFO: using shared memory (fles_in_shared_memory)
[09:51:36] INFO: timeslice buffer 0: fles_out_shared_memory {a8cc71f9-6a6d-4702-abbd-5cb1dc03ea21}, size: 1 * (128 MiB + 16 MiB) = 144 MiB
[09:51:36] INFO: [i0] |####################|__________| 0 B/s (0 Hz)
[09:51:36] INFO: [c0] |____________________|__________| 
[09:51:36] INFO: worker connected: TimesliceAutoSource at PID 19318 (s1/o0/p0/g0)
TimesliceReceiver: opened shared memory fles_out_shared_memory {a8cc71f9-6a6d-4702-abbd-5cb1dc03ea21}
[09:51:37] INFO: [i0] |##################..|__________| 14.9 MB/s (1.49 kHz)
[09:51:37] INFO: exiting
[09:51:37] ERROR: ZMQ: Interrupted system call
[09:51:37] INFO: total timeslices processed: 11

Notice the line [09:51:37] INFO: total timeslices processed: 11 in contrast to the configured -n 15 in the Flesnet command. The amount of processed timeslices can vary across multiple runs.

On the other hand, when running entry and build node in dedicated processes (order matters):

Firstly start the build node:

./flesnet -n 15 -t zeromq -I shm://127.0.0.1/0 -o 0 -O shm:/fles_out_shared_memory/0 --processor-instances 1 -e "./tsclient -i shm:%s -o file:timeslice_archive.tsa"

Secondly start the entry node:

./flesnet -n 15 -t zeromq -i 0 -I shm:/fles_in_shared_memory/0 -O shm:/fles_out_shared_memory/0 --processor-instances 1 -e "./tsclient -i shm:%s -o file:timeslice_archive.tsa"

Output of the build node:

[09:58:28] INFO: this is output 0 (of 1)
[09:58:28] INFO: timeslice buffer 0: fles_out_shared_memory {dee21f63-e3fa-457c-b6ea-9fefb0802ff4}, size: 1 * (128 MiB + 16 MiB) = 144 MiB
[09:58:28] INFO: [c0] |____________________|__________| 
[09:58:28] INFO: worker connected: TimesliceAutoSource at PID 19769 (s1/o0/p0/g0)
[09:58:30] INFO: [c0] |____________________|__________| 
TimesliceReceiver: opened shared memory fles_out_shared_memory {dee21f63-e3fa-457c-b6ea-9fefb0802ff4}
[09:58:30] INFO: exiting
[09:58:30] ERROR: ZMQ: Interrupted system call
[09:58:30] INFO: total timeslices processed: 15

Notice the line [09:58:30] INFO: total timeslices processed: 15 - the expected 15 timeslices were built.

If you switch the start order and first start the entry node and then the build node, then something similiar will happen to when you run both in a combined process - less timeslices getting built than configured.

Expected Behavior

In either case, dedicated processes or not, Flesnet should build the configured amount of timeslices. Furthermore I would expect that the start order of entry and build would not matter.

Additional Information

I am currently extending the mstool to verify produced timeslice archives using the input microslice archives (see draft PR ). Using my first prototypical implementations I've analyzed the contents of the produced timeslice archives. My findings explained using an example: Let's say you run Flesnet with a timeslice size of 100, and you want to create 15 timeslices but flesnet will stop after 11 as shown in this bug report. What you will find is, that the first microslice in your output timeslice archive will have the index 400. Which means that, for some reason 400 microslices, which are the 4 missing timeslices, are skipped but counted as produced timeslices anyway.

cbm-fles / flesnet