search

foxsi / foxsi-4matter

Code for FOXSI-4 formatter.
https://foxsi.umn.edu/
1 stars 3 forks source link

Implement downlink frame segmentation #23

Closed thanasipantazides closed 11 months ago

thanasipantazides commented 1 year ago

The trouble

Originally, this design presumed one parseable frame was less than an Ethernet packet (1.5 kB). But we know now that some frames are ~32 kB, and must be parsed on the ground as a single block.

Each such frame will be read in one block out of the supplying detector system. But, before downlinking, we must add a layer to segment the block. Segmentation implies:

  1. Split the large block into <1.5 kB pieces.
  2. To each piece, prepend a header <source system><subframe counter>. This way, regardless of the arrival order of packets, we can sort by source system hex code, then subframe counter, and reassemble complete frames to parse.
  3. To each piece, either append a unique (i.e. not appearing in the rawdata stream) delimiter, or pad the piece to a constant, fixed packet length. This way, we discriminate subframes across source systems.

Leaning towards using fixed-length packets to send subframes. This incurs low overhead and is more robust than trying to find a unique delimiter that will not appear in any detector's data stream. Only the final subframe Ethernet packet for a given frame will waste downlink capacity.

Note that on the ground, packets should arrive in one piece (not TCP). So the packet-to-packet boundaries should be clear when reading from the ground socket.

thanasipantazides commented 1 year ago

Maybe the header should be <source system><data size><subframe counter>, with <data size> taking the max possible value (1024 or something) for most packets, but can determine the last subframe's content size from this header value (don't need to guarantee that the pad character at end of unfull subframe is different from last character(s) of frame).

The calling context (i.e. not Fragmenter) should prepend the <source system> field.

thanasipantazides commented 1 year ago

More info: TCP and UDP can fragment and reform packets smaller than some MTU size. MTU size over Ethernet is required to be at least 576 B by IPv4 standard. It could be larger (up to 65535 B minus header size) if the underlying hardware/socket interfaces support it, but that is not a given. So should check with TM.

If all onboard systems produce parsable frames smaller than MTU size for the flight system, can let the underlying network stack implement frame fragmentation. Otherwise we need to write something.