search

CCSDSPy / ccsdspy

I/O interface and utilities for CCSDS binary spacecraft data in Python. Library used in flight missions at NASA, NOAA, and SWRI
https://ccsdspy.org
BSD 3-Clause "New" or "Revised" License
74 stars 18 forks source link

iter_packet_bytes incrementally read from stream, blocking for wait #92

Open scandey opened 10 months ago

scandey commented 10 months ago

It would be nice to have iter_packet_bytes (or similar util function) that can just sit and listen at a socket for packets by grabbing small chunks of data at a time. It would be a reasonable assumption (in my opinion) to trust that the packets are all well-formed, but that there might be extra bytes at the end of a given chunk of data that belong to the next packet. Originated from https://github.com/CCSDSPy/ccsdspy/issues/83#issuecomment-1663412555

ddasilva commented 10 months ago

I agree, this is a good idea. I will accept pull request from the community if someone wants to implement this before I get around to

ehsteve commented 3 months ago

Having written this kind of functionality myself (without using CCSDSPy), I do worry that it may not be very simple to write something like this that is generic enough to be useful widely useful. The main difference is that the strategy to find packets inside an existing file versus finding packets in real-time as they arrive are very different in my experience.

In real-time data, if you lose where your packets start and end then you end up reading bytes trying to get in sync again. This can be easier or harder based on how many APIDs you have in your stream. I also found that this is highly dependent on your communication protocol. Some read byte by byte (like serial) others send whole packets (like udp) which also has to be baked into the real-time packet finding strategy. Maybe a lot of this can be abstracted if we can generate a byte stream.

Don't get me wrong, I would love a generic solution to this problem and if it can be provided would definitely use it. If there is only one APID in the data stream then that may be a solvable problem.