Add the ability to define a packet format using a file as input

[ehsteve]

This may be easier to do if a new object was created that held the list of PacketFields. This object could then read a file. Such a file could be as simple as a csv file.

[ddasilva]

The key issue here is that we need to define our own format, or find out the formats that existing flight software packages might already be writing (eg CFS/CFE). If we define ourselves, each project would still probably get the packet definitions in some other format and it would be on the ccsdspy user to convert to our format. But maybe it’s better to have it in our format so one doesn’t have to do the conversation every time the program is run.

I imagine this would be a class method FixedLength.from_csv() or FixedLength.from_json()

[ehsteve]

Many missions handle their packet definitions in excel or table formats which can easily be exported to a csv file. Defining our own straightforward format seems like a good idea and it could be possible to add support for other formats in the future. This would allow for users to manager their packet definition outside of code which is always a good thing.

The format could be as simple as

SHCOARSE, uint, 32
SHFINE, uint, 20
OPMODE, uint, 3
SPACER, uint, 1
VOLTAGE, int, 8

I suggest we also support the following format

SHCOARSE, uint, 48, 80
SHFINE, uint, 80, 100
OPMODE, uint, 100, 103
SPACER, uint, 103, 104
VOLTAGE, int, 104, 112

[rstrub]

I used this to defs.csv file ingest electronFast for L1A...

name,data_type,bit_offset,bit_length,calibration PREPENDED_APID,uint,5,11, pkt130_ss_count,uint,16,16 pkt130_dlen,uint,32,16, START_CORSTIME,uint,48,32, START_FINETIME,uint,80,20, PKT_CIDP_RSIDE,uint,100,1, PKT_SPACCRFTID,uint,101,1, PKT_SECHDREXID,uint,102,2, MCPATTEN_STATE,uint,104,8, STARTDELPHICNT,uint,112,16, TALLYOVRCNTPIX,uint,128,18, MAGMSGUNITXCMP,uint,144,16, MAGMSGUNITYCMP,int,160,16, MAGMSGUNITZCMP,uint,176,16, MAGMSGFLGNOVLD,uint,192,4, MAGMSGFLGNVSNR,uint,196,4, MAGMSGFLGFGSRC,uint,200,2, MAGMSGFLGFGRNG,uint,202,2, MAGMSGFLGFGNFG,uint,204,2, MAGMSGFLAGNONE,uint,206,2, MAGMSGMAGNITUD,uint,208,16, CMPTRIGGERTERM,uint,224,3600, COMPRESSSTATUS,uint,3824,16 CCSDS_CMPRSIMG,uint,3840,15696

I just needed to make a slight adjustment to decode.py: if meta.nbytes_final > 10 (something that is obviously an array): field_arrays[field._name] = arr continue else: arr.dtype = meta.np_dtype

for these two guys : cmpTriggerTerm: bytarr(3, 150), $ ; LUT-compressed trigger terms. CCSDS_cmprsImg: bytarr(1962) $ ; Maximum size of compressed image.

in: PREPENDED_APID UINT 288 PREPENDED_DLEN ULONG 2436 START_CORSTIME ULONG 1892519740 START_FINETIME ULONG 229818 PKT_CIDP_RSIDE BYTE 0 PKT_SPACCRFTID BYTE 0 PKT_SECHDREXID BYTE 0 MCPATTEN_STATE BYTE 0 STARTDELPHICNT UINT 2057 TALLYOVRCNTPIX UINT 0 MAGMSGUNITXCMP INT -133 MAGMSGUNITYCMP INT -8214 MAGMSGUNITZCMP INT 31728 MAGMSGFLGNOVLD BYTE 0 MAGMSGFLGNVSNR BYTE 0 MAGMSGFLGFGSRC BYTE 0 MAGMSGFLGFGRNG BYTE 0 MAGMSGFLGFGNFG BYTE 0 MAGMSGFLAGNONE BYTE 0 MAGMSGMAGNITUD ULONG 2250 CMPTRIGGERTERM BYTE Array[3, 150] COMPRESSSTATUS UINT 25557 CCSDS_CMPRSIMG BYTE Array[1962]

[ddasilva]

Thanks Richard and Steve.

I suggest we also support the following format

This sounds reasonable. A pull request would be welcome for a FixedLength.from_csv() class method that reads a CSV file and siphons the columns to PacketField() keyword arguments, returning a FixedLength instance at the end. It would be a lot better if we required column names in that CSV. Also we are currently not depending on pandas, so would have to use the builtin csv parser.

[ehsteve]

I'll work on this and I'll add column names. Simple csv reader should be straightforward without pandas. No need to add that dependency!

[ehsteve]

@rstrub what is the meaning of the calibration column?

[ddasilva]

I don't know about @rstrub's specific example, but in other projects I've seen calibration data (to convert from digital to analog units) included in the same file as the packet layout definitions. For example, if a thermal sensor packet field reports in 8-bit digital units, the calibration data might be coefficients of a polynomial to convert from the digital units to Kelvin. They might also be a reference key (eg, ID number) of an equation in another table somewhere of calibration curves.