search

CCSDSPy / ccsdspy

I/O interface and utilities for CCSDS binary spacecraft data in Python. Library used in flight missions at NASA, NOAA, and SWRI
https://ccsdspy.org
BSD 3-Clause "New" or "Revised" License
75 stars 18 forks source link

Implementation of PacketArray #35

Closed ddasilva closed 1 year ago

ddasilva commented 1 year ago

Closes #28

This implements a PacketArray class with keyword arguments array_shape: Tuple[int] or int and array_order: either 'C' (C-order) or 'F' (Fortran-order). It works with the FixedLength class, and tests are included for 1d and multidimensional arrays.

For now, I made PacketArray extend the PacketField class, but I'm not married to this idea. I'm curious what @ehsteve thinks. It might make sense to have them all inherit from PacketEntity, but am cautious about changing the docstrings to "accepts PacketEntity" without clearly explaining to users that's an base class that one doesn't create directly.

ddasilva commented 1 year ago

Also included in this pull request is a couple commits that cleans up the tests for the hs test data. The ground truth data that had been provided to us was wrong so we weren't using it. I removed it from the repository so it doesn't confuse anyone in the future.

ddasilva commented 1 year ago

Hey @rstrub, since you have a vested interest in array fields, do you think you could give this branch a whirl and see if it meets your use case? it would be great if you were our first user of this. I'm available on chat (I think we're still connected on hangouts) if you have any questions

rstrub commented 1 year ago

Yes, I’m figgering it out now… Last time your tests allowed me to figure it out by myself. So this time I’m hoping the same.

The result is clearly 7200 packets of an 8 element array It seems that the normal_result is similar to the old code whereby the ith element in each of the 8 fields contains the the contents of the ith array result… So I can see where last time I could have Instead of:

…

CMPTRIGGERTERM,uint,224,3600. (3x150)

COMPRESSSTATUS,uint,3824,16 CCSDS_CMPRSIMG,uint,3840,15696. (1x1962) Perhaps I could have: … CMPTRIGGERTERM[0],uint,224,1200 CMPTRIGGERTERM[1],uint,1424,1200 CMPTRIGGERTERM[2],uint,2624,1200 CCSDS_CMPRSIMG[0],uint,3840,8 CCSDS_CMPRSIMG[1],uint,3848,8 CCSDS_CMPRSIMG[2],uint,3856,8 … CCSDS_CMPRSIMG[1961],uint,15688,8 And then pulled out the data in a more controlled way rather than just going through 1600(nominally) packets of 1962 bytes = 3139200 bytes and pulling out 1962 at a time though that worked wonderfully. I could have done the above and pulled out the first byte from each of my …whatevers…

But if you could send me a defs.csv that would process my 2 arrays, (CMPTRIGGERTERM and CCSDS_CMPRSIMG) that would be cool! Original again was: …

MAGMSGMAGNITUD,int,208,16, CMPTRIGGERTERM,uint,224,3600, COMPRESSSTATUS,uint,3824,16 CCSDS_CMPRSIMG,uint,3840,15696

From: Daniel da Silva @.> Reply-To: ddasilva/ccsdspy @.> Date: Wednesday, December 21, 2022 at 2:22 PM To: ddasilva/ccsdspy @.> Cc: Richard Strub @.>, Mention @.***> Subject: [EXTERNAL] Re: [ddasilva/ccsdspy] Implementation of PacketArray (PR #35)

Hey @rstrubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Frstrub&data=05%7C01%7Crichard.f.strub%40nasa.gov%7Cf2e147be7ca84eb7064008dae388b91f%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C638072473603462641%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tjRUrUuWKQXlOC8S%2F608lRhJkEAk4DyLw8wqYHWdjUw%3D&reserved=0, since you have a vested interest in array fields, do you think you could give this branch a whirl and see if it meets your use case? it would be great if you were our first user of this. I'm available on chat (I think we're still connected on hangouts) if you have any questions

— Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fddasilva%2Fccsdspy%2Fpull%2F35%23issuecomment-1361983753&data=05%7C01%7Crichard.f.strub%40nasa.gov%7Cf2e147be7ca84eb7064008dae388b91f%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C638072473603462641%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8zt0MpuPsn7IqXSlReziCZ49LYCDurhEe6y4frdeunY%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAD54776RIUIS2IRR2URXF4DWONKHXANCNFSM6AAAAAATC4H43Q&data=05%7C01%7Crichard.f.strub%40nasa.gov%7Cf2e147be7ca84eb7064008dae388b91f%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C638072473603462641%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=TuNLjFUg5XCjN%2FvJdLyAQjZ2M6qw4uX9nyx4KGsyYOw%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

rstrub commented 1 year ago 
            Though now that I think about it maybe what is important is which field I ingest as a PacketField and which as a PacketArray…

PacketArray(name="CCSDS_CMPRSIMG", data_type="uint", bit_length=8, array_shape=1962), PacketArray(name="CMPTRIGGERTERM[0]", data_type="uint", bit_length=8, array_shape=150), PacketArray(name="CMPTRIGGERTERM [1]", data_type="uint", bit_length=8, array_shape=150), PacketArray(name="CMPTRIGGERTERM [2]", data_type="uint", bit_length=8, array_shape=150),

hmmm

From: Daniel da Silva @.> Reply-To: ddasilva/ccsdspy @.> Date: Wednesday, December 21, 2022 at 2:22 PM To: ddasilva/ccsdspy @.> Cc: Richard Strub @.>, Mention @.***> Subject: [EXTERNAL] Re: [ddasilva/ccsdspy] Implementation of PacketArray (PR #35)

Hey @rstrubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Frstrub&data=05%7C01%7Crichard.f.strub%40nasa.gov%7Cf2e147be7ca84eb7064008dae388b91f%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C638072473603462641%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tjRUrUuWKQXlOC8S%2F608lRhJkEAk4DyLw8wqYHWdjUw%3D&reserved=0, since you have a vested interest in array fields, do you think you could give this branch a whirl and see if it meets your use case? it would be great if you were our first user of this. I'm available on chat (I think we're still connected on hangouts) if you have any questions

— Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fddasilva%2Fccsdspy%2Fpull%2F35%23issuecomment-1361983753&data=05%7C01%7Crichard.f.strub%40nasa.gov%7Cf2e147be7ca84eb7064008dae388b91f%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C638072473603462641%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8zt0MpuPsn7IqXSlReziCZ49LYCDurhEe6y4frdeunY%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAD54776RIUIS2IRR2URXF4DWONKHXANCNFSM6AAAAAATC4H43Q&data=05%7C01%7Crichard.f.strub%40nasa.gov%7Cf2e147be7ca84eb7064008dae388b91f%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C638072473603462641%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=TuNLjFUg5XCjN%2FvJdLyAQjZ2M6qw4uX9nyx4KGsyYOw%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

ddasilva commented 1 year ago

Richard, a basic code example would be this. If the field is a single value, use PacketField, if it's repeated, use PacketArray.

A PacketField will always result in an array of shape (npackets,) in the return dictionary, where PacketArray of array_shape=(m, n) will result in an array of shape (npackets, m, n). If the PacketArray is of array_shape=8, then it will result in an array of shape (npackets, 8).


   pkt = ccsdspy.FixedLength([
        PacketField(name='SHCOARSE', data_type='uint', bit_length=32),
        PacketField(name='SHFINE',   data_type='uint', bit_length=20),
        PacketField(name='OPMODE',   data_type='uint', bit_length=3),
        PacketField(name='SPACER',   data_type='fill', bit_length=1),
        PacketField(name='VOLTAGE',  data_type='int',  bit_length=8),
    PacketArray(
            name='SENSOR_GRID',
            data_type='uint',
            bit_length=16,
            array_shape=(32, 32),
            array_order='C'
    ),
   ])```
rstrub commented 1 year ago

I’m running into a problem with if all(field._bit_offset is None for field in fields): assert counter == packet_nbytes 8, "Field definition != packet length".format( n=counter - packet_nbytes 8 ) because counter isn’t incrementing properly because PacketField no longer has bit_offset (so they are always None) What happened to bit_offset?

From: Daniel da Silva @.> Reply-To: ddasilva/ccsdspy @.> Date: Wednesday, December 21, 2022 at 3:47 PM To: ddasilva/ccsdspy @.> Cc: Richard Strub @.>, Mention @.***> Subject: [EXTERNAL] Re: [ddasilva/ccsdspy] Implementation of PacketArray (PR #35)

Richard, a basic code example would be this. If the field is a single value, use PacketField, if it's repeated, use PacketArray.

A PacketField will always result in an array of shape (npackets,) in the return dictionary, where PacketArray of array_shape=(m, n) will result in an array of shape (npackets, m, n). If the PacketArray is of array_shape=8, then it will result in an array of shape (npackets, 8).

pkt = ccsdspy.FixedLength([

    PacketField(name='SHCOARSE', data_type='uint', bit_length=32),

    PacketField(name='SHFINE',   data_type='uint', bit_length=20),

    PacketField(name='OPMODE',   data_type='uint', bit_length=3),

    PacketField(name='SPACER',   data_type='fill', bit_length=1),

    PacketField(name='VOLTAGE',  data_type='int',  bit_length=8),

    PacketArray(

        name='SENSOR_GRID',

        data_type='uint',

        bit_length=16,

        array_shape=(32, 32),

        array_order='C'

    ),

])```

— Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fddasilva%2Fccsdspy%2Fpull%2F35%23issuecomment-1362073578&data=05%7C01%7Crichard.f.strub%40nasa.gov%7C9f65f910e7584603181508dae39485c5%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C638072524287375573%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=U5t4IAgjbIkSUSj9moawn7BSWdZQ8y1TpLWcHq46Cts%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAD54773TO7P5YV7W4TYBOSTWONUEPANCNFSM6AAAAAATC4H43Q&data=05%7C01%7Crichard.f.strub%40nasa.gov%7C9f65f910e7584603181508dae39485c5%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C638072524287375573%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5Cv5gXO8kNBmpLw6gSdg%2BS%2BjEHkWo9lHILcx%2FdU%2BM0Q%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

ddasilva commented 1 year ago

Usually that error happens because the packet you defined is longer or shorter than what the packet actually is. Can you check that?

ddasilva commented 1 year ago

Can you post your code?

rstrub commented 1 year ago

Yes that’s what I’m trying to do is define the packet but my offsets are not getting set. I found them ….I’ll try again. …

valid_byte_orders = ("big", "little") if byte_order not in valid_byte_orders: raise ValueError(f"byte_order must be one of {valid_byte_orders}")

self._field_type = "element" self._name = name self._data_type = data_type self._bit_length = bit_length self._bit_offset = bit_offset self._byte_order = byte_order

From: Daniel da Silva @.> Reply-To: ddasilva/ccsdspy @.> Date: Wednesday, December 21, 2022 at 5:05 PM To: ddasilva/ccsdspy @.> Cc: Richard Strub @.>, Mention @.***> Subject: [EXTERNAL] Re: [ddasilva/ccsdspy] Implementation of PacketArray (PR #35)

Usually that error happens because the packet you defined is longer or shorter than what the packet actually is. Can you check that?

— Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fddasilva%2Fccsdspy%2Fpull%2F35%23issuecomment-1362164524&data=05%7C01%7Crichard.f.strub%40nasa.gov%7C2db34535da5b40310dd608dae39f79a3%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C638072571365424184%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6vR8xQmc4Ukc0rLCA%2FzbsVu7cLZh09Yuw9lHwOJJOkY%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAD54772L6NKTLHBK7SRR6UTWON5KNANCNFSM6AAAAAATC4H43Q&data=05%7C01%7Crichard.f.strub%40nasa.gov%7C2db34535da5b40310dd608dae39f79a3%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C638072571365424184%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jnNrl2UnDK1w%2Bt5FvzDDaN%2BsZGQVSy9DB3Eabfc7BrY%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

ehsteve commented 1 year ago

@ddasilva I tested this code with the following code for a simple one dimensional array and it returned the array as expected

"""Create simple packets with variable lengths"""
import struct

import numpy as np

OUTPUT_FILENAME = 'packet_array.bin'
num_packets = 10
AP_ID = 0x08E2
PACKET_ID = int("0001100000000000", 2) + AP_ID

packet = b""

ARRAY_SHAPE = 6
data = np.arange(ARRAY_SHAPE, dtype='uint16')

for packet_num in range(num_packets):
    packet_length = 2 * ARRAY_SHAPE - 1 + 1 # factor of two because using H 16 bit elements
    this_packet = struct.pack(">HHH", PACKET_ID, packet_num, packet_length)

    # add a filler field
    this_packet += struct.pack(">B", packet_length)

    # add a data field with value equal to packet_num
    for j in range(ARRAY_SHAPE):
        this_packet += struct.pack(">H", data[j] + packet_num)
    packet += this_packet

f = open(OUTPUT_FILENAME, "wb")
f.write(packet)
f.close()

import ccsdspy
from ccsdspy import PacketField, PacketArray

pkt = ccsdspy.FixedLength([
         PacketField(name='DATA', data_type='uint', bit_length=8),
         PacketArray(
             name='ARRAY',
             data_type='uint',
             bit_length=16,
             array_shape=ARRAY_SHAPE,
             array_order='C'),
    ])

result = pkt.load(OUTPUT_FILENAME)

print(result)
ehsteve commented 1 year ago

I also tried the following which tests a multidimensional array and it also worked as expected.

"""Create simple packets with variable lengths"""
import struct

import numpy as np

OUTPUT_FILENAME = 'packet_array.bin'
num_packets = 10
AP_ID = 0x08E2
PACKET_ID = int("0001100000000000", 2) + AP_ID

packet = b""

ARRAY_SHAPE = (6, 18)
data = np.arange(ARRAY_SHAPE[0] * ARRAY_SHAPE[1], dtype='uint8')
data = data.reshape(ARRAY_SHAPE)
print(data)

for packet_num in range(num_packets):
    packet_length = ARRAY_SHAPE[0] * ARRAY_SHAPE[1] - 1 + 1
    this_packet = struct.pack(">HHH", PACKET_ID, packet_num, packet_length)

    # add a filler field
    this_packet += struct.pack(">B", packet_length)

    # add a data field with value equal to packet_num
    for i in range(ARRAY_SHAPE[0]):
        for j in range(ARRAY_SHAPE[1]):
            this_packet += struct.pack(">B", data[i, j] + packet_num)
    packet += this_packet

f = open(OUTPUT_FILENAME, "wb")
f.write(packet)
f.close()

import ccsdspy
from ccsdspy import PacketField, PacketArray

pkt = ccsdspy.FixedLength([
         PacketField(name='DATA', data_type='uint', bit_length=8),
         PacketArray(
             name='ARRAY',
             data_type='uint',
             bit_length=8,
             array_shape=ARRAY_SHAPE,
             array_order='C'),
    ])

result = pkt.load(OUTPUT_FILENAME)

print(result)
ddasilva commented 1 year ago

Thanks @ehsteve! I'm going to merge this pull request. @rstrub: If you need 1-1 support, let me know and I can help you over chat.