Dynamic Arrays

[ArkadyKoretsky]

@MrHIDEn Hi, I'm using your awesome package for parsing and building buffers from objects. Unfortunately the other side that communicating with me via UDP, use different package to build/parse it's own buffers, which kind of creating a small issue when we working with dynamic arrays.

The API we using includes the length of the array in each message but as a separate property. For example something like that:

{ ab: "Ab[i16]", numberOfAbs: "i8" }

I noticed that when you building the array you adding to the buffer it's length right before the beginning of the array, which is also might be a little bit problematic from others side's parsing.

Can you please advise how to deal with such case using your package? Both as a sender and as a receiver.

Thanks in Advance, Arkady

[MrHIDEn]

Hi, I am sure we can write proper frame for that. Can you describe other side frame?

Yes, when you use dynamic length array such "u16[i16]" the length will be transmitted first. In this case length is i16 type and data is u16, but data can be any kind.

[MrHIDEn]

Hi, Arkady. Size of array is added for the reason. example, let say you transmit dynamic {a:'u8',b:'u8} and you do not know the length, do you? So, types:{XY: {a:'u8',b:'u8'}} model {dynamicLengthArray: XY[u8]} -> "0201020304" -> {dynamicLengthArray: [{a:1,b:2},{a:3,b4}]. Because it is dynamic array you cannot place size of it on the end. How would you guess where the array ends? You must place size on the beginig. Right? Is it possible to parse data with size on the end?

Regards Marek

[MrHIDEn]

{ab: "Ab[i16]", numberOfAbs: "i8" } wouldn't be just {ab: "Ab[i8]"} this will be as "(i8)(Ab repeated i8 times)" Can you describe "{ab: "Ab[i16]", numberOfAbs: "i8" }" as a example frame? kind a "(size i8)(Ab blocks size times)".

Regards Marek

[MrHIDEn]

mrhiden@outlook.com

[ArkadyKoretsky]

@MrHIDEn Thanks for the comments! I'm not saying you're wrong with what you're doing. Personally I prefer your way either (adding the length at the beginning of the array), but unfortunately we forced to use such kind of API.

I'm not sure I fully understand the term frame but I took the simplest example to demonstrate the issue we struggling with. For example Ab class defined like this:

{ a: 'u8', b: 'u8' }

And used in more bigger class as described in my example above:

{ ab: 'Ab[i16]', numberOfAbs: 'i8' }

It mostly used in more complex classes like this:

{ propX: 'u8', propY: 'i32', numberOfAbs: 'i8', propZ: 'u16', ab: 'Ab[i16]' }

We are expected to parse it some thing like that: (u8 - propX)(i32 - propY)(i8 - numberOfAbs)(u16 - propZ)(Ab repeated numberOfAbs times). Which means we should parse Ab array after we already parsed the property numberOfAbs.

Again, personally I prefer your way and not working with those extra properties that might be located in random places at the class. But unfortunately, the other side already implemented that messed up logic and it's nearly impossible to convince them to rebuilt their logic and get rid of this unnecessary property and just as convention add the length right where array offset starts.

[MrHIDEn]

Hi @ArkadyKoretsky , hmm...

So frame is (u8 - propX)(i32 - propY)(i8 - numberOfAbs)(u16 - propZ)(Ab repeated numberOfAbs times). = = { propX: 'u8', propY: 'i32', numberOfAbs: 'i8', propZ: 'u16', ab: 'Ab[${numberOfAbs}]' }

Array length must be first in any kind of frame because we cannot guess when to stop parsing array and start parse other things. Your frame folows that rule but numberOfAbs is placed too early before propZ instead ab.

{ ab: 'Ab[i16]', numberOfAbs: 'i8' }

This is confusing. Is type of Ab size 'i8' or 'i16'? I assumed that 'i8', numberOfAbs: 'i8'.

I would suggest to brake this frame into two frames. This is why I added offset when we read/make/write buffers. (u8 - propX)(i32 - propY)(i8 - numberOfAbs)(u16 - propZ)(Ab repeated numberOfAbs times). So, 1st frame: model1: { propX: 'u8', propY: 'i32', numberOfAbs: 'i8', propZ: 'u16'} Here you will read propX, propY, numberOfAbs and propZ. This way you will read length of Ab first. Pick up offset here, "offset1". You will need it in the second part.

2nd frame: types: { Ab: {a: 'u8', b: 'u8' } } model2: { ab: Ab[${numberOfAbs}] } <- {ab: Ab[3]} - this is static array of Ab with length 3, numberOfAbs: 3 in the example.

Length of dynamic array must be right before that array. In your frame the length is unfortunately before propZ

Final example, without using classes. I can convert this to allow return classes.

Test it, please. If I understood your case well, below code works for you. It is bad that propZ is after the numberOfAbs. If propZ was before numberOfAbs, everything would be much simpler.

// split-example.ts
import { CStructBE, printBuffer } from '@mrhiden/cstruct';

{
    // NOTE
    // (u8 - propX)(i32 - propY)(i8 - numberOfAbs)(u16 - propZ)(Ab repeated numberOfAbs times).
    // Ab:{ a: 'u8', b: 'u8' }
    // { propX: 'u8', propY: 'i32', numberOfAbs: 'i8', propZ: 'u16', ab: 'Ab[i16]' }
    // Part1 { propX: 'u8', propY: 'i32', numberOfAbs: 'i8', propZ: 'u16'}
    // Part2 { ab: `Ab[${numberOfAbs}]` }

    // Frame part1 --------------------------------------
    const modelPart1 = {
        propX: 'u8',
        propY: 'i32',
        numberOfAbs: 'i8',
        propZ: 'u16'
    };
    const cStructPart1 = CStructBE.fromModelTypes(modelPart1);

    // Frame part2 --------------------------------------
    type Ab = {a: number, b: number};
    const types = {
        Ab: {a: 'u8', b: 'u8'},
    };
    const getAbArrayStructPart2 = (numberOfAbs: number) => CStructBE.fromModelTypes({ab: `Ab[${numberOfAbs}]`}, types);

    // Sender --------------------------------------
    const sendData = {
        propX: 0x01,
        propY: 0x00000002,
        // numberOfAbs: 0x03,
        propZ: 0x0004,
        ab: [ // numberOfAbs = 3
            {a: 0x05, b: 0x06},
            {a: 0x07, b: 0x08},
            {a: 0x09, b: 0x0A},
        ]
    };
    const makeSenderData = (propX: number, propY: number, propZ: number, ab: Ab[]) => {
        const numberOfAbs = ab.length;
        const {buffer: bufferPart1} = cStructPart1.make({propX, propY, numberOfAbs, propZ});
        const {buffer: bufferPart2} = getAbArrayStructPart2(numberOfAbs).make({ab});
        return Buffer.concat([bufferPart1, bufferPart2]);
    }
    const exchangeDataBuffer = makeSenderData(sendData.propX, sendData.propY, sendData.propZ, sendData.ab);
    printBuffer(exchangeDataBuffer);
    // 010000000203000405060708090a
    // 01 00000002 03 0004 05060708090a
    // 01 00000002 03 0004 [0506 0708 090a]
    // propX:01 propY:00000002 numberOfAbs:03 propZ:0004 ab:[{a:05 b:06}, {a:07 b:08}, {a:09 b:0a}]

    // Receiver --------------------------------------
    const readSenderData = (buffer: Buffer) => {
        const {struct: part1, offset: offset1} = cStructPart1.read(buffer);
        const {struct: part2} = getAbArrayStructPart2(part1.numberOfAbs).read(buffer, offset1);
        return {...part1, ...part2};
    }
    const receivedData = readSenderData(exchangeDataBuffer);
    console.log(receivedData);
    // { propX: 1, propY: 2, numberOfAbs: 3, propZ: 4, ab: [ { a: 5, b: 6 }, { a: 7, b: 8 }, { a: 9, b: 10 } ] }
    // So, the receivedData is the same as sendData
    /* {
            propX: 0x01,
            propY: 0x00000002,
            numberOfAbs: 0x03,
            propZ: 0x0004,
            ab: [
                {a: 0x05, b: 0x06},
                {a: 0x07, b: 0x08},
                {a: 0x09, b: 0x0A},
            ]
        }*/
}

[MrHIDEn]

Because in part2 we use static array we have to "compile" part2 each time. This makes it less effective, but it works.

[ArkadyKoretsky]

@MrHIDEn Sorry, my bad - in my example I meant:

{ ab: 'Ab[i16]', numberOfAbs: 'i16' }

Thanks for the advise and solution, I'll try this out.

[MrHIDEn]

Hi @ArkadyKoretsky

Well...

import { CStructBE, hexToBuffer, printBuffer } from '@mrhiden/cstruct';

BAD This is just bad architecture. Don't do this.

const badFrame = {
    ab: 'Ab[i16]',
    numberOfAbs: 'i16'
};

Because this way you do not know when you are done reading array of Abs. Where is the end of the array? You have to know the size of the array before you start reading it. This is not good. Example buffer:

const receivedBuffer = hexToBuffer(`
    0102 0304 0506 0003   0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
`);
// [{a: 01, b: 02}, {a: 03, b: 04}, {a: 05, b: 06}, <PROBLEM--> {a: 00, b: 03} ... {} {} {} ... <--WHERE IS LENGTH OF THE ARRAY?>]

This is trivial example, and you can spot the length of the array by looking at the buffer. But in real world you will not be able to do that. You will have to know the length of the array before you start reading it.

Buffer transmission doesn't have internal structure. It is just a bunch of bytes. It isn't like JSON or XML where you can see the structure of the data.

receivedBuffer - How can you know when you are done reading array of Abs? It is impossible to know that.

GOOD

const goodFrame = {
    ab: 'Ab[i16]' 
    // i16 is 'numberOfAbs', the length of the array and will be transmited before the array.
    // [ (i16)numberOfAbs, (Ab)ab[numberOfAbs] ] <-- this is how it will be transmited
};

This way you know how many Abs you will receive. You will know when you are done reading array of Abs.

// {ab: 'Ab[i16]'}
const goodFrame = hexToBuffer(`
    0003 0102 0304 0506   0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
`);
// [(i16)0003, (Ab){a: 01, b: 02}, (Ab){a: 03, b: 04}, (Ab){a: 05, b: 06} ... (other data)]

When you don't have transmitted structure of data you MUST know size of the array before you start reading it. There is no other way.

If you are sure that that buffer is correct and your frame ends with { ab: 'Ab[i16]', numberOfAbs: 'i16' } you can just remove last 2 bytes from the buffer and try to read it in the loop till the end each time reading Ab and push it to the array. As you noticed the length of the array is not important this way, you can remove it from the frame.

[MrHIDEn]

Transmitting length of dynamic (unknown length) array after that array is pointless.