add Raw Type (packed data)

[abossard]

For data length calculation we pack data into a list until we hit a certain threshold. Then we take the packed values and put them into a MessagePack Array.

For this pre-packaging we introduced the raw type. It allows to have Bytes, that already represent MessagePack, and put them into another MessagePack.

It sounds similar to Binary, but it isn't the same: Binary marks the data explicitly as being binary and thus you'll also get a binary array back after unpack.

Raw, in contrast, does not add any marks and thus, if the raw bytes represent e.g. a .string("hello") after unpack, you won't get back the bytes you supplied, but .string("hello").

Thank you for your great work and the time to take this PR into consideration.

[a2]

I'm not sure I understand the use case of the .raw type. Can you provide an example?

[abossard]

Here's how we use it to not exceed a certain message size:

        // in reality our data is far from uniform and each element has a different size and subelements
        let strings = ["s1", "s2", "s3", "s4"]
        let packed = strings.map {string in pack(.string(string))}  // 1

        var size = 0
        let threshold = 5
        var currentMessage: [Data] = []
        packed.forEach {data in
            if size > threshold { // 2
                let message = pack(.array(currentMessage.map {it in .raw(it)})) // 3
                sendMessage(message) // 4
                currentMessage.removeAll()
                size = 0
            }
            currentMessage.append(data) // 2
            size += data.count
        }

1. first we pack data
2. we collect the packed data until we reach a threshold
3. we then pack the collected data into a message
4. we send the message

The alternative for us would have been to either make our own size calculation without actually packing the data or do the packaging twice, as we found no way of using already packed bytes.

I decided to create the .raw type. I understand if it isn't seen as something generally useful.

[a2]

Sorry for the delay. I'm not quite sure how the .raw type is useful beyond this use case, which seems very specific to your project. I understand what it's doing, but not how it would be useful to the larger MessagePack.swift user base.

[abossard]

Thanks for the consideration. I also doubt that it's generally useful and if someone would look for it, he'd stumple upon this PR now. On Sat, 13 May 2017 at 18:57, Alexsander Akers notifications@github.com wrote:

Sorry for the delay. I'm not quite sure how the .raw type is useful beyond this use case, which seems very specific to your project. I understand what it's doing, but not how it would be useful to the larger MessagePack.swift user base.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/a2/MessagePack.swift/pull/47#issuecomment-301260545, or mute the thread https://github.com/notifications/unsubscribe-auth/AAFSUxsnLMuKDMOVv06_uhWkFEzP7A7Bks5r5eD7gaJpZM4NPGTY .

[a2]

Going to close this for now but we can always reevaluate it in the future.