Failed to unpack msgpack data in Windows for large strings

[ywx217]

The foo.gz file is generated by the following python script:

import msgpack
with open('foo.gz', 'rb') as f:
    f.write(msgpack.packb(['a'*i for i in xrange(0, 100000, 20000)]))

When deserializing in windows using the following code, it failed to parse all strings.

    std::ifstream fileStream("foo.gz", std::ios::in | std::ios::binary);
    MsgPack::Deserializer deserializer(fileStream.rdbuf());
    deserializer.deserialize([](std::unique_ptr<MsgPack::Element> parsed) {
        std::cout << *parsed << std::endl;
        exit(0);
        return false;
    }, true);

the output is:

["", "aaaa...", 97, 97, 97]

foo.gz

[Lichtso]

What platform do you use and specifically which CPU architecture? This looks like the endianness was not detected correctly, because on my machine it correctly interprets the string lengths.

Can you check what the following macros give you:

#define STR_VALUE(arg) #arg
#define MACRO_VALUE(name) STR_VALUE(name)

int main(int argc, char** argv) {
    printf("__BYTE_ORDER__ %s\n", MACRO_VALUE(__BYTE_ORDER__));
    printf("BYTE_ORDER %s\n", MACRO_VALUE(BYTE_ORDER));
    printf("LITTLE_ENDIAN %s\n", MACRO_VALUE(LITTLE_ENDIAN));
    printf("BIG_ENDIAN %s\n", MACRO_VALUE(BIG_ENDIAN));
    printf("_BIG_ENDIAN %s\n", MACRO_VALUE(_BIG_ENDIAN));
    printf("__BIG_ENDIAN %s\n", MACRO_VALUE(__BIG_ENDIAN));
    printf("__BIG_ENDIAN__ %s\n", MACRO_VALUE(__BIG_ENDIAN__));
    printf("__ORDER_BIG_ENDIAN__ %s\n", MACRO_VALUE(__ORDER_BIG_ENDIAN__));

[ywx217]

@Lichtso Platform: Windows 10 64bit Intel Core i5-4570 3.20GHz Visual Studio Community 2015

output is:

__BYTE_ORDER__ __BYTE_ORDER__
BYTE_ORDER BYTE_ORDER
LITTLE_ENDIAN LITTLE_ENDIAN
BIG_ENDIAN BIG_ENDIAN
_BIG_ENDIAN _BIG_ENDIAN
__BIG_ENDIAN __BIG_ENDIAN
__BIG_ENDIAN__ __BIG_ENDIAN__
__ORDER_BIG_ENDIAN__ __ORDER_BIG_ENDIAN__

[Lichtso]

Ok, that means none of these macros was defined by the compiler. Still surprising that it seems to be big endian even though it is an intel CPU.

Can you try to find a reliable way to detect the endianness of your machine? https://stackoverflow.com/questions/2100331/c-macro-definition-to-determine-big-endian-or-little-endian-machine#2103095

[ywx217]

It seems it's little endian

    printf("ORDER IS LITTLE ENDIAN: %d\n", O32_HOST_ORDER == O32_LITTLE_ENDIAN);
    printf("ORDER IS BIG ENDIAN: %d\n", O32_HOST_ORDER == O32_BIG_ENDIAN);

it prints:

ORDER IS LITTLE ENDIAN: 1
ORDER IS BIG ENDIAN: 0

Should I include some other header file first?

[Lichtso]

Ok, I think I understand now. Your intel machine is little endian but all endianness macros are undefined. Thus #if BYTE_ORDER == BIG_ENDIAN is true because undefined == undefined.

Try replacing the line 24 in src/MsgPack.cpp with this: #if defined(BYTE_ORDER) && BYTE_ORDER == BIG_ENDIAN effectively adding defined(BYTE_ORDER) &&

[ywx217]

This works! @Lichtso

[Lichtso]

I just committed a fix for this, thank you!