32 bit Linux Serialization Errors

[bytemaster]

https://bitsharestalk.org/index.php?topic=5523.msg75207#msg75207

./bitshares_client 
Loading blockchain from "/home/myhometalk/.BitSharesXTS/chain"
Loading config "/home/myhometalk/.BitSharesXTS/config.json"
Initializing genesis state from built-in genesis file
Please be patient, this could take a few minutes.
Successfully re-indexed 0 blocks in 0 seconds.
------------ error --------------
8 out_of_range_exception: Out of Range
read datastream of length 23 over by 1
    {"method":"read","len":23,"over":1}
    th_a  datastream.cpp:6 throw_datastream_range_error
error unpacking uint8_t
    {"type":"uint8_t"}
    th_a  raw.hpp:455 unpack
optional<fc::exception>
    {"type":"fc::exception"}
    th_a  raw.hpp:179 unpack
Error unpacking field last_error
    {"field":"last_error"}
    th_a  raw.hpp:237 operator()
error unpacking bts::net::potential_peer_record
    {"type":"bts::net::potential_peer_record"}
    th_a  raw.hpp:455 unpack

[drltc]

We need a strategy for dealing with 32-bit / 64-bit targets. Here are the options:

• Option 1. Declare 32-bit targets unsupported. Least work for us, but very "unprofessional" -- we probably want to support 32-bit targets for 1.0 release.
• Option 2. Running some 32-bit nodes. We should at least run a few testnet and mainnet nodes on a 32-bit OS.
• Option 3. Static analysis. Basically have a script that automatically goes through all potentially serialized fields and outputs sizeof. When the output of the script differs on different platforms, we have a problem.
• Option 3a. Implement Option 3 with FC_REFLECT. Those macros are too hairy and poorly documented for me to really tell if FC_REFLECT is the right tool for this.
• Option 3b. Implement Option 3 with gccxml. Not viable, support for C++11 appears to be poor.
• Option 3c. Implement Option 3 with clang. We might want to check out https://github.com/sk-havok/clang-extract, normally I'd recommend gccxml but it doesn't support C++11.
• Option 4. Comprehensive unit testing of all serialization logic. Basically serialize every type, and check it against the known serialization.

[nathanielhourt]

I will also propose Option 5, which is to use a well-supported serialization library such as Capn Proto. This has multiple advantages, such as bindings for other languages, backwards compatibility baked in, better documentation, etc.

The downside is it will require slightly more work to make our types serializable than just an FC_REFLECT macro, but I personally feel that is a small price to pay.

[emfrias]

For what it's worth, this issue has nothing to do with compatibility between 32-bit and 64-bit serialization. In commit 4dd9c7f8fe64a9d57278ef42194a440248505645, a new field was added to a field in the database but there was no upgrade mechanism for this particular database. When a user started the new version of their client for the first time, it tried to read the old database and couldn't deserialize the newly-added field. This was "fixed" by telling the user to delete their ~/.BitShares directory whenever they upgraded, which used to be common practice.

If it happens in the future when the "delete everything" approach isn't acceptable, fc reflect has a versioning mechanism used in Keyhotee (I'm not sure if it's used in bitshares_toolkit yet) that works well.

[vikramrajkumar]

@emfrias Can you provide a link to an example of the versioning mechanism being used? A quick search didn't lead me to anything.

[dnotestein]

Here's part of it here (contains docs for usage in comments): https://github.com/BitShares/bitshares_toolkit/blob/develop/libraries/db/include/bts/db/upgrade_leveldb.hpp

[drltc]

Based on @emfrias comment, I think the OP issue is resolved. +5% for @nathanhourt's Option 5, filing a new ticket.

[drltc]

Cap'n Proto is still in beta and has not been security audited. Maybe one of us could audit it. But I'm thinking we have plenty of more urgent tasks, so I'm thinking Option 4.

[vikramrajkumar]

For the record, fc is still in alpha and has not been audited either.