Implementing custom type serializing

[Relrin]

I'm trying to implement custom type serializer for the BigInt type. So, the first thing that I've made is a look into the serde docs. But this case a little bit complex, that described inside the doc.

That's a code, which I'm using for a serializing:

use num::bigint::{BigInt, Sign};  // other crate
use serde::ser;

pub struct Serializer<W>{
    writer: W
}

impl<W> ser::Serializer for Serializer<W> where W: io::Write {
  // some code there...
}

impl ser::Serialize for BigInt {
    fn serialize<S>(&self, serializer: &mut S) -> Result<()> where S: Serializer {
        let (sign, bytes) = *self.to_bytes_le();
        let length = bytes.len();
        match length {
            0...255 => self.get_small_big_number(sign, length as u8, bytes),
            _ => self.get_large_big_number(sign, length as u32, bytes),
        }
    }
}

As a result, the complier generates an error:

src/serializers.rs:591:71: 591:81 error: `Serializer` is not a trait [E0404]
src/serializers.rs:591     fn serialize<S>(&self, serializer: &mut S) -> Result<()> where S: Serializer {
                                                                                             ^~~~~~~~~~
src/serializers.rs:591:71: 591:81 help: run `rustc --explain E0404` to see a detailed explanation
error: cannot continue compilation due to previous error
error: Could not compile `bert`.

How can I connect all this stuff together and provide method to serialize BigInt type? I've also trying to implement serializing for BigInt type from the num crate as described there, but getting the same error as I said earlier.

[arthurprs]

fn serialize<S>(&self, serializer: &mut S) -> Result<()> where S: Serializer { should be fn serialize<S>(&self, serializer: &mut S) -> Result<()> where S: ser::Serializer {

note the ser::

you can also bring the traits to scope using use

[Relrin]

@arthurprs I've tried it, but this didn't help me also:

src/serializers.rs:593:5: 603:6 error: method `serialize` has an incompatible type for trait:
 expected associated type,
    found enum `errors::Error` [E0053]
src/serializers.rs:593     fn serialize<S>(&self, serializer: &mut S) -> Result<()> where S: ser::Serializer
                           ^
src/serializers.rs:593:5: 603:6 help: run `rustc --explain E0053` to see a detailed explanation

[arthurprs]

I think the result type is wrong, please have a look at https://serde.rs/impl-serialize.html

[Relrin]

Type for which I'm implementing right now isn't default (Rust) type. I'd taken it from the num crate, which provide access to the big numbers (or shortly BigInt in our case). After than I have looked into the implementing custom serialiazing/deserializing section of the serde-rs docs, I have written next part of code (by analogy):

struct BigInteger(BigInt);

impl Deref for BigInteger {
    type Target = BigInt;
    fn deref(&self) -> &Self::Target {
        &self.0
    }
}

impl ser::Serialize for BigInteger {
    fn serialize<S>(&self, serializer: &mut S) -> Result<()> where S: ser::Serializer
    {
        let (sign, bytes) = self.to_bytes_le();
        let length = bytes.len();
        match length {
            0...255 => self.get_small_big_number(sign, length as u8, bytes),
            _ => self.get_large_big_number(sign, length as u32, bytes),
        };

        Ok(())
    }
}

But nonetheless, the Rust compiler still generating the same error with an incompatible type for trait. 🤔

[arthurprs]

Result<> usually takes 2 type arguments, your's have one, it's supposed to be Result<(), S::Error>

[dtolnay]

The Serialize trait returns Result<(), S::Error>. Your impl is returning bert::Result<()> which is Result<(), bert::Error>. Your impl needs to return the same type that is returned in the definition of the Serialize trait.

src/serializers.rs:593:5: 603:6 error: method `serialize` has an incompatible type for trait:
 expected associated type,
    found enum `errors::Error` [E0053]
src/serializers.rs:593     fn serialize<S>(&self, serializer: &mut S) -> Result<()> where S: ser::Serializer
                           ^
src/serializers.rs:593:5: 603:6 help: run `rustc --explain E0053` to see a detailed explanation

The compiler is saying that you returned something containing enum 'errors::Error' and it expected something containing an associated type (S::Error) instead.

[Relrin]

Hmmm. I've moved my code with this custom type to an another rust source file and fixed some types of interface, but still getting the same error:

use std::ops::Deref;
use std::result::Result; // instead of errors::{Result}

use num::bigint::{BigInt, Sign};
use serde::ser;

use errors::{Error}; // from my library

pub struct BigInteger(BigInt);

impl Deref for BigInteger {
    type Target = BigInt;
    fn deref(&self) -> &Self::Target {
        &self.0
    }
}

impl ser::Serialize for BigInteger {
    fn serialize<S>(&self, serializer: &mut S) -> Result<(), Error>
        where S: ser::Serializer
    {
        let (sign, bytes) = self.to_bytes_le();
        let length = bytes.len();
        let binary = match length {
            0...255 => serializer.get_small_big_number(sign, length as u8, bytes),
            _ => serializer.get_large_big_number(sign, length as u32, bytes),
        };
        serializer.writer.write_all(binary.as_slice()).map_err(From::from);
        Ok(())
    }
}

[dtolnay]

Your impl is still returning Result<(), bert::Error>. Your impl needs to return Result<(), S::Error>. For example when the serializer S is a JSON serializer, you don't want the JSON serializer to be producing BERT errors.

[Relrin]

Ok, I've got it. The signature of result of my interface has changed from Result<(), bert::Error> to the Result<(), S::Error>.

Now we have a little bit different error. It's related to the S. So, as far as I understand that happens because compiler think that its Serializer, not my Serializer<W> :

src/wrappers.rs:28:35: 28:55 error: no method named `get_small_big_number` found for type `&mut S` in the current scope
src/wrappers.rs:28             0...255 => serializer.get_small_big_number(sign, length as u8, bytes),
                                                     ^~~~~~~~~~~~~~~~~~~~
src/wrappers.rs:29:29: 29:49 error: no method named `get_large_big_number` found for type `&mut S` in the current scope
src/wrappers.rs:29             _ => serializer.get_large_big_number(sign, length as u32, bytes),
                                               ^~~~~~~~~~~~~~~~~~~~
src/wrappers.rs:31:9: 31:26 error: attempted access of field `writer` on type `&mut S`, but no field with that name was found
src/wrappers.rs:31         serializer.writer.write_all(binary.as_slice()).map_err(From::from);
                           ^~~~~~~~~~~~~~~~~
error: aborting due to 3 previous errors
error: Could not compile `bert`.

How can I fix this? Change a template? Or maybe make one more wrapper?

[Relrin]

@dtolnay Does serde framework have a "special wrappers" which are help me to promt to the compiler, that necessary to use Serializer<W> where W: io::Write?

[dtolnay]

Not yet, as that depends on specialization which is not stable yet. We are working on a design in #455 for when it makes it to stable Rust. Separately we're working on the bignum story in https://github.com/serde-rs/rfcs/pull/1.

For now I would recommend serializing BigInteger using serialize_newtype_struct with a special name that your Serializer<W> recognizes and treats the value as a BigInteger. Something like:

let (sign, bytes) = self.to_bytes_le();
let sign = Sign(sign); // a serializable newtype wrapper around num::bigint::Sign
let bytes: serde::bytes::Bytes = bytes.into();
serializer.serialize_newtype_struct("BertBigInteger", (sign, bytes));

Then your Serializer looks for the specific sequence of calls:

serialize_newtype_struct("BertBigInteger", ...)
serialize_seq(Some(2))
serialize_seq_elt(...)
serialize_unit_variant("Sign", ...)
serialize_seq_elt(...)
serialize_bytes(...)
serialize_seq_end(...)

And instead of serializing them the normal way, it serializes the bytes as a BERT big_number.

[Relrin]

When I will use code which similar as at the previous comment, then how to write a bytes which is a field of the Serializer<W> structure? I'm asking about this, because our serializer has a different methods for serializing integers/floats/enums/etc. And each method of serializer writing into the Serializer.writer which using as a buffer. So, I'm expecting that I can write something like this:

impl ser::Serialize for BigInteger {
    fn serialize<S>(&self, serializer: &mut S) -> Result<(), S::Error>
        where S: ser::Serializer
    {
        let (sign, bytes) = self.to_bytes_le();
        let sign = Sign(sign); // a serializable newtype wrapper around num::bigint::Sign
        let bytes: serde::bytes::Bytes = bytes.into();

        // the next lines very important there
        let bignum_tag = vec![BertTag::Bignum as u8];  // 1
        serializer.writer.write_all(bignum_tag.as_slice());  // 2

        // some serializing further...
        Ok(())
    }
}

Especially, "marked" lines are required for me, I will have written it to the buffer. It helps to the Erlang serializer understand, that our BERT serializer provides him a BigNum instead of something wrong.

[dtolnay]

That won't work until we start using specialization. For now your Serialize implementation has to work for any Serializer. If you serialize BigInteger using the normal Serializer methods like I showed, then your BERT Serializer can tell when a BigInteger is being serialized based on the serialize_newtype_struct("BertBigInteger" ...) call and it can write the BertTag::BigNum then. Every other Serializer will serialize it as a normal newtype struct.

[Relrin]

I don't know how to do this stuff better, because for each type of data which serializer writing to the buffer, it should to specify a tag (as a u8 number). The best solution is just make a call write/write_all for io::Write of a passed buffer.

There you can look onto my current state of codebase.

As a result, I'm expecting that Erlang's code can get a part of data like [131, 110, 5, 97, 200, ....]

[dtolnay]

This code needs to change to handle _name == "BertBigInteger" as a special case.

[Relrin]

I'm somewhere close to solve this issue. So, I have the next code at the moment:

pub const BIGNUM_STRUCT_NAME: &'static str = "_BertBigNumber";

impl ser::Serialize for BertBigInteger {
    fn serialize<S>(&self, serializer: &mut S) -> Result<(), S::Error>
        where S: ser::Serializer
    {
        let (num_sign, bytes) = self.to_bytes_le();
        let length = bytes.len();
        let sign: u8 = match num_sign {
            Sign::Plus => 0,
            Sign::Minus => 1,
            _ => panic!("Invalid bignum sign.")
        };
        let binary: Vec<u8> = match length {
            0...255 => {
                vec![BertTag::SmallBigNum as u8, length as u8, sign]
            },
            _ => {
                let mut binary = vec![BertTag::LargeBigNum as u8];
                binary.write_u32::<BigEndian>(length as u32).unwrap();
                binary.write_u8(sign).unwrap();
                binary
            }
        };
        binary.extend(bytes.iter().clone());

        serializer.serialize_newtype_struct(BIGNUM_STRUCT_NAME, &binary);
    }
}

// serializer
#[inline]
    fn serialize_newtype_struct<T>(
        &mut self, _name: &'static str, value: T
    ) -> Result<()> where T: ser::Serialize {
        match _name {
            BIGNUM_STRUCT_NAME => {
                try!(self.writer.write_all(value.as_slice()));
                Ok(())
            },
            _ => {
                let header = vec![BertTag::SmallTuple as u8, 2u8];
                try!(self.writer.write_all(header.as_slice()));

                let structure_name_atom = self.get_atom(_name);
                try!(self.writer.write_all(structure_name_atom.as_slice()));

                value.serialize(self)
            }
        }
    }

How can I somehow convert passed value (as T type) into the Vec<u8> or it slice?

[dtolnay]

In serialize_newtype_struct you are given a value of type T: Serialize which means literally the only useful thing Rust will let you do with it is call Serialize::serialize on it. So call Serialize::serialize:

BIGNUM_STRUCT_NAME => {
    // some Serializer that implements serialize_bytes only (and the other methods panic)
    let mut bignum_serializer = BigNumSerializer(&mut self.writer);
    value.serialize(&mut bignum_serializer)
}

Also you're going to want to wrap binary in serde::bytes::Bytes or ByteBuf as we talked about in #518.

[Relrin]

I guess it will be a good solution. Can I somehow do not implement manually all required methods? Maybe serde have a macro which will make dummies for all specified methods?

[dtolnay]

You can avoid implementing all the methods by doing it this way instead:

BIGNUM_STRUCT_NAME => {
    self.in_bignum = true;
    value.serialize(self)
}

where then serialize_bytes will check for the in_bignum flag.

[Relrin]

Stuck with Serializer<W> and BigNumSerializer<W> instances.

This is the current state of code:

impl<W> ser::Serializer for BigNumSerializer<W> where W: io::Write {
    // ...
    #[inline]
    fn serialize_bytes(&mut self, data: &[u8]) -> Result<()> {
        for byte in data {
            try!(self.writer.write_all(&[*byte]))
        };
        Ok(())
    }
}

impl<W> ser::Serializer for Serializer<W> where W: io::Write {
    // ....

    #[inline]
    fn serialize_newtype_struct<T>(
        &mut self, _name: &'static str, value: T
    ) -> Result<()> where T: ser::Serialize {
        match _name {
            BIGNUM_STRUCT_NAME => {
                let mut bignum_serializer = BigNumSerializer::new(
                    &mut self.writer
                );
                value.serialize(&mut bignum_serializer)  // <-- there
            },
            _ => {
                let header = vec![BertTag::SmallTuple as u8, 2u8];
                try!(self.writer.write_all(header.as_slice()));

                let structure_name_atom = self.get_atom(_name);
                try!(self.writer.write_all(structure_name_atom.as_slice()));

                value.serialize(self)
            }
        }
    }

Not so good understand, why calling Serializer<W> instead of method of BigNumSerializer<W>. As a result I've taken a wrong data in my tests:

---- test_serializers::test_serialize_bignum stdout ----
    thread 'test_serializers::test_serialize_bignum' panicked at 'assertion failed: `(left == right)` (left: `[131, 109, 0, 0, 0, 5, 110, 2, 0, 232, 3]`, right: `[131, 110, 2, 0, 232, 3]`)', tests/test_serializers.rs:627
note: Run with `RUST_BACKTRACE=1` for a backtrace.

failures:
    test_serializers::test_serialize_bignum

test result: FAILED. 31 passed; 1 failed; 0 ignored; 0 measured

Result on the left side is a binary, not a big number.

[dtolnay]

As I said, you need serde::bytes::Bytes or else it will go through Vec<T>'s Serialize implementation instead of serialize_bytes.

[Relrin]

Yeah, I already though about it. I'm using serde::bytes::ByteBuf as you propose earlier:

impl ser::Serialize for BertBigInteger {
    fn serialize<S>(&self, serializer: &mut S) -> Result<(), S::Error>
        where S: ser::Serializer
    {
        let (num_sign, bytes) = self.to_bytes_le();
        let length = bytes.len();
        let sign: u8 = match num_sign {
            Sign::Plus => 0,
            Sign::Minus => 1,
            _ => panic!("Invalid bignum sign.")
        };
        let mut binary: Vec<u8> = match length {
            0...255 => {
                vec![BertTag::SmallBigNum as u8, length as u8, sign]
            },
            _ => {
                let mut binary = vec![BertTag::LargeBigNum as u8];
                binary.write_u32::<BigEndian>(length as u32).unwrap();
                binary.write_u8(sign).unwrap();
                binary
            }
        };
        binary.extend(bytes.iter().clone());

        let bytes: serde::bytes::ByteBuf = binary.into();
        serializer.serialize_newtype_struct(BIGNUM_STRUCT_NAME, bytes)
    }
}

[dtolnay]

I don't understand either. Figure out where the unexpected data is being written and then figure out what calls it made to get there. I can take another look when I get home from work - it would help if you can point me to a commit that I can look at locally.

[Relrin]

The issue was in the signature of new(..) method of BigNumSerializer<W>. It had returned Serializer<W> instead of expected serializer. Thank you so much for the answers. It really helped me. ☺️

[dtolnay]

Happy to help and I am glad you figured it out. Sorry for the complicated approach - we (and Rust) are working on a better one via specialization.