Overflow on associative arrays with length > 65535

[glathoud]

Thanks for the great lib, using it everyday now.

One of my associative arrays went above 65535 elements and I got an overflow error from cerealise, there's code below to reproduce the issue. I noticed the ushorts in cerealed's source code and tried somewhat haphazardly to replace them with size_t, got it half working but not fully.

Would it be thinkable to have an optional type parameter, default ushort (as now) ? Or any other solution supporting bigger assoc. arrays ? . overflow use case on assoc. arrays:

import cerealed;

import std.algorithm;
import std.range;
import std.stdio;

void main()
{
  test( 10 );  // ok
  test( 65535 );  // ok
  test( 65536 );  // overflow
}

ubyte[] test( in size_t n )
{
  bool[size_t] data;
  n.iota.each!( (i) { data[ i ] = true; } );

  writeln( "About to cerealise data for n: ", data.length );
  auto ret = cerealise( data );
  writeln( "Done." );

  return ret;
}

[atilaneves]

There's a way to select the size of the variable used to encode the length - by default it's ushort, explaining the limit you encountered. However, it's only usable at the very fine-grained level, I didn't find a good way to expose this API to the cerealise function. What you can do is mark that array as @NoCereal, and write a postblit that calls grain with a different integral type such as size_t.

[glathoud]

Ok thanks, I'll try.

[arredondos]

What you can do is mark that array as @NoCereal, and write a postblit that calls grain with a different integral type such as size_t.

Hello Atila. I'm having a similar problem to this one. Could you add an example of how to do this postblit with grain please? Thanks!

[atilaneves]

@arredondos This is a good example: https://github.com/atilaneves/mqtt/blob/9d64362a9139bec41d76ff4f886c2c716770ae5c/mqttd/message.d#L122