Too many segments error

[clavin-xlnx]

We are using capnproto-java to encode a large FPGA device database but we have run into what seems a bug or limitation (see Xilinx/RapidWright#176). When we have essentially a long list of arrays, we get an error on read of the message similar to:

Exception in thread "main" java.io.IOException: too many segments
    at org.capnproto.Serialize.read(Serialize.java:69)
    at org.capnproto.Serialize.read(Serialize.java:52)
    at com.xilinx.rapidwright.examples.TestCase.main(TestCase.java:44)

It appears that there is a limit set of 512 segments that can be read which is fine (legitimate security precaution). However, when using capnproto-java to encode this data, it generates several thousand segments for some reason. I have some example code that can reproduce this issue with our existing schema (see Gist: https://gist.github.com/clavin-xlnx/1688dfaf0e708737c2743577111a4c7f):

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.channels.Channels;

import org.capnproto.MessageBuilder;
import org.capnproto.Serialize;
import org.capnproto.StructList;

import com.xilinx.rapidwright.interchange.DeviceResources;

public class TestCase {
  public static void main(String[] args) throws FileNotFoundException, IOException {
    int nodeCount = 600000;
    int wireArrayLength = 500;

    // Construct message
    MessageBuilder message = new MessageBuilder();
    StructList.Builder<DeviceResources.Device.Node.Builder> nodeBuilders = 
        message.initRoot(DeviceResources.Device.factory).initNodes(nodeCount);
    for(int i=0; i < nodeCount; i++) {
      nodeBuilders.get(i).initWires(wireArrayLength);
    }

    // Write out message and read back from a file
    String fileName = "tooManySegments.message";
    Serialize.write(Channels.newChannel(new FileOutputStream(fileName)), message);
    Serialize.read(Channels.newChannel(new FileInputStream(fileName)));
  }
}

To reproduce the error, run the following:

wget https://gist.githubusercontent.com/clavin-xlnx/1688dfaf0e708737c2743577111a4c7f/raw/e6b1a5918dc375ce967b799cf7d4823d1d282a69/TestCase.java
mkdir -p com/xilinx/rapidwright/interchange
wget https://raw.githubusercontent.com/Xilinx/RapidWright/master/src/com/xilinx/rapidwright/interchange/DeviceResources.java -O com/xilinx/rapidwright/interchange/DeviceResources.java
wget https://repo1.maven.org/maven2/org/capnproto/runtime/0.1.5/runtime-0.1.5.jar
javac TestCase.java com/xilinx/rapidwright/interchange/DeviceResources.java
java -cp .:runtime-0.1.5.jar TestCase

For reference, the original schema used to generate the DeviceResources.java file can be found here: https://github.com/SymbiFlow/fpga-interchange-schema/blob/04bc92a22306351e1b848a6d8a3fd7720c4aed58/interchange/DeviceResources.capnp

[mithro]

+1

[dwrensha]

Thanks for the report! This was an integer overflow problem. It is fixed in https://github.com/capnproto/capnproto-java/commit/eb73990798758bd08bc9411127a95749df4847dc.

[dwrensha]

I have published 0.1.6 with this change to the central repository.