Cannot create a tensor proto whose content is larger than 2GB

One of the underlying data structures of Tensorflow (protocol buffers) are capped at 2GB. On certain large networks, this makes nengo_dl.Simulator fail where nengo.Simulator and nengo_ocl.Simulator succeed.

This issue and some workarounds are documented https://stackoverflow.com/questions/51470991/create-a-tensor-proto-whose-content-is-larger-than-2gb

See the Nengo forum for a conversation about this issue https://forum.nengo.ai/t/nengo-dl-cannot-create-a-tensor-proto-whose-content-is-larger-than-2gb/1243

To reproduce: (you will need >10GB RAM to make it to the exception in question)

import numpy as np
import nengo
import nengo_dl

transform = np.random.normal(size=(3*10**4, 3*10**4))  # Fails
# transform = np.zeros((3*10**4, 3*10**4))  # Works
print(transform.nbytes / 1e9, 'GB')  # 7.2 GB

with nengo.Network() as net:
    ens = nengo.Ensemble(transform.shape[0], 1)
    nengo.Connection(ens.neurons, ens.neurons, transform=transform)
sim = nengo_dl.Simulator(net)

raises

ValueError: Tried to convert 'input' to a tensor and failed. Error: Cannot create a tensor proto whose content is larger than 2GB.

see the forum for full stack trace.

nengo / nengo-dl

Cannot create a tensor proto whose content is larger than 2GB #160