Open yajivunev opened 2 years ago
@yajivunev Were you able to solve this? Currently struggling with the same issue. Originally I would get a page size error, but after ramping up the page size I get the same error as you, even though a single sample should easily fit into my memory.
Update: I was able to resolve this issue by further increasing the page size even more
Hello!
I'm having trouble writing my dataset.
The batches are already made as numpy arrays, so I just have to feed them through a dataset so it can be written to .beton. Each batch has 4 numpy arrays (see field dictionary below), and all batches are in a dictionary called
batches
with indices as keys.batches[n]
would return a tuple of the 4 numpy arrays. I even stack them by field to resemble to LinearRegressionDataset example in the docs. i.e, I dobatches = [np.stack([batches[i][j] for i in range(len(batches))]) for j in range(4)]
, just in case.My dataset class is:
Here is my writer and write command:
This always results in the follow traceback and hangs until I ctrl+c:
Each batch is about 10mb.
Appreciate any help! Thank you so much!! Cheers on the project, would love to get it working for my case :)