twitter-archive / torch-ipc

A set of primitives for parallel computation in Torch
Apache License 2.0
95 stars 28 forks source link

ipc.map with large table #14

Closed joeyhng closed 8 years ago

joeyhng commented 8 years ago

I am trying to do some file processing in parallel, so I made a table of filenames (around 20000 files), and put it as the first argument in ipc.map, and use the 2nd argument as the process ID to handle different files.

When I tried to pass a large table inside the ipc.map function to do some additional processing, it failed.

The following code reproduces the error:

local ipc = require 'libipc'

local n = 906

local data = {}
for i=1,n do
  data[i]=i
end
ipc.map(3, function(data, mapid)
  return data[mapid]
end, data):join()

print('finish')

The error message showed:

PANIC: unprotected error in call to Lua API (attempt to index a function value)

The code runs fine to me when n < 906. Thanks a lot!