torch / torch7

http://torch.ch
Other
8.97k stars 2.38k forks source link

torch.load runs very slow in for loop #1110

Open shatrughanmodi opened 6 years ago

shatrughanmodi commented 6 years ago

I am new to torch. I have created a lua script in which I am loading 64 't7' files, which contains my training data, in a for loop. Each file is approx. 1 GB in size. Following is the sample code:

for i = 1,64 do
    t = sys.clock();
    x = torch.load("input_" .. i .. ".t7");
    print("Data loaded in" .. (sys.clock()-t) .. " seconds");
    --process data
end

On average it took 28 seconds for loading 1 file. I ran this code directly on 'th' prompt also but same results.

Whereas if I load 1 data file in th prompt it took approx. 1-2 secs.

I am not able to identify the issue. Kindly suggest something.