Support for Lua5.2?

[NightFury13]

This is not really an issue. I need some help. I see that you use warp-ctc by Baidu. Can you tell me what version Lua do you have with your torch? [ th> print(_VERSION) ] I recently upgraded my Lua version from 5.1 to 5.2 in torch to tackle some of the memory limitations of Lua5.1 and am unable to install warp-ctc thereafter.

Reference Issue : https://github.com/baidu-research/warp-ctc/issues/36

Any help/inputs would much appreciated! :D

[SeanNaren]

This took way too long to respond to, sorry about that! I'm running Lua 5.1 currently. I've been slowly fleshing out the tests, which run both 5.1 and 5.2. If there is an issue with 5.2 I'll have a look how to resolve em!

What memory issues have you been having? Something that has helped me is to use this library. I use this instead of tables due to the 2gb limit! Maybe that can help :)

[NightFury13]

Thanks! Yes, I switched to tds and read the large files line-by-line. That helps me tackle the memory limits of Luajit. :)

As for CTC, with Lua5.2+Torch, I am unable to 'make' the library and it throws an error at the early env checking states that : /usr/lib/ld -lluajit not found or something. It works fine with Lua5.1 though. Does Lua5.2 not use Luajit compiler? Or does this mean something else? Any thoughts?

[SeanNaren]

@NightFury13 did you ever manage to fix this?

[NightFury13]

@SeanNaren : turns out it is not supported for 5.2

I used the nnx package and that has a wrapping of the baidu-ctc implementation. That got me through the task.

[SeanNaren]

Sorry to hear that, but thanks for the update! Hope the module is doing well for you :)

[maciejkorzepa]

I am not sure if I understand how the problem was solved. I also installed Lua5.2 as I experienced "not enough memory" errors when memory usage reached ~1GB when preparing lmdb data for 100h Libri dataset. But as NightFury13 mentioned, warp-ctc cannot be successfully installed for Lua5.2. I read the documentation of nnx package and it says that it requires warp-ctc torch bindings, so I guess that Lua5.2 is not an option right? So how do you handle memory issues? (I am using Scientific Linux 6.4)

[NightFury13]

@maciejkorzepa I use Lua5.1 and the nnx packages work just fine for me.

As a side-note, you may want to use TDS datastructures.. These are pretty easy to use (much like Lua tables) C++ based datastructures for torch that use main-memory instead of the Lua memory (don't get me wrong on the terminology here, just my understanding). The out-of-memory issues occur in Lua5.1 and below as there is a bound on the amount of memory that can allocated to a Lua process (~2GB's, but I've seen that even to load a much smaller file say ~700-800MB, this limit is reached and the memory error thrown - memory leak issues?), however this problem is completely thrown away in the case of TDS as now you can allocate all the memory you have as RAM. TDS however would take a little more memory (RAM, not Lua) to store the same amount of data. Wasn't really a problem in practice though. Hope this helps :)

[maciejkorzepa]

@NightFury13 Thanks for your advice! I got back to Lua5.1, so now I need to deal with memory limits. The error shows up when running MakeLMDB script and memory usage reaches ~1GB (seems independent of the number of processes used). The memory info is collected at this point:

if x % 5000 == 0 then
    print(gcinfo())
    collectgarbage()
    print(gcinfo())
end

To be honest, it's my first time with Torch and Lua and I don't really know what changes I should make to employ TDS to solve this problem (It's already used in MakeLMDB script for a couple of variables). Any further suggestions are highly appreciated :)

[NightFury13]

TDS is about exactly the same to use as Lua tables. Start off with the readme on github. I've also only recently started using Torch and feel its about the best thing out there for Deep Learning tasks. For specific changes, keep track of the Lua tables that you are making in the code and then decide which ones can you really do away with. (You can actually replace them all! Except for some specific ones which you feed to your NN layers as inputs). See which tables are storing the huge amounts of data and replace them with a TDS instance.

[SeanNaren]

I can try help out here, I've also ran into memory issues in the MakeLMDB script which is annoying, I think it's due to loading a bunch of large tensors into memory before committing to the LMDB dataset here. Could you try reducing the x value? It may help handle this.