Should dp.TextSource provide an impl of frequencyTree()?

[hsheil]

When running recurrentlanguagemodel.lua with a custom text dataset and --softmaxtree the following error occurs:

/home/hsheil/torch/install/bin/luajit: recurrentlanguagemodel.lua:222: attempt to call method 'frequencyTree' (a nil value)
stack traceback:
    recurrentlanguagemodel.lua:222: in main chunk
    [C]: in function 'dofile'
    ...heil/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:131: in main chunk
    [C]: at 0x00406670

Line 222 is the culprit:

   elseif opt.softmaxtree then -- uses frequency based tree
      local tree, root = ds:frequencyTree()
      softmax = nn.SoftMaxTree(inputSize, tree, root, opt.accUpdate)
   end

I guess --softmaxtree is most beneficial for the billion word dataset anyway - but to avoid this error should dp.TextSource provide an impl of the frequencyTree() method?

Cmd-line params to reproduce (only the combination of --softmaxtree and --dataset TextSource is pertinent I think):

th recurrentlanguagemodel.lua --lstm --cuda --dataPath data --dataset TextSource --softmaxtree

[adonisues]

I had same problem. It seems that there is no initialization of softmaxtree. So I have solved it as following

add below( it is from dp/data/penntreebank.lua) to dp/data/textsource.lua

-- this can be used to initialize a SoftMaxTree function TextSource:frequencyTree(binSize) binSize = binSize or 100 local wf = torch.IntTensor(self:wordFrequency()) local vals, indices = wf:sort() local tree = {} local id = indices:size(1) function recursiveTree(indices) if indices:size(1) < binSize then id = id + 1 tree[id] = indices return end local parents = {} for start=1,indices:size(1),binSize do local stop = math.min(indices:size(1), start+binSize-1) local bin = indices:narrow(1, start, stop-start+1) assert(bin:size(1) <= binSize) id = id + 1 table.insert(parents, id) tree[id] = bin end recursiveTree(indices.new(parents)) end recursiveTree(indices) return tree, id end