search

zhoususheng / berkeleylm

Automatically exported from code.google.com/p/berkeleylm
1 stars 0 forks source link

Runtime exception: Hash map is full with 100 keys. Should never happen. #11

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago 
What steps will reproduce the problem?
1. building a LM over some input files consistently generates this exception
2.
3.

What is the expected output? What do you see instead?

The expected output is a learned LM written to a file. Instead, I get the 
exception:
Runtime exception: Hash map is full with 100 keys. Should never happen.

What version of the product are you using? On what operating system?
berkeleylm 1.1.3 on Windows 7

Please provide any additional information below.

java.lang.RuntimeException: Hash map is full with 100 keys. Should never happen.
    at edu.berkeley.nlp.lm.map.ExplicitWordHashMap.put(ExplicitWordHashMap.java:56)
    at edu.berkeley.nlp.lm.map.HashNgramMap.putHelpWithSuffixIndex(HashNgramMap.java:283)
    at edu.berkeley.nlp.lm.map.HashNgramMap.putWithOffsetAndSuffix(HashNgramMap.java:247)
    at edu.berkeley.nlp.lm.io.KneserNeyLmReaderCallback.addNgram(KneserNeyLmReaderCallback.java:171)
    at edu.berkeley.nlp.lm.io.KneserNeyLmReaderCallback.call(KneserNeyLmReaderCallback.java:148)
    at edu.berkeley.nlp.lm.io.KneserNeyLmReaderCallback.call(KneserNeyLmReaderCallback.java:37)
    at edu.berkeley.nlp.lm.io.TextReader.countNgrams(TextReader.java:80)
    at edu.berkeley.nlp.lm.io.TextReader.readFromFiles(TextReader.java:53)
    at edu.berkeley.nlp.lm.io.TextReader.parse(TextReader.java:47)
    at edu.berkeley.nlp.lm.io.LmReaders.createKneserNeyLmFromTextFiles(LmReaders.java:301)
    at edu.berkeley.nlp.lm.io.MakeKneserNeyArpaFromText.main(MakeKneserNeyArpaFromText.java:57)
    at yr.haifa.NLP.lm.BerkleyLanguageModel.train(BerkleyLanguageModel.java:51)

Original issue reported on code.google.com by jethro...@gmail.com on 24 Apr 2013 at 1:14

GoogleCodeExporter commented 8 years ago 
Can you try building the latest version from SVN? I believe I have fixed this 
bug, but I was waiting for others to verify before releasing it.

Original comment by adpa...@gmail.com on 24 Apr 2013 at 2:16

GoogleCodeExporter commented 8 years ago 
Problem solved!

Original comment by jethro...@gmail.com on 28 Apr 2013 at 6:01

GoogleCodeExporter commented 8 years ago 
Version 1.1.4 has been released.

Original comment by adpa...@gmail.com on 12 May 2013 at 12:59

GoogleCodeExporter commented 8 years ago 
I had this problem, replaced the 1.1.3 with 1.1.5 and I still have it. Any 
thoughts?

Original comment by benajiba...@gmail.com on 5 Dec 2013 at 8:23

GoogleCodeExporter commented 8 years ago 
I'm also getting this bug on 1.1.5. Is there a chance it's related to input 
data?

Original comment by bryan.t....@gmail.com on 6 Dec 2013 at 1:56

GoogleCodeExporter commented 8 years ago 
Can you find a small input file + command line that I can run myself to 
reproduce the error?

Original comment by adpa...@gmail.com on 6 Dec 2013 at 4:01

GoogleCodeExporter commented 8 years ago 
I'm getting the error in this relatively small file of Japanese text in 1.1.5.

Original comment by acgris...@gmail.com on 7 Mar 2014 at 10:12

Attachments:

GoogleCodeExporter commented 8 years ago 
Also getting this in 1.1.5

Original comment by AWNyst...@gmail.com on 3 Jul 2014 at 6:47

GoogleCodeExporter commented 8 years ago 
I also get in in 1.1.5

Original comment by arthurxl...@gmail.com on 5 Sep 2014 at 4:10

GoogleCodeExporter commented 8 years ago 
I have added a fix in the latest from SVN. Could you all please try updating 
and seeing if your issues are fixed. I will release a new version once I've 
verified the issue is fixed for all.

Original comment by adpa...@gmail.com on 7 Sep 2014 at 7:00

GoogleCodeExporter commented 8 years ago 
[deleted comment]
GoogleCodeExporter commented 8 years ago 
jethro...@gmail.com, How you get to solve the problem? Also getting this in 
1.1.5. Thanks.

Original comment by alexport...@gmail.com on 2 Nov 2014 at 10:59

GoogleCodeExporter commented 8 years ago 
[deleted comment]
GoogleCodeExporter commented 8 years ago 
Problem solved..

Original comment by alexport...@gmail.com on 6 Nov 2014 at 6:56

GoogleCodeExporter commented 8 years ago 
Hi, just to confirm: is the problem solved at head in SVN?

Original comment by adpa...@gmail.com on 6 Dec 2014 at 11:23

GoogleCodeExporter commented 8 years ago 
Yes. I downloaded the latest version of SVN and it worked. Thank you.

Original comment by alexport...@gmail.com on 7 Dec 2014 at 12:46