meriems / sphinxsearch

Automatically exported from code.google.com/p/sphinxsearch
GNU General Public License v2.0
0 stars 0 forks source link

wordbreaker breaks on long line #20

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago
wordbreaker breaks on long line 
and it is also generally not possible to use dicts bigger than 512 chars.

What steps will reproduce the problem?
1. use a line longer than 512 chars in the dict file (aka wordbreaker-dict.txt)
2. if chat at 511 is not a "space", one gets "bad freqdict line: zero freq"

What is the expected output? 
- program runs
What do you see instead?
- program breaks with "bad freqdict line: zero freq"

What version of the product are you using? 
- development
On what operating system?
- linux
Please provide any additional information below.
- program should support also multi-lines dict files.
- program should read line by line till end of file.

Original issue reported on code.google.com by aab...@gmail.com on 9 Jun 2013 at 10:56