search

Ignorato / lapdftext

Automatically exported from code.google.com/p/lapdftext
0 stars 0 forks source link

Word tokenisation issue #4

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago 
What steps will reproduce the problem?
1. make a pdf containing multiple dashed words (test-case-word) and apostrophe 
(oma's)
2. generate output
3. check words found

What is the expected output? What do you see instead?
test-case-word should be detected as one word, but was split
oma's is split in words oma and s, dropping the ', which is part of the word

What version of the product are you using? On what operating system?
latest snapshot, Ubuntu Linux

Please provide any additional information below.

Word tokenisation is not trivial, especially when the apostroph is involved. 
Nevertheless, - and ' are word characters in Dutch.

Original issue reported on code.google.com by i...@taaltik.nl on 10 Dec 2012 at 6:03

GoogleCodeExporter commented 8 years ago 
Can you attach the test document to a comment? We'll try to address the 
question in an upcoming release.

Original comment by GullyBu...@gmail.com on 29 Jan 2013 at 5:25