Closed rasoolims closed 7 years ago
In the new version (to be released very soon), there should be a special input format for sentence-segmented but untokenized input (one sentence per line), I believe.
In the current version v1.0.0, you have to use a workaround where you merge together the extra-split sentences (or maybe input_format_presegmented_tokenizer, but I am not sure how to use it).
What should be the path for PERL_INCLUDE? I tried different paths to install Perl bindings but was not successful.
If you need the Perl binding, install it from CPAN, e.g. with cpanm Ufal::UDPipe
.
The workaround I referenced can be easily done in any other of the supported programming languages.
That said, I would suggest to wait for the new release. It will published soon and the new models for this release were already published.
As @martinpopel said, tokenizer respecting given segmentation will be present in UDPipe 1.1.
As for PERL_INCLUDE, if you prefer to compile the bindings manually instead of using cpan, PERL_INCLUDE should point to a directory containing perl.h
, as the Makefile suggests. On my machine it is at /usr/lib/x86_64-linux-gnu/perl/5.20/CORE
.
I found a hacky way to do this by changing this line to the following:
return input_format::new_presegmented_tokenizer(result);
This is what the new code does (but in a configurable way :-), see https://github.com/ufal/udpipe/blob/pre1.1/src/model/model_morphodita_parsito.cpp#L37 .
The UDPipe 1.1 is out, with presegmented
tokenizer option respecting input segmentation (documented at https://ufal.mff.cuni.cz/udpipe/users-manual#run_udpipe_tokenizer).
Hi, I was not able to find any option to turn off sentence delimitation for tokenization. My data needs to be with the same number of sentences after tokenization but UDPIPE separates some of the sentences. Thanks