search

aymara / lima

The Libre Multilingual Analyzer, a Natural Language Processing (NLP) C++ toolkit.
http://aymara.github.io/lima/
Other
105 stars 21 forks source link

If using the same tokenizer instance, deeplima tokenizer blocks on second file #172

Open kleag opened 5 months ago

kleag commented 5 months ago

Describe the bug When deeplima is asked to analyze several files, the first one is analyzed correctly, but then the program stalls.

To Reproduce Steps to reproduce the behavior:

  1. Run deeplima --tok-model ~/.local/share/lima/resources/RnnTokenizer/ud/tokenizer-eng-UD_English-EWT.pt test-eng*.txt
  2. See that the program is stalled before printing the result for second file
deeplima --tok-model ~/.local/share/lima/resources/RnnTokenizer/ud/tokenizer-eng-UD_English-EWT.pt  test-eng*.txt
one_pos_mask == 00000000 00000000 00000000 00000000 00000000 00011111 11111111 11111111
mask  [0]    == 00000000 00000000 00000011 11111111 11111111 11100000 00000000 00000000
shift [0]    == 21
one_pos_mask == 00000000 00000000 00000000 00000000 00000000 00011111 11111111 11111111
mask  [1]    == 00000000 00000000 00000011 11111111 11111111 11111111 11111111 11111111
shift [1]    == 0
one_pos_mask == 00000000 00000000 00000000 00000000 00000000 00011111 11111111 11111111
mask  [2]    == 01111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111
shift [2]    == 0
one_pos_mask == 00000000 00000000 00000000 00000000 00000000 00000000 00000000 11111111
mask  [3]    == 00000000 00000000 00000000 00000000 00000000 00000000 11111111 00000000
shift [3]    == 8
one_pos_mask == 00000000 00000000 00000000 00000000 00000000 00000000 00000000 11111111
mask  [4]    == 00000000 00000000 00000000 00000000 00000000 00000000 11111111 11111111
shift [4]    == 0
one_pos_mask == 00000000 00000000 00000000 00000000 00000000 00000000 00000000 11111111
mask  [5]    == 00000000 00000000 00000000 00000000 00000000 11111111 11111111 11111111
shift [5]    == 0
one_pos_mask == 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000001
mask  [6]    == 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000010
shift [6]    == 1
locked_buffer_set_t::init
locked_buffer_t::locked_buffer_t()0x655515b3a530
locked_buffer_t::locked_buffer_t()0x655515b3a548
locked_buffer_t::locked_buffer_t()0x655515b3a560
locked_buffer_t::locked_buffer_t()0x655515b3a578
Reading file: test-eng10.txt
SegmentationImpl::parse_from_stream
locked_buffer_set_t::init
SegmentationImpl::parse_from_stream Reading callback: 55 bytes, continue_reading=0 counter=55
SegmentationImpl::parse_from_stream locking (m_buff_set) buff 0
locked_buffer_t::lock 0x655515b3a530 0 -> 1
BUFFS:  | 1 | 0 | 0 | 0 |
SLOTS:  | 2 | 0 | 0 | 0 |
BUFFS:  | 1 | 0 | 0 | 0 |
SLOTS:  | 2 | 0 | 0 | 0 |
BUFFS:  | 1 | 0 | 0 | 0 |
SLOTS:  | 2 | 0 | 0 | 0 |
1       electric        _       _       _       _       0       _       _       _
2       servo   _       _       _       _       1       _       _       _
3       motor   _       _       _       _       2       _       _       _
4       six     _       _       _       _       3       _       _       _
5       axes    _       _       _       _       4       _       _       _
6       robot   _       _       _       _       5       _       _       _
7       application     _       _       _       _       6       _       _       _
8       domain  _       _       _       _       7       _       _       _
locked_buffer_t::unlock 0x655515b3a530 1 -> 0
BUFFS:  | 0 | 0 | 0 | 0 |
SLOTS:  | 0 | 0 | 0 | 0 |
Parsed: 8 in 0.86 seconds.
Parsing speed: 9.32 tokens / sec.
SegmentationImpl::finalize
Reading file: test-eng11.txt
SegmentationImpl::parse_from_stream
locked_buffer_set_t::init
SegmentationImpl::parse_from_stream Reading callback: 124 bytes, continue_reading=0 counter=124
SegmentationImpl::parse_from_stream locking (m_buff_set) buff 0
locked_buffer_t::lock 0x655515b3a530 0 -> 1
BUFFS:  | 1 | 0 | 0 | 0 |
SLOTS:  | 1 | 0 | 0 | 0 |
BUFFS:  | 1 | 0 | 0 | 0 |
SLOTS:  | 1 | 0 | 0 | 0 |
BUFFS:  | 1 | 0 | 0 | 0 |
SLOTS:  | 1 | 0 | 0 | 0 |
  1. In gdb, one can see that it loops in segmentation_impl.cpp here: https://github.com/aymara/lima/blob/7a33899a4c731bacd7aa628d4750bca1c5160a8c/deeplima/libs/tasks/segmentation/inference/segmentation_impl.cpp#L156

Expected behavior All files should be analyzed successfully.

kleag commented 5 months ago

Temporary solution in b462061 by creating a new instance of the tokenizer on each file. This is not too costly because tokenization models are very small but it cannot be counted as a correction. And in fact, now the tagger crashes on second file.