search

openvinotoolkit / openvino_tokenizers

OpenVINO Tokenizers extension
Apache License 2.0
20 stars 16 forks source link

Update Data Type For Windows #215

Closed apaniukov closed 1 month ago

apaniukov commented 1 month ago

CVS-147954

ilya-lavrenov commented 1 month ago

do you think we can cover it with tests?

apaniukov commented 1 month ago

do you think we can cover it with tests?

It should be covered by existing tests (GLM tokenizers with added tokens), but, for some reason, the tokenizers tests have passed on Windows before.

ilya-lavrenov commented 1 month ago

Can the failures be connected with new optimum / transformers release? New version was released yesterday

apaniukov commented 1 month ago

Can the failures be connected with new optimum / transformers release? New version was released yesterday

It is. The xgen tokenizer is broken in a new release, so I removed it from tests.