search

axinc-ai / ailia-models

The collection of pre-trained, state-of-the-art AI models for ailia SDK
2.04k stars 325 forks source link

Implement japanese-reranker-cross-encoder-large-v1 #1486

Closed ooe1123 closed 3 months ago

ooe1123 commented 5 months ago

https://github.com/axinc-ai/ailia-models/issues/1443 のPRです

kyakuno commented 3 months ago

モデルアップロード済み。 https://storage.googleapis.com/ailia-models/japanese-reranker-cross-encoder/japanese-reranker-cross-encoder-large-v1.onnx

kyakuno commented 3 months ago

BERT Japaneseを使っている。

ailia tokenizer

 INFO japanese-reranker-cross-encoder.py (98) : Start inference...
[[    2 22695  4037   460 12632   461 12584   456     3 15729 13383   500
  12968 12735   484   384  5553  7056   680   464  2149   500  2547  7263
   7383  7056 26752   385 12593 12784   464 32329 15313   430  4261  9255
    457   384 17163   465  3358 14176   457   465  5538 12504 12494   385
      3]
 [    2 22695  4037   460 12632   461 12584   456     3 13229   460 16238
   2183   465 13232 12591   430   384 23749  5613   430 14255   464   457
  23646   430 31140   457 13300   449   385 13782 15066 19730 14253   430
  14395   466 29955   449   385     3     0     0     0     0     0     0
      0]
 [    2 22695  4037   460 12632   461 12584   456     3 13500   461   484
  14637 13753   461 32429 13346   430  3235   461 12493   449   385 14924
   3388  7213   464 12485 12932 13046   430  5538 20925   449   385     3
      0     0     0     0     0     0     0     0     0     0     0     0
      0]
 [    2 22695  4037   460 12632   461 12584   456     3 15504 13950   430
  15248 18790   385  5538   456   422   456  6643  7137 12494   385 14391
    465 22527   450   430   384 12546   430  6007   461 14459   385     3
      0     0     0     0     0     0     0     0     0     0     0     0
      0]]
[[0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
  1 1 1 1 1 1 1 1 1 1 1 1 1]
 [0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
  1 1 1 1 1 1 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
  0 0 0 0 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
  0 0 0 0 0 0 0 0 0 0 0 0 0]]

transformers

[[    2 22695  4037   460 12632   461 12584   456     3 15729 13383   500
  12968 12735   484   384  5553  7056   680   464  2149   500  2547  7263
   7383  7056 26752   385 12593 12784   464 32329 15313   430  4261  9255
    457   384 17163   465  3358 14176   457   465  5538 12504 12494   385
      3]
 [    2 22695  4037   460 12632   461 12584   456     3 13229   460 16238
   2183   465 13232 12591   430   384 23749  5613   430 14255   464   457
  23646   430 31140   457 13300   449   385 13782 15066 19730 14253   430
  14395   466 29955   449   385     3     0     0     0     0     0     0
      0]
 [    2 22695  4037   460 12632   461 12584   456     3 13500   461   484
  14637 13753   461 32429 13346   430  3235   461 12493   449   385 14924
   3388  7213   464 12485 12932 13046   430  5538 20925   449   385     3
      0     0     0     0     0     0     0     0     0     0     0     0
      0]
 [    2 22695  4037   460 12632   461 12584   456     3 15504 13950   430
  15248 18790   385  5538   456   422   456  6643  7137 12494   385 14391
    465 22527   450   430   384 12546   430  6007   461 14459   385     3
      0     0     0     0     0     0     0     0     0     0     0     0
      0]]
[[0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
  1 1 1 1 1 1 1 1 1 1 1 1 1]
 [0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
  1 1 1 1 1 1 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
  0 0 0 0 0 0 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
  0 0 0 0 0 0 0 0 0 0 0 0 0]]