search

yzhangcs / parser

:rocket: State-of-the-art parsers for natural language.
https://parser.yzhang.site/
MIT License
825 stars 138 forks source link

low performance of "dep-biaffine-xlmr"? #124

Closed cyr19 closed 1 year ago

cyr19 commented 1 year ago

hi,

thanks for your great work!

I tried to use the xlmr model for dependency parsing on German texts, but the results didn't make sense at all.

text = "Es ist ein einfacher Satz."

1 Es 6 nmod 2 ist 6 flat 3 ein 6 flat 4 einfacher 6 nmod 5 Satz 6 appos 6 . 0 root

So I further evaluated it on the UD German treebank, which also only led to unreasonable results.

loss: 9.8960 - UCM: 1.33% LCM: 0.61% UAS: 4.18% LAS: 0.58%

The parsing results on English sent were also like this:

text = "It is a simple sentence."

1 It 6 appos 2 is 6 appos 3 a 6 appos 4 simple 6 nmod 5 sentence 6 appos 6 . 0 root

So I wonder if it would be possible that this checkpoint is not the one you evaluated on the UD treebanks? Could you maybe run it on the above examples if your time permits?

Thanks a lot!

Best, YC

yzhangcs commented 1 year ago

@cyr19 Hi, thank you for reporting this. I'm not sure what exactly happened. Possible causes are transformers version/weight mismatch. I'll retrain the model and upload it recently, please stay tuned!

yzhangcs commented 1 year ago

@cyr19 the file has been updated, please check it out :)

cyr19 commented 1 year ago

Hi, thanks for your quick response! I was off github.

The whole lib doesn't work now as i switched to another environment. I'm not sure where it results: the current error log is "ModuleNotFoundError: No module named 'supar.config'". Could you help me with this? image

Thanks a lot :)

Btw. i'm using the version from pip installation (version 1.1.4)

yzhangcs commented 1 year ago

@cyr19 The config has been moved to other places, you may need to install supar from source code.