search

mandarjoshi90 / coref

BERT for Coreference Resolution
Apache License 2.0
440 stars 92 forks source link

How to evaluate SpanBERT using sample test data? #110

Open brendanc2122 opened 1 year ago

brendanc2122 commented 1 year ago

I do not really understand the instructions to be very frank. I am using the Google Colab file, have already been trying to break the code in the colab file for a month. Figured out most of the file except on how to evaluate the model using sample test data? Also, is there a way to train more data on Google Colab? I have only been able to train max. 300 rows of a dataset.

It's just errors after errors which I have no idea at all what they mean or why they occur... I am quite frustrated at this point since I have not been able to get any sort of help anywhere else.... @mandarjoshi90 @jkkummerfeld

I am not really well versed in this area of Coref models aI am a student who was tasked to conduct Coref Resolution.I do not have much experience in understanding GitHub files and the code is quite complex at my current level. Its really tough and I hope you guys could shed some light. Thank you!!!

jkkummerfeld commented 1 year ago

The notebook does not have evaluation code. To add that you will need to look at the contents of this repository and make suitable additions.

On running on more data, I used the colab to run small experiments and am not sure how scaling would go. The problems you are hitting could be due to the limited resources provided for free with Colab. You could take the code and run it on your own server instead, which may also provide more information for debugging.

Unfortunately, I do not have time to help you with these changes, but hopefully the rest of the team / your advisor who tasked you with this can help. Good luck!

brendanc2122 commented 1 year ago

The notebook does not have evaluation code. To add that you will need to look at the contents of this repository and make suitable additions.

On running on more data, I used the colab to run small experiments and am not sure how scaling would go. The problems you are hitting could be due to the limited resources provided for free with Colab. You could take the code and run it on your own server instead, which may also provide more information for debugging.

Unfortunately, I do not have time to help you with these changes, but hopefully the rest of the team / your advisor who tasked you with this can help. Good luck!

@jkkummerfeld Hi there, I appreciate your reply, can I get some insight on why there the metrics in the files but unable to produce evaluation results?

Is it possible to run on windows? As the front part of the notebook contains some commands that are not available on windows.

Thank you!

jkkummerfeld commented 1 year ago

Evaluation - The code I wrote does not call the functions that do evaluation. I didn't look at the evaluation set up at all, so I am not sure how it works in this codebase.

Windows - I haven't used Windows for programming in over a decade, so I'm not much help there. Those first few special commands are installing packages. Maybe you can use a virtual machine of some sort? Or ssh into a server that runs linux?