search

DaShenZi721 / HRA

14 stars 2 forks source link

NLP Task Reproduction Unsuccessful #4

Open LQF39466 opened 1 day ago

LQF39466 commented 1 day ago

I followed the steps in your README to reproduce results on SST2 dataset. While installing dependencies pip could not find the 0.1.13 version of DeBERTa. I cloned the official DeBERTa repository and packed it using wheels by myself.

I was able to run the sst2.sh script successfully, yet the experimental results had very poor accuarcy during evaluation. It would seem your experiment script used my compiled version of DeBERTa's run.py instead of yours in the HRA repository, hence adapters didn't seem to be loaded. (final output as attached below) eval_results_dev_deberta-v3-base.txt

I would like to know is there anything you haven't mentioned in your README that could cause this. Thank you.

DaShenZi721 commented 7 hours ago

Thanks for your appreciation of our work! For your first question, did you run the command python setup.py install? After any modification to the codes, you need to run this command. For the second question, you can check this scripts disentangled_attention.py and bert.py where we inject HRA adapters.