search

zhu-minjun / PAlign

Personality Alignment of Language Models
18 stars 2 forks source link

about file directory IPIP and file selected_IPIP300_samples.json #1

Closed SHIFTTTTTTTT closed 2 months ago

SHIFTTTTTTTT commented 2 months ago

hello,I want to ask : 1.does the directory 'IPIP' appeared in the code equal to the directory 'PAPI'? (cause I didn't found directory"IPIP" in your published code)
2.And does "selected_IPIP300_samples.json" appeared in the code equal to "Test-set.json"?(cause I didn't found the former too.) 3.I read the code file which generates "selected_IPIP300_samples.json",and it shows that only 300 piece of data were used for the experiment .Do I understand it correctly? Thank you for answering！

zhu-minjun commented 2 months ago

Thank you very much for your interest in our work. Regarding the first two questions, you are correct, and we apologize for not promptly correcting these errors when releasing the code. We have now fixed this issue and will load the correct data. Concerning the 300 questionnaires, when training the probe, we actually used questions from one person.

For the PAPI dataset, Test-set.json contains questionnaire sets for 300 subjects (people), and each subject needs to answer 300 personality-related questions (total number of questions is 300*300). Our experiment is conducted for each subject individually. From each subject's 300 questions, 120 are used for training the probe, and the remaining 180 are used for testing.

SHIFTTTTTTTT commented 2 months ago

Thank you! During the experiment,I also encountered another problem: WechatIMG646 I 've configured the new environment with “pip install .”,using the latest version of transformers.But it still didn't work.Do you know how to fix it ?

SHIFTTTTTTTT commented 2 months ago

Thank you very much for your interest in our work. Regarding the first two questions, you are correct, and we apologize for not promptly correcting these errors when releasing the code. We have now fixed this issue and will load the correct data. Concerning the 300 questionnaires, when training the probe, we actually used questions from one person.

For the PAPI dataset, Test-set.json contains questionnaire sets for 300 subjects (people), and each subject needs to answer 300 personality-related questions (total number of questions is 300*300). Our experiment is conducted for each subject individually. From each subject's 300 questions, 120 are used for training the probe, and the remaining 180 are used for testing.

Thank you! During the experiment,I also encountered another problem⬆️.I 've configured the new environment with “pip install .”,using the latest version of transformers.But it still didn't work.Do you know how to fix it ?

zhu-minjun commented 2 months ago

Can you try the laest code? We have update the model files!

SHIFTTTTTTTT commented 2 months ago

Can you try the laest code? We have update the model files!

still the same...

zhu-minjun commented 2 months ago

Hello! We have reproduced your error and resolved it. The reason for this error was that the previous version used hooks to obtain activations, but when adapting to transformers, it would replace the original attention blocks. We have updated the code and solved this issue!

SHIFTTTTTTTT commented 2 months ago

Hello! We have reproduced your error and resolved it. The reason for this error was that the previous version used hooks to obtain activations, but when adapting to transformers, it would replace the original attention blocks. We have updated the code and solved this issue!

Problem solved.Thank you! But I noticed that the main.py is not for Open-Ended Generation. Could you please update the code in which I can try to test PAS in open ended generation? Thank you very much !