search

ChemFoundationModels / ChemLLMBench

What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks
https://arxiv.org/abs/2305.18365
127 stars 6 forks source link

Question about the prompt for molecular property prediciton #5

Closed bbjy closed 9 months ago

bbjy commented 1 year ago

Hi! Thanks for your work!

I have a question about the prompt for molecular property prediction.

In the Section 4.1 (Tasks with selectively competitive (SC) performance), it said "the prompt includes inhibit HIV replication or drugs failed clinical trials for toxicity reason, and we observed a significant decline in the performance of GPT models upon removing property labels from the prompt", but the template for the property prediction in this repository seems not including such detailed information.

So could you please explain it more detailedly? Thanks a lot!

taichengguo commented 11 months ago

Hi! Thanks so much for your attention to our work!

For the code in this repository: The current code we submitted to GitHub is an initial version (with label). We did the experiments (without label) during the NeurIPS rebuttal phase, so we haven't submitted the prompt template for this case (without label) to Github.

For the Label interpretation: More details are in the Appendix of our paper on Page 18.

截屏2023-12-29 下午5 18 29

Thanks!

Data-reindeer commented 11 months ago

Hi, Taicheng Thank you for bringing this work to the community! However, a friendly reminder that some of the current prompts for Propert Prediction (PP) datasets are not matched correctly. For example, the prompts for HIV and Clintox seems to be swapped.

taichengguo commented 11 months ago

Thanks for your kind reminder! @KehanGuo2 Could you fix the prompts for the property prediction? Thanks!

KehanGuo2 commented 9 months ago

thanks for the reminder, we have fixed the issue now.