search

lupantech / chameleon-llm

Codes for "Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models".
https://chameleon-llm.github.io
Apache License 2.0
1.08k stars 88 forks source link

Questions on TabWMP #10

Open Jungle728 opened 6 months ago

Jungle728 commented 6 months ago

Thank you for your work. When running the TabWMP dataset, I found that some of the examples are executing very slowly, is there any way you can speed them up? It only takes 4-5 hours to complete on the Science QA dataset, but it can take 20 times longer on the TabWMP dataset. Looking forward to hearing from you, thanks!

lupantech commented 6 months ago

Thank you for your feedback. Generally, the duration of downstream tasks primarily depends on the number of test examples. In our experiments, the processing times for Science and TabWMP were relatively similar.

Could you please provide more details, such as the running log or the code snippet that leads to the time bottleneck, to better identify the cause? Thank you!