Guidance on Integrating New Benchmarks with Chain-of-experts

xzymustbexzy / Chain-of-Experts

Official implementation of the paper "Chain-of-Experts: When LLMs Meet Complex Operation Research Problems"

41 stars 6 forks source link

Guidance on Integrating New Benchmarks with Chain-of-experts #5

Open tangzhy opened 2 months ago

tangzhy commented 2 months ago

Hello,

I am currently assessing the performance of Chain-of-experts using new benchmarks, specifically MAMO and IndustryOR. I observed that for existing benchmarks like NL4OPT and ComplexOR, you have implemented few-shot examples.

Is the creation of few-shot examples mandatory for integrating any new benchmarks? If so, could you provide guidance or suggest any modifications necessary to adapt your pipeline to accommodate these new benchmarks?

Thank you for your support!

xzymustbexzy commented 1 month ago

Thank you for your question

Few-shot examples are not mandatory for integrating new benchmarks. I'm not familiar with MAMO, but IndustryOR appears to have a similar structure to NL4Opt, so you can try directly.