search

Leolty / tablellm

https://arxiv.org/abs/2312.16702
Apache License 2.0
14 stars 0 forks source link

issues about code in agent #2

Open Yang-xiao-zhen opened 2 months ago

Yang-xiao-zhen commented 2 months ago

Hello, in your run_agent.py code, line 60 "with open("output/wtq_agent_wo_norm/result.jsonl", "r") as f:", I didn't see any information about the file. I want to know where the file from, the result of the run_cot? or other result? Thanks

Yang-xiao-zhen commented 2 months ago

Also, I don't know how to implement pyagent+cot combination. Do you use both of them in eval_wtq? Be similar to checkpoints=["tablellm/output/wtq_dp/result.jsonl", "tablellm/output/wtq_agent/result.jsonl"] n_times=1 )

Leolty commented 2 months ago

Hello Zhen,

Thank you for your patience. Let me address your questions:

  1. Regarding the line you referenced in run_agent.py: As explained in our paper, we implement a norm process to convert tables into a format preferred by the LLM. However, this process often leaves some tables unchanged. To optimize API usage and reduce costs, we reuse results when possible. If you don't need this, you can directly comment out the related code. Sorry I should have made this step optional. Ideally, there should be a check to see if the file exists.

  2. For evaluation code and using the agent without running on specific datasets: We have provided several example notebooks in our repository. You can find them here: https://github.com/Leolty/tablellm/tree/main/examples. And just run them separately and aggregate any files using the eval_wtq.

These notebooks demonstrate various ways to use the agent and should help you get started.

Please let me know if you need any further clarification or assistance.