AutoSurveys / AutoSurvey

205 stars 12 forks source link

Web Demo and Cloned Code Produce Different Outputs #18

Closed Fuyujia799 closed 1 week ago

Fuyujia799 commented 1 week ago

Description: I have noticed a recurring discrepancy between the results produced by the GitHub-hosted web demo and the results generated from the cloned code. Specifically:

1. Web Demo: When I use the web demo, the generated markdown files have a well-structured and organized format. 2. Cloned Code: However, when I clone the repository and run the code locally, the markdown files often have issues such as duplicate headings and formatting inconsistencies. This is not an isolated case, as most runs of the cloned code exhibit these structural issues. I suspect there might be some post-processing steps or adjustments that are missing in the cloned code. Could this be the cause of the formatting discrepancies?

To help clarify, I have attached two images for comparison:

The output from the web demo. The output from the cloned code.

Questions: Is the code used for the web demo the same as the code available in the repository? What model and parameters are being used in the web demo? Are there any differences in configuration that I should be aware of when running the code locally? Thank you for your attention to this issue. I look forward to your response.

Attachments: [Web Demo Output Screenshot] img_v3_02fj_f982584f-e2e3-4a73-a0cd-e86570d6017g

[Cloned Code Output Screenshot] img_v3_02fj_8b035788-18f2-40e9-9d4c-f19be10b25eg

image
GuoQi2000 commented 1 week ago

Answer for your questions: Q: Is the code used for the web demo the same as the code available in the repository? A: Yes, the code is the same. Q: What model and parameters are being used in the web demo? Are there any differences in configuration that I should be aware of when running the code locally? A: Claude-haiku is used in the web demo. Try to replace the model and keep other configurations as default.

Actually, we have observed that Claude outperforms other models in terms of formatting. We strongly suggest using Claude in practice.

Fuyujia799 commented 1 week ago

Answer for your questions: Q: Is the code used for the web demo the same as the code available in the repository? A: Yes, the code is the same. Q: What model and parameters are being used in the web demo? Are there any differences in configuration that I should be aware of when running the code locally? A: Claude-haiku is used in the web demo. Try to replace the model and keep other configurations as default.

Actually, we have observed that Claude outperforms other models in terms of formatting. We strongly suggest using Claude in practice.

Thank you for your answer! Claude-haiku really shows better structure!So are the datasets used in the web demo also built from titles and abstracts? Or should we use the first 1500 tokens mentioned in your paper? Also, how can I obtain a complete dataset? I'm sorry I sent you an email but didn't receive a reply.

GuoQi2000 commented 1 week ago

Yes, the web demo only uses titles and abstracts to save API costs. We are sorry for not noticing your email earlier. Could you please send it again? We will share a link for you to download the dataset.

Fuyujia799 commented 1 week ago

Yes, the web demo only uses titles and abstracts to save API costs. We are sorry for not noticing your email earlier. Could you please send it again? We will share a link for you to download the dataset.

Thanks a lot!!!! I have received the full dataset.