search

taesiri / ArXivQA

WIP - Automated Question Answering for ArXiv Papers with Large Language Models (https://arxiv.taesiri.xyz/)
https://arxiv.taesiri.xyz/
286 stars 11 forks source link

Removing faulty answers. #3

Open taesiri opened 12 months ago

taesiri commented 12 months ago

It seems that either the Latex flattening code or the Claude-2.0 API (which one is unclear) is doing something strange, and some answers are generated without the body of the paper. For instance, for this question, the answer clearly states that it does not have access to the paper material. We should investigate this and find a filtering process to remove bad answers.

Without having access to the full paper, it's difficult to provide an accurate 1-sentence summary. However, academic papers often have an abstract at the beginning that summarizes the key points and contributions. The abstract would be a good starting point to understand the main idea of the paper in a concise way. If a 1-sentence summary is still needed, it would require looking at the introduction and conclusion sections to identify the overarching theme and outcomes of the research. Having access to key sections like these would help generate a very brief summary statement.

taesiri commented 12 months ago

A simple text search shows that following papers contain similar phrases:

# total of 43

['./papers/2301.13616.md',
 './papers/2308.16463.md',
 './papers/2212.08073.md',
 './papers/2309.0791.md',
 './papers/2107.13586.md',
 './papers/2305.19835.md',
 './papers/2306.17194.md',
 './papers/2306.04031.md',
 './papers/2304.04746.md',
 './papers/2305.15581.md',
 './papers/2310.03714.md',
 './papers/2305.07185.md',
 './papers/2207.05739.md',
 './papers/2302.04023.md',
 './papers/2303.17651.md',
 './papers/2307.01848.md',
 './papers/2303.04673.md',
 './papers/2307.04577.md',
 './papers/2307.10350.md',
 './papers/2306.17582.md',
 './papers/2308.11551.md',
 './papers/2305.09515.md',
 './papers/2205.11916.md',
 './papers/2309.08637.md',
 './papers/2309.15129.md',
 './papers/2305.10855.md',
 './papers/2309.03409.md',
 './papers/2304.10970.md',
 './papers/2201.07207.md',
 './papers/2305.14540.md',
 './papers/2308.01313.md',
 './papers/1803.11203.md',
 './papers/2309.16588.md',
 './papers/2308.13954.md',
 './papers/2306.09539.md',
 './papers/2309.16235.md',
 './papers/2305.16960.md',
 './papers/2307.16715.md',
 './papers/2306.09557.md',
 './papers/2306.05425.md',
 './papers/2107.03374.md',
 './papers/2210.09261.md',
 './papers/2304.13169.md']
taesiri commented 11 months ago

The above papers have been manually removed and re-submitted to the QA queue. Now, we have only four bad responses.

['./papers/2307.01848.md',
 './papers/2205.11916.md',
 './papers/2309.08637.md',
 './papers/2305.10855.md']