Ailixir is an application that utilises LLMs and custom user input to generate AI agent prototypes specialised in fields such as health, economics, physics etc. The prototypes enable the user, which is an entrepreneur-developer, to compare the results produced by different LLMs.
Develop a strategy and write a workaround for the PubMed API's limitation that caps free users' results at 10K. The solution might involve segmenting API calls by publication year to avoid hitting the cap.
User Story
As a user,
I want as much of the lastest medical research information available as possible,
to populate the custom context that will imporve the responses I get from querrying the LLMs.
Acceptance Criteria
[ ] The system can handle PubMed API queries without hitting the 10K limit by segmenting searches by year or other criteria.
[ ] The implementation includes a method for dynamically adjusting query parameters to stay within the API limits.
[ ] Testing confirms that the workaround allows for complete data retrieval from PubMed without errors related to the limits.
[ ] The solution is integrated into the existing data acquisition pipeline and works correctly in tandem with the rest of the scrapers.
Definition of Done
[ ] The feature has been fully implemented.
[ ] The feature has been manually tested and works as expected without critical bugs.
[ ] The feature code is documented with clear explanations of its functionality and usage.
[ ] The feature code has been reviewed and approved by at least one team member.
[ ] The feature branches have been merged into the main branch and closed.
[ ] The feature utility, function and usage have been documented in the respective project wiki on github.
Domain
data pipeline
Description
Develop a strategy and write a workaround for the PubMed API's limitation that caps free users' results at 10K. The solution might involve segmenting API calls by publication year to avoid hitting the cap.
User Story
Acceptance Criteria
Definition of Done