Feature/article checker workers with queue

Kseymur commented 5 months ago

This pull request resolves https://github.com/1712n/dn-institute/issues/425.

When attempting to create a worker that responds to GitHub webhooks and starts processing the diff, I encountered the following limitations:

GitHub expects a response to the webhook within 10 seconds. This caused the worker to crash as it couldn't complete the processing in time.
Using waitUntil, which could send a response to GitHub immediately and continue processing in the background, is also not feasible in our case due to time constraints. Waiting for a response from OpenAI might take longer.

Considering these limitations, I decided to use a worker with Durable Objects and Alarms. This allows the worker to respond to GitHub immediately and then proceed with processing the diff in the background.

Improvements that can be made:

Add full article parsing by links in the RAG part.
Add a worker for plagiarism detection.

evgenydmitriev commented 5 months ago

GitHub expects a response to the webhook within 10 seconds. This caused the worker to crash as it couldn't complete the processing in time.

Why do you need to maintain your connection with GitHub? The webhook is only needed to trigger the worker. What you do after (and for how long) is none of GitHub's business. Get whatever you need from GitHub, drop the connection, do your thing, and establish a new connection to post the PR comment whenever you are ready.

Kseymur commented 4 months ago

@evgenydmitriev Rewrote it into a system with two workers and a queue.

1712n / dn-institute

Feature/article checker workers with queue #447