Graceful Shutdown of Scraper Script in Container Pod

[extreme4all]

As a developer, I want the scraper script to run a shutdown sequence when the container pod is killed, So that it continues processing all messages in its send and receive queue until they are empty before shutting down.

Acceptance Criteria

1. When the container pod running the scraper script is killed, the script should initiate a shutdown sequence.
2. The shutdown sequence should ensure that all messages in the send and receive queue are processed before the script shuts down.
3. No messages should be lost during the shutdown process.
4. If processing all messages is not possible, the script should go into an error sequence, sending all messages back to Kafka.

Notes

• The goal is to prevent message loss during unexpected shutdowns and maintain data integrity.
• The error sequence should be designed to handle exceptions and errors during the shutdown process.

[RusticPotatoes]

by scraper script you mean https://github.com/Bot-detector/bot-detector-scraper/blob/19dcbc20d0ec773f74d181bd4dd1fc3a7f12883b/docker-compose.yml#L68 (src\main_v2.py)?