Scraper Crashes when sleep time is minimized.. solution that worked for me

[Deweygattringer]

I dont know If anyone Had the Problem of the scraper crashing when the sleep time is Set top a faster pace. Did it so it might have a slight time Advantage dir the announcement. The Script startet to Crash every 2h or so. So i came Up with the Idea of keeping the sleeptime to a Minimum and simply restarting it hourly via bash Script.

Create a Folder in Home called Scripts New File called restart-script.sh and Insert the following exactly

#!/bin/bash
#gives you a timestamp
echo "###########"
date

# don't change anything above this line

# important: no spaces before or after equal sign 
SCRIPT_DIRECTORY="/home/pi/gateio-crypto-trading-bot-binance-announcements-new-coins"
SCRIPT_NAME="main.py"
WAIT_SECONDS=3

# don't change anything below this line

function start_script() {
    # save old working directory - only needed for manual testing
    # but doesn't matter for cron
    OLD_WD=$PWD

    # change working directory
    cd "$SCRIPT_DIRECTORY"

    # restart script
    echo "starting $SCRIPT_NAME in the background"
    python3.9 "$SCRIPT_NAME" &

    # return to old working directory
    cd "$PWD"
}

# kill script
SCRIPT_PID=$(pgrep -f "$SCRIPT_NAME")
if [ -n "${SCRIPT_PID}" ];
then
    echo "$SCRIPT_NAME running - killing it "
    pkill -f "$SCRIPT_NAME"
else
    echo "$SCRIPT_NAME not running"
    start_script

Then open a cronjob: cronjob -e

0 * * * * /bin/bash /home/pi/scripts/restart-script.sh >> scripts/restart.log 2>1&

[nitad54448]

What was the time you used for sleep ? (it is in the sleep in main.py ?) thanks

[Deweygattringer]

No, sleep in Listing scraper. Set it to a second