guanquann / Stocksera

Finance application that provides more than 60 different alternative data to retail investors
MIT License
620 stars 103 forks source link

How to run tasks on schedule #21

Closed CodeInFilth closed 2 years ago

CodeInFilth commented 2 years ago

How do you recomend running tasks on a schedule as some of the scripts require you to call them from the root of the project directory.

Your insight here would be helpful as I can only think of hickory and cron jobs and those do not seem to work properly.

CodeInFilth commented 2 years ago

Do you have plan on sharing the way you achive this? As I still get errors when running wsb_live() and trying to cron them.

Frequency to run tasks

Script Name Functions Frequency
scheduled_tasks/reddit/stocks/scrape_trending_posts.py main() Daily (PM)
scheduled_tasks/reddit/stocks/scrape_discussion_thread.py wsb_live(), wsb_change(), get_mkt_cap() 10 Mins
scheduled_tasks/reddit/stocks/scrape_discussion_thread.py update_hourly() Hourly
scheduled_tasks/reddit/crypto/scrape_trending_posts.py main() Daily (PM)
scheduled_tasks/reddit/crypto/scrape_discussion_thread.py crypto_live(), crypto_change() 10 Mins
scheduled_tasks/reddit/crypto/scrape_discussion_thread.py update_hourly() Hourly
scheduled_tasks/reddit/get_subreddit_count.py main() Daily (MH)
scheduled_tasks/reddit/buy_trending_tickers.py main() Daily (MH)
scheduled_tasks/twitter/get_twitter_followers.py main() Daily
scheduled_tasks/twitter/scrape_trending_posts.py main() Daily (MH)
scheduled_tasks/economy/get_reverse_repo.py reverse_repo() 1.30PM
scheduled_tasks/economy/get_daily_treasury.py download_json() 4.00PM
scheduled_tasks/economy/get_inflation.py inflation() Monthly
scheduled_tasks/economy/get_retail_sales.py retail_sales() Monthly
scheduled_tasks/economy/get_initial_jobless_claims.py jobless_claims() Weekly
scheduled_tasks/economy/get_upcoming_events_date.py main() 6.00PM
scheduled_tasks/miscellaneous.py main() Daily (AH)
scheduled_tasks/get_earnings_calendar.py main() Daily (AH)
scheduled_tasks/get_failure_to_deliver.py main() 2 Weeks
scheduled_tasks/get_latest_insider_trading.py main() 2 Hours
scheduled_tasks/get_short_volume.py main() 6.00PM
scheduled_tasks/get_stocks_summary.py main() 10 Mins
scheduled_tasks/get_stocktwits_summary.py main() Hourly
scheduled_tasks/get_senate_trading.py main() Daily (AH)
scheduled_tasks/reset_options_cache.py reset_options() Daily (AH)
scheduled_tasks/reset_stocksera_trending.py reset_trending_db() 30 Mins
guanquann commented 2 years ago

The reason why I didn't share it from the start is because I'm using pythonanywhere to host/run my scheduled tasks, where the method of running a scheduled script is quite different from how we will run on our OS.

img1

Basically at the start of every hour (00 mins past), I will run wsb_live() and update_hourly(). Note that you only run update_hourly() once every hour.

For the rest of the hour (in 10mins interval each), I will run wsb_live(), wsb_change() and get_mkt_cap().

tasks_to_run.py is purely for reference. You are free to experiment how you want to run the scripts.