zeroone2numeral2 / reddit-test

3 stars 1 forks source link

Run subreddits jobs in their own thread #128

Closed RememberTheAir closed 4 years ago

RememberTheAir commented 4 years ago

Subreddits jobs should run in their own threads.

The main job should just collect the subreddits to check, decide whether it is time to post, and if yes, a new thread is spawned that will fetch the submissions from reddit.

In this way non-network work is done in the same thread (ptb's jobqueue one). As soon as we decide we need to fetch the submissions, a new thread for that subreddit is spawned that will run asynchronously.

What to decide: should we use a queue of jobs, or simply spawn a new one when needed? How to decide how many jobs to run at the same time at max? Should we allocate some threads for that?