The idea behind using a multithreaded queue data structure is such that the whole pipelined process(from scraping to newsletter generation) can be treated as a transaction and if the process fails at any step due to any issue we can push it into a queue shared over the running processes.
This would give me a running solution that I could use as a placeholder and move forward with the project. I know an optimal solution may be more complex or even require writing a custom multithreaded priority queue, but I need a working solution at the moment.
PS:
This is the first time I working with concurrency control in python, and I don't have much prior knowledge of the same, If you have another solution, suggestion or idea, please please do comment.
The entire structure of the project is as follows:
I am facing a critical issue with implementing the multithreaded queue data structure which should be connected with the Manganelo Scraper, Page Scraper, Sentiment Analysis, and Summary Generator and final Newsletter Generation files.
The idea behind using a multithreaded queue data structure is such that the whole pipelined process(from scraping to newsletter generation) can be treated as a transaction and if the process fails at any step due to any issue we can push it into a queue shared over the running processes.
This would give me a running solution that I could use as a placeholder and move forward with the project. I know an optimal solution may be more complex or even require writing a custom multithreaded priority queue, but I need a working solution at the moment.
PS: This is the first time I working with concurrency control in python, and I don't have much prior knowledge of the same, If you have another solution, suggestion or idea, please please do comment.