Current Queue objects are not process safe

[MayoG]

Currently we are using the queue module from here which is thread safe but not process safe. The bug occurs when we are changing one of the routines to execute as a process and its queue cannot get the information from the other routines.

We need to change this queue module and refactor all of its uses to this one which is thread and process safe and have almost the same interface as the first module so it shouldn't cause alot of changes.

[OmriKaduri]

Hi @MayoG ,

Can you specify exactly what queues do you think should be multiprocessing queues and what queues should be "regular" thread-safe queues? Let's say we're talking about VideoCapture, why should we change its queue to a multiprocessing queue, if its routines are threads?

[MayoG]

@OmriKaduri

I want to add the option for users to choose whether they want to run their routines as threads or processes. When we create a new queue for a component, we won't be able to change one of the component routines to run as a process.

[OmriKaduri]

At which cases would we like to run routines as processes and not as threads?

[MayoG]

@OmriKaduri The requirement is that component and routines can run as threads or processes (Component default is process, Routine default is thread). Currently i am not familiar with a case but @ItamarWilf might know one.

[itamarwilf]

In general, a process should be used when your computation is bound by CPU and not I/O. If you have a compute heavy logic that you want to run, it could be done with a process. Now there's a catch, if the component is already a process, it isn't needed every time. In my opinion, the best use for a process routine is if you want to run multiple routines like a "multiprocessing pool". Also, remember that using a thread queue is more efficient then a multiprocessing queue. Same with the context switch, so every developer needs to decide which is best for his use case, and probably try both use profiling to decide.