Can PyVirtualDisplay be used in parallel threads without issues?

[oddmario]

Hello,

I am starting multiple PyVirtualDisplay Xvfb displays in a few parallel threads using the threading module. and I was wondering if this can cause any problem (like one display starts inside another)?

I'm currently doing it this way:

import threading
from selenium import webdriver
from pyvirtualdisplay import Display

def thread():
    display = Display(visible=0, size=(1024, 768), backend="xvfb")
    display.start()
    browser = webdriver.Firefox()
    browser.get('https://google.com')
    browser.quit()
    display.stop()

def main():
    # Start 15 threads
    for i in range(15):
        t = threading.Thread(target=thread)
        t.start()

if __name__ == "__main__":
    main()

Kind regards.

[ponty]

Parallel run should work both in threads and in processes. Check also "Concurrency" in README.

[Elody-07]

Parallel run should work both in threads and in processes. Check also "Concurrency" in README.

Hello ponty, I am using pyvirtualdisplay to compile videos, my code runs the code below in a loop. But it can only render one video, the second one has error like this: XIO: fatal IO error 0 (Success) on X server ":88". It seems like the second one want to use same X server port as before but failed. And after display.stop(), it seems like the port doesn't release successfully. Do you have any idea how to fix this?

display = Display(visible=0, size=(1280, 720)) display.start() time.sleep(2) compile(meta_dir, pic_dir, timestamps) display.stop()

Thank you very much!!!

[ponty]

This issue is for parallel threads. Start a new thread or continue #31 which has the same error message. I don't see what "compile" does. Maybe you have to start and stop Display only once outside the loop.

[db3000]

When starting displays in multiple threads, is it possible there will be race conditions involving the DISPLAY environment variable, as all threads share the same os.environ? Like if thread 1 starts its the display first and sets DISPLAY first then thread 2 might start inside thread 1's virtual display and set DISPLAY again, so everything running in both threads will use thread 2's display?

Is it worth making a mode that doesn't set DISPLAY at all, so its up to the user to pass this environment variable to any subprocesses (e.g. by passing it in env to subprocess.run)?

[ponty]

You are right. I will add an option for thread-safe operation which doesn't change the global os.environ.

[oddmario]

Hello again,

If anyone is still confused about this issue, I will explain another solution here (the first solution is using manage_global_env) as explained above by ponty.

This solution in my case is better since any Python code that runs (like PyAutoGUI for example) will use the DISPLAY value set by pyvirtualdisplay without causing any interference with any other threads.

Basically the solution is to use multiprocessing instead of the threading module. Processes don't share the global variables with each other unlike threads (this may be a disadvantage if you want to share any other variables globally across all of your processes - but a solution always exists... use Google to learn more about multiprocessing)

Below I have done two tests that show how is an environment variable shared when we use threading and multiprocessing

Test 1

Using threading

import threading
import time
import os

def thread1():
    print("thread one: Setting TESTENV123 value to 1")
    os.environ["TESTENV123"] = "1"
    print("thread one: Successfully set")
    try:
        print("thread one: I can see the TESTENV123 value as " + str(os.environ["TESTENV123"]))
    except Exception as e:
        print("thread one: TESTENV123 isn't even defined")

def thread2():
    try:
        print("thread two: I can see the TESTENV123 value as " + str(os.environ["TESTENV123"]))
    except Exception as e:
        print("thread two: TESTENV123 isn't even defined")

def main():
    print("Testing with the threading module...")
    time.sleep(2)

    t1 = threading.Thread(target=thread1, args=())
    t1.start()
    time.sleep(2)
    t2 = threading.Thread(target=thread2, args=())
    t2.start()

    time.sleep(10)

if __name__ == "__main__":
    main()

Explanation: First of all thread one have set the environment variable TESTENV123 to 1, then thread two was started.

• thread one could see the variable value as 1 - that's for sure cause its the thread that have set the value.
• thread two also could see the environment variable as 1

Conclusion: Using threading will share the environment variables across all the threads.

Test 2

Using multiprocessing

from multiprocessing import Process
import time
import os

def thread1():
    print("thread one: Setting TESTENV123 value to 1")
    os.environ["TESTENV123"] = "1"
    print("thread one: Successfully set")
    try:
        print("thread one: I can see the TESTENV123 value as " + str(os.environ["TESTENV123"]))
    except Exception as e:
        print("thread one: TESTENV123 isn't even defined")

def thread2():
    try:
        print("thread two: I can see the TESTENV123 value as " + str(os.environ["TESTENV123"]))
    except Exception as e:
        print("thread two: TESTENV123 isn't even defined")

def main():
    print("Testing with the multiprocessing module...")
    time.sleep(2)

    p1 = Process(target=thread1, args=())
    p1.start()
    time.sleep(2)
    p2 = Process(target=thread2, args=())
    p2.start()

    time.sleep(10)

if __name__ == "__main__":
    main()

Explanation: First of all thread one have set the environment variable TESTENV123 to 1, then thread two was started.

• thread one could see the variable value as 1 - that's for sure cause its the process that have set the value.
• thread two couldn't even recognise TESTENV123

Conclusion: Using multiprocessing will NOT share the environment variables across all the processes.

---

Sorry if I couldn't explain it in a simple manner, I'm usually not good at explaining. However you can try both tests to see it on your own.

But in brief, you can try using multiprocessing which will solve the parallelism issue and the race that happens when we use threading

[ponty]

Resolved.