Race condition multiprocessing

[ubmarco]

When running the Sphinx write phase in parallel on Linux, I'm gettig this

sphinx.errors.SphinxParallelError:
  FileNotFoundError: 
    [Errno 2] No such file or directory:
      '/tmp/tmppcibdr7c/doc_needflow/_build/html/_plantuml/0d/0db02546484441d6314299f668aecf48fff9d4ca.png.new' -> '/tmp/tmppcibdr7c/doc_needflow/_build/html/_plantuml/0d/0db02546484441d6314299f668aecf48fff9d4ca.png'

After some investigation I found the source problem:

• 2 RSTs have the same plantuml diagram
• both RSTs are written in a separate process in parallel
• both plantuml visitors are running in parallel
• the 2 def render() functions generates a unique key in key = hash_plantuml_node(node)
• the 2 keys are identical as both diagrams have identical sources
• one process is first calling rename(outfname + '.new', outfname) while the other process is still generating
• the file handle gets stale after moving the file
• the second rename leads to the FileNotFoundError

I see 2 solutions:

• Make the PID (os.getpid()) part of the hashing. That way the diagrams will still be unique when running in serial and when running in parallel, all processes generate own diagrams in case they have the same content.
• Introduce a semaphore (e.g. write a file named after the hash before starting puml) that detects whether another process already started creating a diagram with the same hash and wait until that is finished and use it. This might be tricky in case the output does not get generated. The solution would need a timeout.

[yuja]

Good catch. I think the .new file should be created with a random name (by using mkstemp(prefix=outfname, dir=..) for example.) Since the output content is identical, we won't need locking nor randomness in the final output.

[ubmarco]

Awesome. Thanks for the quick fix, it works!