cannot pyplot.savefig within pmp().map

jessexknight commented 2 years ago

MWE

from pathos.multiprocessing import ProcessingPool as pmp
import matplotlib.pyplot as plt
def plotfun(i):
  plt.plot([0,i],[0,i])
  plt.savefig('tmp-{}.pdf'.format(i))
  plt.close()
vec = [1,2,3,4,5,6,7]
# for i in vec: plotfun(i) # OK
pmp().map(plotfun,vec)

Errors:

First, this appears, and it hangs:

Gdk-Message: 20:53:06.913: main.py: Fatal IO error 11 (Resource temporarily unavailable) on X server :0.

Gdk-Message: 20:53:06.913: main.py: Fatal IO error 11 (Resource temporarily unavailable) on X server :0.

If I CTRL+C, I get:

Process ForkPoolWorker-11:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/multiprocess/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.8/dist-packages/multiprocess/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/dist-packages/multiprocess/pool.py", line 114, in worker
    task = get()
  File "/usr/local/lib/python3.8/dist-packages/multiprocess/queues.py", line 358, in get
    with self._rlock:
  File "/usr/local/lib/python3.8/dist-packages/multiprocess/synchronize.py", line 101, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt

One time, randomly it seems, I also got this error message appended to the first error:

This probably reflects a bug in the program.
The error was 'BadGC (invalid GC parameter)'.
  (Details: serial 504 error_code 13 request_code 60 (core protocol) minor_code 0)
  (Note to programmers: normally, X errors are reported asynchronously;
   that is, you will receive the error a while after causing it.
   To debug your program, run it with the GDK_SYNCHRONIZE environment
   variable to change this behavior. You can then get a meaningful
   backtrace from your debugger if you break on the gdk_x_error() function.)

Additional Info:

OS: Linux Mint 20.1 DE: X-Cinnamon / Muffin

Thanks in advance!

PS: #153 may be related (and also this matplotlib issue), although those seem related to fonts and the fix didn't help, nor did plt.close(), evidently above...

jessexknight commented 2 years ago

Workaround

Okay, I have a workaround from here: if you import matplotlib.pyplot as plt within the plotting function, it works:

from pathos.multiprocessing import ProcessingPool as pmp
def plotfun(i):
  import matplotlib.pyplot as plt
  plt.plot([0,i],[0,i])
  plt.savefig('tmp-{}.pdf'.format(i))
  plt.close()
vec = [1,2,3]
# for i in vec: plotfun(i) # OK
pmp().map(plotfun,vec)

Convenience

Importing is kinda ugly, and if you have multiple modular plot functions, maybe importing plt multiple times breaks things? Anyways, with help from here you can decorate the plot function(s) to import plt only once as needed:

def iplt(fun):
  def decorator(*args,**kwds):
    if 'plt' not in fun.__globals__:
      import matplotlib.pyplot as plt
      fun.__globals__['plt'] = plt
    result = fun(*args,**kwds)
    return result
  return decorator

Usage:

@iplt
def plotfun(i):
   ...

mmckerns commented 2 years ago

It's good practice to make sure that any function that you plan to run in parallel has full encapsulation (i.e. all variables used are defined inside the function). So, imports should go inside the function definition. There is no harm in having an import executed twice (unless you are importing *).

Let me know what you are looking to get answered in this issue, or if you have resolved it yourself.

jessexknight commented 2 years ago

Thanks -- I think you can close the issue, as the local import workaround is very reasonable. Plus, from this and this matplotlib issues, it seems the root cause may be upstream related to different backends (as shown below)...

Another Possible Workaround: Change Backends

If I run the MWE with:

import matplotlib
matplotlib.use('<<backend>>')

with <<backend>> as any of: agg, cairo, pdf, pgf, ps, svg, template , the MWE works. I didn't test all backends as I didn't want to install all packages, and frankly I'm not clear what the differences are, but leaving this here in case it's helpful...

mmckerns commented 2 years ago

Nice follow-up. Thanks for posting.

uqfoundation / pathos