search

eqcorrscan / EQcorrscan

Earthquake detection and analysis in Python.
https://eqcorrscan.readthedocs.io/en/latest/
Other
166 stars 86 forks source link

Strange behaviour of pre_processing.dayproc in Jupyter notebook #375

Closed shicks-seismo closed 4 years ago

shicks-seismo commented 4 years ago

Describe the bug Hi, I am running an eqcorrscan workflow in a Jupyter Notebook. I am running the different parts of the workflow in different cells - e.g. importing the modules at the start, then defining variables, building the templates, running the matched filter detection, etc. When I run pre_processing.dayproc the first time after opening the notebook, it runs fine. But if I try to re-run it, I get many lines of the following error printed in the terminal:

libc++abi.dylib: terminating with uncaught exception of type std::runtime_error: Couldn't close file

and the process doesn't appear to finish.

If I restart the kernel and try everything again, it works fine, but then fails again on the second time.

So it seems like some file is not being closed properly in pre_processing.dayproc which requires a full restart of the kernel to run. This would be fine in a script, but not in a Jupyter Notebook.

I wonder if you are able to reproduce something similar? I'm happy to share my Notebook if that's easier.

Desktop (please complete the following information):

calum-chamberlain commented 4 years ago

I'm fairly sure no files are opened in this process, so it isn't immediately obvious why this is happening. Sharing the notebook would be handy.

I'm in the field at the moment so won't get onto this for at least a couple of weeks.

shicks-seismo commented 4 years ago

Thanks, Calum. In the process of making a trimmed-down version of the notebook and sample dataset for you, it now seems to be fine. This time around, I saved the template streams as a pickle file in a separate cell before then subsequently reloading the templates in the next cell and that seems to be fine. Not really sure what the underlying cause was, but will close this now and this can be a reference for anyone else that might encounter a similar problem.

shicks-seismo commented 4 years ago

Just digging this issue back up but not reopening it for now ... I got a similar error for a bit of code that I just created using multiprocessing.pool (I assume this is what Eqcorrscan uses ... I haven't checked). After a bit of digging, the issue seems to relate to Mac OSX platforms running the Catalina OS and Matplotlib. See https://github.com/matplotlib/matplotlib/issues/15410

shicks-seismo commented 4 years ago

A few more details ... this wasn't just happening in Jupyter Notebooks - it also happened in the terminal. The symptoms I saw are very consistent with those reported in the above mpl issue. One solution that seems to work for me is to import matplotlib AFTER the multiprocessing.pool calls.