FFTW crashes on RedHat when fftw1n is called

[gabrielebonanno]

Hi,

FFTW fails in case fftw1n is called. Not sure if the issue is RedHat vs. Ubuntu, or if it is a general Unix problem.

in PyFIArray_WrappedFFTW.cpp in void fft1n

it aborts when calling: fftwf_execute (planF);

For now, we will use dimensions node to have the dimensions to be FFTed in the last dimensions, which works.

Let us know what to try if you have an idea how to fix it. Gabri and Mike

[nckz]

Hi Gabri,

When you run the FFTW node in a simple network (shown in the image below) it will run fft1n() -does this crash the FFTW node? If not, you'll have to give me more information about how you're calling fft1n() when it aborts.

Nick

[gabrielebonanno]

Hi Nick,

Yes I can reproduce the error with a simple network like the one you show on RedHat. It doesn't happen on Mike's Mac.

We are wondering if installing the fftw libraries in /usr/include for bart is causing a problem.

Thanks, Gabri

[nckz]

I doubt thats the issue, we are running this on Ubuntu with fftw installed in the usual /usr/lib (and /usr/include) and don't have problems with these interfering. Are you using the stock "core" library or have you recompiled the "core" after installing fftw? -so fft1n() worked before the extra fftw install?

[gabrielebonanno]

the error occurs independent of using the newly compiled /gpi/core/math/fftw version or the "stock core" version of fftw in gpi_stack that we did not compile. So yes, we agree that the other fftw library in usr/include has nothing to do with it.

anything else we should try? reinstall GPI maybe?

[aganders3]

Hi Gabri,

Did it ever work? If not, I don't think reinstalling GPI will help. It seems likely that the FFTW library we provide through Anaconda (built on Ubuntu) is somehow not compatible.

I am making a Fedora VM right now to try to replicate this. Can you post some more information about your system? e.g. your OS version, gcc version, any output from the GPI terminal when it crashes...

[aganders3]

OK, I have GPI running in a Fedora machine and I can replicate the bug. I will try a couple things to fix it and report back here (likely tomorrow).

[gabrielebonanno]

Hi Ashley,

I am playing with GPI on a local installation of RedHat EL 7.2 (workstation) for the first time, before was using the Ubuntu VM. We actually discovered the bug while trying a bart example network.

I can see that the fftw provided with anaconda is the latest (3.3.4), while the one I have installed in /usr from yum repositories to work with BART is version 3.3.3. The gcc compiler is version 4.8.5 (everything for arch x86_64). Please let me know if you need more information.

GPI does not give any error message. Though I have few warnings when I make /core/math/fft (see below). Which may be related to the bug since it happens in void fftn1() at the call _fftwfexecute (planF);

Thanks for replicating the issue.

Gabri

---

Tue Mar 1 19:31:08 2016 - gpi.config:415 - ---- - /home/gbonanno/.gpirc has been loaded.

3.5.0 |Continuum Analytics, Inc.| (default, Oct 19 2015, 21:57:25) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]

Adding GPI include dirs GPI_LIBRARY_PATH '/home/gbonanno/gpi' for lib 'bart' GPI_LIBRARY_PATH '/home/gbonanno/gpi' for lib 'core' GPI_LIBRARY_PATH '/home/gbonanno/gpi' for lib 'bni' Adding Anaconda lib and inc dirs... Adding POSIX-Threads lib Making target: fft running build_ext building 'fft' extension gcc -pthread -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/include -I/home/gbonanno/gpi_stack/lib/python3.5/site-packages/numpy/core/include -I/home/gbonanno/gpi_stack/include -I/home/gbonanno/gpi -I/home/gbonanno/gpi_stack/include/python3.5m -c fft_PyMOD.cpp -o build/temp.linux-x86_64-3.5/fft_PyMOD.o -DMOD_NAME=fft cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default] In file included from /home/gbonanno/gpi_stack/include/PyFI/PyFI.h:87:0, from fft_PyMOD.cpp:49: /home/gbonanno/gpi_stack/include/PyFI/PyFIArray_WrappedFFTW.cpp: In function ‘void PyFI::FFTW::fft1n(PyFI::Array&, PyFI::Array&, int, uint64_t) [with T = std::complex; uint64_t = long unsigned int]’: /home/gbonanno/gpi_stack/include/PyFI/PyFIArray_WrappedFFTW.cpp:942:6: warning: ‘planD’ may be used uninitialized in this function [-Wmaybe-uninitialized] fftw_destroy_plan (planD); ^ /home/gbonanno/gpi_stack/include/PyFI/PyFIArray_WrappedFFTW.cpp:945:6: warning: ‘planF’ may be used uninitialized in this function [-Wmaybe-uninitialized] fftwf_destroy_plan (planF); ^ g++ -pthread -shared -L/home/gbonanno/gpi_stack/lib -Wl,-rpath=/home/gbonanno/gpi_stack/lib,--no-as-needed build/temp.linux-x86_64-3.5/fft_PyMOD.o -L/home/gbonanno/gpi_stack/lib -L/usr/lib -L/home/gbonanno/gpi_stack/lib -lpthread -lfftw3f -lfftw3 -lfftw3f_threads -lfftw3_threads -lpython3.5m -o /home/gbonanno/gpi/core/math/fft.cpython-35m-x86_64-linux-gnu.so SUCCESS: fft

[aganders3]

This seems to be a problem with the latest version of Anaconda which includes MKL for free, and somehow introduces a conflict with fftw. We're going to look at this, because we'd like to use MKL, but this would be a dealbreaker.

In the meantime, you can get things working by installing nomkl in your Anaconda environment like so:

<gpi_stack_dir>/bin/conda uninstall mkl
<gpi_stack_dir>/bin/conda install nomkl numpy

[gabrielebonanno]

Hi Ashley,

Thanks for looking into this. I tried your solution and the issue seems solved. I'll keep this fix in mind if I need to use mkl in the future.

Gabri

[aganders3]

For those interested in the progress on this issue, I've at least narrowed it down: https://github.com/aganders3/fftw_example

I asked about this on the Anaconda Support newsgroup, so we'll see if there is a good solution. In the meantime, I will re-package GPI with mkl as a conflict and/or a dependency on nomkl as a workaround so this bug won't bite anyone else.

[aganders3]

I'm closing this since there has been no response through Anaconda Support and including nomkl seems to be an adequate workaround for the time being.

[gabrielebonanno]

Thanks for the effort!