scipy.stats installs FORTRAN Control-C handler

[mabl]

Assume you do a simple code like this\

import time
import scipy.stats

while True:
    time.sleep(0.1)

At least under Windows, you will be greated by

forrtl: error (200): program aborting due to control-C event
Image              PC                Routine            Line        Source             
libifcoremd.dll    000007FEE0864314  Unknown               Unknown  Unknown
kernel32.dll       0000000077424C53  Unknown               Unknown  Unknown
kernel32.dll       00000000773E59BD  Unknown               Unknown  Unknown
ntdll.dll          000000007761A2E1  Unknown               Unknown  Unknown

I have had this bug since quite some time, both with Python 2&3. This is critical for me, since this exception is uncachable and hence will not shut down my experiment into a save state.

This bug has been reported multiple times already on other bug trackers:

1. https://github.com/scipy/scipy/issues/2386
2. http://stackoverflow.com/questions/15457786/ctrl-c-crashes-python-after-importing-scipy-stats
3. https://github.com/scipy/scipy/issues/3246
4. https://software.intel.com/en-us/forums/intel-visual-fortran-compiler-for-windows/topic/518251
5. https://github.com/scipy/scipy/pull/3880

I do think it is a packaging issue, since it is specific to the intel library.

[mabl]

The following code can "fix" the issue for me, however it seams to me that disableing the error handler in mkl is the best way to go (haven't figured that one out yet)

import ctypes
import os

def handler(sig):
    try:
        import _thread
    except ImportError:
        import thread as _thread
    _thread.interrupt_main()
    return 1

basepath = r'C:\Users\User\Anaconda3\pkgs\mkl-11.3.1-0\Library\bin'
ctypes.CDLL(os.path.join(basepath, 'libmmd.dll'))
ctypes.CDLL(os.path.join(basepath, 'libifcoremd.dll'))
routine = ctypes.WINFUNCTYPE(ctypes.c_int, ctypes.c_uint)(handler)
ctypes.windll.kernel32.SetConsoleCtrlHandler(routine, 1)

[mabl]

More information on this can be found in the MKL documentation

However the Fortran 15 do not mention any way to diasable it

• https://software.intel.com/en-us/node/525381

but the Fortran 16 docs do!

• https://software.intel.com/en-us/node/579775

Whished this was better documented...

import time
import os
os.environ['FOR_DISABLE_CONSOLE_CTRL_HANDLER'] = '1'

while True:
    time.sleep(0.1)

[acourdavault]

Hello everyone,

In case anyone may be interested, still in this issue. To solve it in my module I set this at the very top of my module to solve the issue, I think adding it to scipy could solve the problem

import os
os.environ['FOR_DISABLE_CONSOLE_CTRL_HANDLER'] = 'T'

would anyone like a PR on this? I've never developed for scipy yet but, i guess i could add my test and look at a proper place to add the os.environ definition in scipy header by default if you think this would be good.

any guidance by an actual developer of scipy would be appreciated as i don't want to make a mess :)

unit test to reproduce the issue:

import platform
import time
import os
from threading import Thread
from multiprocessing import Process, Value
import unittest

def async_sleep(s):
    time.sleep(s)

def run_crash(res_code, do_it):
    try:
        if do_it:
            os.environ['FOR_DISABLE_CONSOLE_CTRL_HANDLER'] = 'T'
        import sklearn
        print('sklearn imported')
        print('Sleeping...')
        t = Thread(target=async_sleep, args=[3])
        t.start()
        import win32api
        win32api.GenerateConsoleCtrlEvent(0, 0)
        t.join()

    except KeyboardInterrupt:
        print('Ha, you pressed CTRL-C!')
        res_code = Value('c', 1)  # True

class TestCrash(unittest.TestCase):
    def setUpClass():
        if platform.system() != 'Windows':
            raise unittest.SkipTest("""we do this test n windows only, not on {}""".format(
                platform.system()))

    def setUp(self):
        self._res_code = Value('c', 0)  # False

    def test_crash_sklearn_ctrlc_nok(self):
        try:
            proc = Process(target=run_crash, args=[self._res_code, False])
            self.assertEqual(self._res_code.value, Value('c', 0).value)
            proc.start()
            proc.join()
            self.assertEqual(self._res_code.value, Value('c', 0).value)
        except KeyboardInterrupt:
            pass
    def test_crash_sklearn_ctrlc_ok(self):
        try:
            proc = Process(target=run_crash, args=[self._res_code, True])
            self.assertEqual(self._res_code.value, Value('c', 0).value)
            proc.start()
            proc.join()
            self.assertEqual(self._res_code.value, Value('c', 1).value)
        except KeyboardInterrupt:
            pass

if __name__ == '__main__':
    unittest.main()

[msarahan]

@acourdavault PR's definitely welcome. Please submit them to https://github.com/conda-forge/scipy-feedstock and we'll pull them into https://github.com/anacondarecipes/scipy-feedstock, or vice versa.

[lesteve]

IIUC this issue is Windows + Intel fortran compiler specific so conda-forge may not be the best place for this?

[msarahan]

OK, then submit it to the latter URL and we'll sort out any upstreaming to conda-forge from there.

[acourdavault]

Ok, sorry @msarahan just a few questions (because i cannot ask a question in https://github.com/anacondarecipes/scipy-feedstock)

I'm not sure I understand how to use this repository, as this would be an extra unit test and a patch (to scipy/__init__.py), it s not just build instructions.

If i understand the patch would be applied on the fly to scipy I've seen a patch already exist in the repo: test_stups.patch (although it has been commented in the yaml)

And how do i add a test (i've seen a run_test.py file, but it's not used)

sorry for these question, i don't really understand how this will behave on the CI machine to integrate the changes.

last but not least numpy uses fortran too. and scipy uses numpy. should the fix be in numpy?

(if i had to put it in scipy i would put it before numpy import in __init__.py)

[msarahan]

hmm - why can't you ask questions in that other repo? If you don't have permission to do that, we need to fix it.

The way that I like to create patches is to:

• download the desired source release
• extract it
• cd into the root of the extracted archive
• git init
• git add * (or just the files you know you're going to change)
• make your changes
• git diff > your_patch_name.patch
• copy your_patch_name.patch into the recipe folder, alongside meta.yaml
• add an entry to the source/patches list with your_patch_name.patch

As for numpy and scipy, it depends on exactly how either uses fortran. You can build numpy completely without fortran, so I'm not sure how much it needs the patch. You're certainly welcome to apply it to numpy as well, though.

[acourdavault]

Thank you @msarahan

I'll work on that when i've time.

About the access to issues, this is what i see on the scipy-feedstock repo:

[msarahan]

bizarre, issues are seemingly disabled. Thanks for reporting. We'll look into it.

[lesteve]

For the record this is what numpy/_distributor_init.py looks like for cgohlke numpy+MKL wheel (http://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy):

# Initialize numpy+MKL

import os

# Disable Intel Fortran default console event handler
env = 'FOR_DISABLE_CONSOLE_CTRL_HANDLER'
if env not in os.environ:
    os.environ[env] = '1'

# Prepend the path of the Intel runtime DLLs to os.environ['PATH']
try:
    path = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'core')
    if path not in os.environ.get('PATH', ''):
        os.environ['PATH'] = os.pathsep.join((path,
                                              os.environ.get('PATH', '')))
except Exception:
    pass

NUMPY_MKL = True

I guess the anaconda recipe for numpy could reuse the FOR_DISABLE_CONSOLE_CTRL_HANDLER part. Not sure how this would be tested though.

[vasselai]

Any progress on this? I have the same issue when hitting Ctrl-C to stop a code that has either scipy.stats or seaborn loaded.

[ctoth]

This still breaks. And usually in the worst possible way, I.E. after doing a lot of computation which then gets discarded.

I wonder how many hours collectively humanity has lost to this bug?

Is there anything I can do to make it stop happening?

[msarahan]

Based on the comment above, it looks like you can set the FOR_DISABLE_CONSOLE_CTRL_HANDLER env var to 1. Perhaps you should add this block to your code:

# Disable Intel Fortran default console event handler
env = 'FOR_DISABLE_CONSOLE_CTRL_HANDLER'
if env not in os.environ:
    os.environ[env] = '1'

We can set that in our packages with activation scripts. I'll add a JIRA ticket for that, but we won't be able to get to it right away. You should definitely try to set that env var on your side to see if it helps.

[adamerose]

@ctoth

Is there anything I can do to make it stop happening?

Reinstalling scipy through pip fixed this for me.