search

microsoft / CNTK

Microsoft Cognitive Toolkit (CNTK), an open source deep-learning toolkit
https://docs.microsoft.com/cognitive-toolkit/
Other
17.53k stars 4.28k forks source link

python crashes when defining Trainer #2844

Open MMRohe opened 6 years ago

MMRohe commented 6 years ago

Hey,

I am running CNTK on python on windows 10. I try to run a code (that used to run correctly) on a new machine. When the code goes to this line:

trainer = C.Trainer(y, (loss_function, eval_function), learner, progress_writers)

Python crashes with the following message: Python has stopped working A problem caused the program to stop working correctly....

with no further info.

Any idea what that might be ? The code is running correctly on other hardware so I do not really understand. And also why does it stop at this specific line.

MMRohe commented 6 years ago

This is the setup in case it helps find the problem Selected GPU[0] Tesla V100-PCIE-16GB as the process wide default device. Build info:

            Built time: Dec  5 2017 18:24:40
            Last modified date: Mon Nov 13 23:06:21 2017
            Build type: Release
            Build target: GPU
            With 1bit-SGD: no
            With ASGD: yes
            Math lib: mkl
            CUDA version: 9.0.0
            CUDNN version: 6.0.21
            Build Branch: HEAD
            Build SHA1: ce72a557b02b30a52c965a237b124688035f720f
            MPI distribution: Microsoft MPI
            MPI version: 7.0.12437.6
nikitbiraj commented 6 years ago

I came across with the same error when I am trying to run run_fast_rcnn.py example. ANy idea what might be the reason?

jackbanh commented 6 years ago

I hit the same issue when running the CNTK tutorials, CNTK_103B and CNTK_103C. Crashes when it gets to trainer = C.Trainer(...). 100% repro. CTNK 2.5 CPU only.

thiagocrepaldi commented 6 years ago

Are you using CNTK for Python ? CNTK 2.5 release requires users installing OpenCV, MKL and CUDA. Crashes can happen if they are not properly configured. Check https://docs.microsoft.com/en-us/cognitive-toolkit/Setup-Windows-Python?tabs=cntkpy25

kyoro1 commented 6 years ago

I had encountered the same issue with CPU-Only CNTK2.5, but once MKL was separately installed, this crash was resolved.