Core dump on circular definition when analyzing qtconsole-4.4.2 (error code -11)

yoheimatt commented 6 years ago

What does it mean for error ?

\u019b Client exited with error code -11

JelleZijlstra commented 6 years ago

Probably that the client segfaulted. You'll have to provide more information for people to help you figure out the exact bug (unless there are known segfault bugs in pyre?). You may be able to capture a core dump or run the client under gdb.

dark commented 6 years ago

That is correct, that's likely caused by a segmentation fault.

Could you please provide the command that you were running, plus its full output? If you were running pyre check, it would also help to run it with --debug, i.e. pyre --debug check. Otherwise, if you were using server mode, the output of pyre rage would help too.

yoheimatt commented 6 years ago

Thanks all. Below is the output from "pyre --preserve-pythonpath --debug check":

root@4993b80bc01c:/workspace/src/ct-lung-nodule-visual-classifier/unit_tests# pyre --preserve-pythonpath   --debug  check
2018-11-08 02:17:29,012 DEBUG No configuration found at `.pyre_configuration.local`.
2018-11-08 02:17:29,012 DEBUG Reading configuration `.pyre_configuration`...
2018-11-08 02:17:29,013 DEBUG Found source_directories: `.`
2018-11-08 02:17:29,014 DEBUG Running `/usr/local/bin/pyre.bin check -debug -project-root /workspace/src/ct-lung-nodule-visual-classifier -filter-directories unit_tests -workers 8 -typeshed /usr/local/lib/pyre_check/typeshed/ -search-path /usr/local/lib/python3.5/dist-packages/ /workspace/src/ct-lung-nodule-visual-classifier/unit_tests`
2018-11-08 02:17:29,080 INFO Finding type stubs in `/workspace/src/ct-lung-nodule-visual-classifier/unit_tests`...
2018-11-08 02:17:29,181 INFO Finding type stubs in `/usr/local/lib/python3.5/dist-packages`...
2018-11-08 02:17:29,255 INFO Finding type stubs in `/usr/local/lib/pyre_check/typeshed/third_party`...
2018-11-08 02:17:29,258 INFO Finding type stubs in `/usr/local/lib/pyre_check/typeshed/stdlib`...
2018-11-08 02:17:29,359 INFO Finding external sources in `/usr/local/lib/python3.5/dist-packages`...
2018-11-08 02:17:29,519 INFO Parsing 8471 stubs and external sources...
2018-11-08 02:17:38,790 PERFORMANCE Stubs parsed: 9.636803s
2018-11-08 02:17:58,700 INFO Parsing 5 sources in `/workspace/src/ct-lung-nodule-visual-classifier/unit_tests`...
2018-11-08 02:17:58,748 PERFORMANCE Sources parsed: 0.116605s
2018-11-08 02:17:58,748 PERFORMANCE Registered ignores: 0.040567s
2018-11-08 02:17:58,849 INFO Building type environment...
2018-11-08 02:19:02,491 ERROR Client exited with error code -11

dark commented 6 years ago

Thanks for the info, I think the exception info is being swallowed for some reason. Could you please try and run the client command manually and post the result? Thanks!

As a single line: /usr/local/bin/pyre.bin check -debug -project-root /workspace/src/ct-lung-nodule-visual-classifier -filter-directories unit_tests -workers 8 -typeshed /usr/local/lib/pyre_check/typeshed/ -search-path /usr/local/lib/python3.5/dist-packages/ /workspace/src/ct-lung-nodule-visual-classifier/unit_tests

yoheimatt commented 6 years ago

Here are two outputs:

root@4993b80bc01c:/workspace/src/ct-lung-nodule-visual-classifier# /usr/local/bin/pyre.bin check -debug -project-root /workspace/src/ct-lung-nodule-visual-classifier -filter-directories unit_tests -workers 8 -typeshed /usr/local/lib/pyre_check/typeshed/ -search-path /usr/local/lib/python3.5/dist-packages/ /workspace/src/ct-lung-nodule-visual-classifier/unit_tests
2018-11-08 11:26:20 INFO Finding type stubs in `/workspace/src/ct-lung-nodule-visual-classifier/unit_tests`...
2018-11-08 11:26:20 INFO Finding type stubs in `/usr/local/lib/python3.5/dist-packages`...
2018-11-08 11:26:20 INFO Finding type stubs in `/usr/local/lib/pyre_check/typeshed/third_party`...
2018-11-08 11:26:20 INFO Finding type stubs in `/usr/local/lib/pyre_check/typeshed/stdlib`...
2018-11-08 11:26:20 INFO Finding external sources in `/usr/local/lib/python3.5/dist-packages`...
2018-11-08 11:26:20 INFO Parsing 8471 stubs and external sources...
2018-11-08 11:26:29 PERFORMANCE Stubs parsed: 9.692490s
2018-11-08 11:26:50 INFO Parsing 5 sources in `/workspace/src/ct-lung-nodule-visual-classifier/unit_tests`...
2018-11-08 11:26:50 PERFORMANCE Sources parsed: 0.101689s
2018-11-08 11:26:50 PERFORMANCE Registered ignores: 0.043681s
2018-11-08 11:26:50 INFO Building type environment...
Segmentation fault (core dumped)

root@4993b80bc01c:/workspace/src/ct-lung-nodule-visual-classifier# gdb /usr/local/bin/pyre.bin  core.29017             
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/bin/pyre.bin...done.
[New LWP 6800]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/bin/pyre.bin check -debug -project-root /workspace/src/ct-lung-nodul'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000001e5c973 in caml_c_call ()
(gdb) bt
#0  0x0000000001e5c973 in caml_c_call ()
Backtrace stopped: Cannot access memory at address 0x7fff2ebb4ff8

dark commented 6 years ago

That's really interesting, it's the first ocaml core dump I see in a long time :)

We have a few options here, not sure which one would be better though.

Are you able to share the code you are running pyre on, or maybe reduce the problem to a smaller size?
Rather than running gdb on the core dump, what happens if you run pyre directly under gdb? I.e. gdb --args /usr/local/bin/pyre.bin check -debug -project-root /workspace/src/ct-lung-nodule-visual-classifier -filter-directories unit_tests -workers 8 -typeshed /usr/local/lib/pyre_check/typeshed/ -search-path /usr/local/lib/python3.5/dist-packages/ /workspace/src/ct-lung-nodule-visual-classifier/unit_tests. This might still not work - we often see gdb behave incorrectly once pyre spawns its internal workers to do the job.
We are releasing pyre binaries compiled with optimizations. If you're willing to compile your own version of pyre from master and re-run gdb that way, these are the instructions for compilation: https://pyre-check.org/docs/installation.html

Last thing, what's the pyre version you are using? (pyre --version)

yoheimatt commented 6 years ago

root@4993b80bc01c:/workspace/src/ct-lung-nodule-visual-classifier# pyre --version pyre version 0.0.17

I will try your option 3. Will keep you all updated.

yoheimatt commented 6 years ago

Below is the end of the output of make. Not sure what to do next?

Installed /root/.opam/4.06.0/lib/hack_parallel/hack_parallel.cmx
Installed /root/.opam/4.06.0/lib/hack_parallel/hack_parallel.cmti
Installed /root/.opam/4.06.0/lib/hack_parallel/META
make[1]: Leaving directory '/workspace/src/ct-lung-nodule-visual-classifier/pyre-check/hack_parallel'
Build info: Linux x86_64 @ Fri Nov 09 2018 (development build)
abort: no repository found in '/workspace/src/ct-lung-nodule-visual-classifier/pyre-check' (.hg not found)!
Git commit: 4c6832f2f8b17542b8a7e4826c605b8f721fdbc8
dune build @install -j auto --profile dev
      menhir parser/generator.{ml,mli}
Note: the nonterminal symbol separated_list (from generator.mly) is renamed generator_separated_list.
Note: the nonterminal symbol separated_nonempty_list (from generator.mly) is renamed generator_separated_nonempty_list.
Warning: 40 states have shift/reduce conflicts.
Warning: 4 states have reduce/reduce conflicts.
Warning: 310 shift/reduce conflicts were arbitrarily resolved.
Warning: 34 reduce/reduce conflicts were arbitrarily resolved.

dark commented 6 years ago

It compiled successfully! The binary will be at _build/default/main.exe.

yoheimatt commented 6 years ago

Hi, I narrowed down to the presence of qtconsole, a third party package, is the cause of the crash. The gdb didn't help.

yoheimatt commented 6 years ago

Is it possible to ignore certain sub search paths?

dark commented 6 years ago

If the problem is with parsing, unfortunately there is no way not to parse a subset of the files from a search path. We have a way to exclude results from a given path by using do_not_check, but that does not stop pyre from parsing and analyzing those files.

That said, we would be very interested in having a look at the file/module/stub that's causing the crash. Did you say it's coming from qtconsole; is this the one? https://pypi.org/project/qtconsole/

dark commented 6 years ago

Crash confirmed by running pyre on https://files.pythonhosted.org/packages/f1/81/a7c62fc0cae20cf96fa94013617f13a1ef561323d92a101739d41e1b1ea8/qtconsole-4.4.2.tar.gz

dark commented 6 years ago

Reproduced on a recent master, this is indeed infinite recursion and stack space exhaustion:


#0  caml_hash (count=21, limit=201, seed=1, obj=3) at hash.c:185
#1  0x0000000000e5e7b1 in camlHashtbl__hash_1004 () at hashtbl.ml:23
#2  0x0000000000e0204a in camlBase__Hashtbl__slot_2859 () at src/hashtbl.ml:70
#3  0x0000000000e024ec in camlBase__Hashtbl__find_and_call_3010 () at src/hashtbl.ml:155
#4  0x0000000000798bb5 in camlAnalysis__TypeOrder__linearize_14109 () at analysis/typeOrder.ml:347
#5  0x0000000000db4810 in camlBase__List__count_map_2188 () at src/list.ml:322
#6  0x0000000000798c00 in camlAnalysis__TypeOrder__linearize_14109 () at src/list.ml:350
#7  0x0000000000db4810 in camlBase__List__count_map_2188 () at src/list.ml:322
#8  0x0000000000798c00 in camlAnalysis__TypeOrder__linearize_14109 () at src/list.ml:350
[...]
#116365 0x0000000000db4810 in camlBase__List__count_map_2188 () at src/list.ml:322
#116366 0x0000000000798c00 in camlAnalysis__TypeOrder__linearize_14109 () at src/list.ml:350
#116367 0x0000000000798e50 in camlAnalysis__TypeOrder__successors_14154 () at analysis/typeOrder.ml:360
#116368 0x00000000007bc6da in camlAnalysis__Environment__refine_11937 () at analysis/environment.ml:217
#116369 0x0000000000dc3d72 in camlBase__Option__map_1932 () at src/option.ml:183
#116370 0x00000000007bc691 in camlAnalysis__Environment__refine_class_definition_11935 () at src/monad.ml:37
#116371 0x0000000000e2e3b1 in camlList__iter_1083 () at list.ml:100
#116372 0x0000000000728189 in camlService__Environment__populate_3394 () at src/list0.ml:26
#116373 0x000000000072843c in camlService__Environment__build_8983 () at service/environment.ml:87
#116374 0x000000000072aa94 in camlService__Environment__populate_shared_memory_19918 () at service/environment.ml:509
#116375 0x0000000000730c7d in camlService__Check__check_5397 () at service/check.ml:269
#116376 0x000000000071644e in camlCommands__CheckCommand__fun_8675 () at command/checkCommand.ml:67
#116377 0x0000000000775226 in camlScheduler__run_process_3281 () at service/scheduler.ml:47
#116378 0x0000000000704794 in caml_curry20_19 ()
#116379 0x0000000000bffe09 in camlCore__Command__fun_58358 () at src/command.ml:2734
#116380 0x0000000000db055d in camlBase__Exn__handle_uncaught_aux_2152 () at src/exn.ml:113
#116381 0x000000000070aec8 in camlMain__entry () at main.ml:28
#116382 0x0000000000702139 in caml_program ()
#116383 0x0000000000eb6d9a in caml_start_program ()
#116384 0x0000000000e9cd6d in caml_startup_common (argv=0x7fffffffd4e8, pooling=<optimized out>, pooling@entry=0) at startup.c:156
#116385 0x0000000000e9cdab in caml_startup_exn (argv=<optimized out>) at startup.c:161
#116386 caml_startup (argv=<optimized out>) at startup.c:166
#116387 0x0000000000e9cdd5 in caml_main (argv=<optimized out>) at startup.c:173
#116388 0x00000000006ff6cc in main (argc=<optimized out>, argv=<optimized out>) at main.c:44

dark commented 6 years ago

The issue seems confined to the rich_jupyter_widget.py file. If everything else is removed, Pyre still crashes.

dark commented 6 years ago

This is because of a circular definition in rich_jupyter_widget.py. This simplified code still crashes:

class RichIPythonWidget(JupyterWidget):                                                                          
    pass                                                                                                         

class RichJupyterWidget(RichIPythonWidget):                                                                      
    pass                                                                                                         

class RichIPythonWidget(RichJupyterWidget):                                                                      
    pass

jamescasbon commented 5 years ago

Same problem here, segfault after "2018-11-26 09:07:51,264 INFO Building type environment..."

Process starts allocating more and more memory and then goes down.

Would be very nice to have excludes for this reason!

dark commented 5 years ago

A new version of Pyre has been released. Running pyre with --debug should show more information.

maldil commented 5 years ago

@dark I encountered the error ƛ Client exited with error code -11 while running pyre. I ran pyre in debug mode and got the below trace. Do you have any idea about this?

(TensorfFlowExample) 10-249-217-99:tensorflow_ecm malinda$ pyre --search-path "/Users/malinda/Documents/Python/VMS/VM1/TensorfFlowExample/lib/python3.7/site-packages/" --debug  --source-directory . check 
2019-01-23 15:42:21,708 DEBUG No configuration found at `.pyre_configuration.local`.
2019-01-23 15:42:21,708 DEBUG No configuration found at `.pyre_configuration`.
2019-01-23 15:42:21,708 INFO No binary specified, looking for `pyre.bin` in PATH
2019-01-23 15:42:21,708 INFO Found: `/Users/malinda/Documents/Python/VMS/VM1/TensorfFlowExample/bin/pyre.bin`
2019-01-23 15:42:21,708 INFO No typeshed specified, looking for it
2019-01-23 15:42:21,708 DEBUG Trying with: `/Users/malinda/Documents/Python/VMS/VM1/TensorfFlowExample/lib/python3.7/site-packages/pyre_check/pyre_check/typeshed/`
2019-01-23 15:42:21,708 DEBUG Trying with: `/Users/malinda/Documents/Python/VMS/VM1/TensorfFlowExample/lib/python3.7/site-packages/pyre_check/typeshed/`
2019-01-23 15:42:21,708 DEBUG Trying with: `/Users/malinda/Documents/Python/VMS/VM1/TensorfFlowExample/lib/python3.7/pyre_check/typeshed/`
2019-01-23 15:42:21,708 DEBUG Trying with: `/Users/malinda/Documents/ Python/VMS/VM1/TensorfFlowExample/lib/pyre_check/typeshed/`
2019-01-23 15:42:21,708 INFO Found: `/Users/malinda/Documents/Python/VMS/VM1/TensorfFlowExample/lib/pyre_check/typeshed/`
2019-01-23 15:42:21,709 WARNING Setting up a `.pyre_configuration` with `pyre init` may reduce overhead 
2019-01-23 15:42:21,709 DEBUG Running `/Users/malinda/Documents/ Python/VMS/VM1/TensorfFlowExample/bin/pyre.bin check -debug -project-root /Users/malinda/Documents/ Python/GoodProjectsForLibAna/tensorflow_ecm -filter-directories . -workers 8 -typeshed /Users/malinda/Documents/ Python/VMS/VM1/TensorfFlowExample/lib/pyre_check/typeshed/ -search-path /Users/malinda/Documents/ Python/VMS/VM1/TensorfFlowExample/lib/python3.7/site-packages/ /Users/malinda/Documents/ Python/GoodProjectsForLibAna/tensorflow_ecm`
2019-01-23 15:42:21,745 INFO Finding type stubs in `/Users/malinda/Documents/ Python/GoodProjectsForLibAna/tensorflow_ecm`...
2019-01-23 15:42:21,847 INFO Finding type stubs in `/Users/malinda/Documents /Python/VMS/VM1/TensorfFlowExample/lib/python3.7/site-packages`...
2019-01-23 15:42:21,916 INFO Finding type stubs in `/Users/malinda/Documents/ Python/VMS/VM1/TensorfFlowExample/lib/pyre_check/typeshed/stdlib`...
2019-01-23 15:42:21,920 INFO Finding type stubs in `/Users/malinda/Documents/ Python/VMS/VM1/TensorfFlowExample/lib/pyre_check/typeshed/third_party`...
2019-01-23 15:42:22,024 INFO Finding external sources in `/Users/malinda/Documents/ Python/VMS/VM1/TensorfFlowExample/lib/python3.7/site-packages`...
2019-01-23 15:42:22,183 INFO Parsing 5570 stubs and external sources...
2019-01-23 15:42:28,141 WARNING Could not parse 1 external file due to syntax errors! Run `pyre --show-parse-errors check` for more details.
2019-01-23 15:42:28,247 PERFORMANCE Stubs parsed: 6.406399s
2019-01-23 15:42:36,469 INFO Parsing 5 sources in `/Users/malinda/Documents/ Python/GoodProjectsForLibAna/tensorflow_ecm`...
2019-01-23 15:42:36,520 PERFORMANCE Sources parsed: 0.055136s
2019-01-23 15:42:36,520 PERFORMANCE Registered ignores: 0.050510s
2019-01-23 15:42:36,621 INFO Building type environment...
2019-01-23 15:42:51,053 ERROR Order is cyclic:
Trace: qtconsole.jupyter_widget.IPythonWidget -> qtconsole.jupyter_widget.JupyterWidget -> qtconsole.jupyter_widget.IPythonWidget
Uncaught exception:

  Analysis__TypeOrder.Cyclic

Raised at file "format.ml" (inlined), line 242, characters 35-52
Called from file "format.ml", line 469, characters 8-33
Called from file "format.ml", line 484, characters 6-24
2019-01-23 15:42:51,054 ERROR Client exited with error code 1

dark commented 5 years ago

@maldil That's the same circular definition that was reported above. At this time, Pyre will crash when encountering such definition.

Until this is solved, my suggestion is to use, as a workaround, the exclude option of the configuration file to avoid parsing the qtconsole stubs: https://pyre-check.org/docs/configuration.html

facebook / pyre-check

Core dump on circular definition when analyzing qtconsole-4.4.2 (error code -11) #119