Macaulay2 / M2

The primary source code repository for Macaulay2, a system for computing in commutative algebra, algebraic geometry and related fields.
https://macaulay2.com
347 stars 231 forks source link

Crash in capture #1469

Closed mahrud closed 3 years ago

mahrud commented 4 years ago

This tool is potentially very powerful, but it crashes arbitrarily:

i1 : capture format concatenate(28:"a"); exit 0
^C^C
Exit (y=yes/n=no/a=abort/b=backtrace)? b
-* stack trace, pid: 100566
 0# basic_stacktrace at /home/mahrud/.linuxbrew/include/boost/stacktrace/stacktrace.hpp:126
 1# interrupt_handler at /home/mahrud/Projects/M2/M2/M2/BUILD/clang/../../Macaulay2/bin/main.cpp:279
 2# 0x00007F538B320A90 in /lib64/libpthread.so.0
 3# simpleflush at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/stdio.d:373
 4# simpleout at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/stdio.d:416
 5# flushnets at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/stdio.d:439
 6# endlfun at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/stdio.d:587
 7# stdio_less_less__12 at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/stdio.d:968
 8# endlfun at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/actors5.d:520
 9# evaluate_evalraw at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:1293
10# evaluate_eval at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:1253
11# evaluate_applyEEE at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:936
12# evaluate_binarymethod at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:1054
13# lesslessfun2 at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/actors5.d:151
14# evaluate_evalraw at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:1285
15# evaluate_eval at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:1253
16# evaluate_evalraw at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:1407
17# evaluate_eval at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:1253
18# evaluate_applyFCE at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:738
19# evaluate_applyEE at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:891
20# runmethod_1(tagged_union*, tagged_union*) at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/interp.dd:42
21# readeval4(parse_TokenFile_struct*, char, parse_Dictionary_struct*, char, char, char) at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/interp.dd:227
22# readeval3(parse_TokenFile_struct*, char, parse_DictionaryClosure_struct*, char, char, char) at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/interp.dd:272
23# internalCapture(tagged_union*) at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/interp.dd:553
24# evaluate_evalraw at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:1293
25# evaluate_eval at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:1253
26# parallelAssignmentFun at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:1146
27# evaluate_evalraw at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:1329
28# evaluate_eval at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:1253
29# evaluate_evalraw at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:1392
30# evaluate_eval at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:1253
31# evaluate_applyFCE at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:738
32# evaluate_applyEE at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:891
33# method1234 at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/actors5.d:734
34# evaluate_evalraw at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:1302
35# evaluate_eval at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:1253
36# evaluate_evalexcept at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:1428
37# readeval4(parse_TokenFile_struct*, char, parse_Dictionary_struct*, char, char, char) at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/interp.dd:178
38# readeval3(parse_TokenFile_struct*, char, parse_DictionaryClosure_struct*, char, char, char) at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/interp.dd:272
39# loadprint(M2_string_struct*, parse_DictionaryClosure_struct*, char) at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/interp.dd:345
40# commandInterpreter_2(tagged_union*) at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/interp.dd:460
41# evaluate_evalraw at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:1293
42# evaluate_eval at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:1253
43# evaluate_evalraw at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:1325
44# evaluate_eval at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:1253
45# evaluate_evalraw at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:1403
46# evaluate_eval at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:1253
47# evaluate_evalexcept at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/evaluate.d:1428
48# readeval4(parse_TokenFile_struct*, char, parse_Dictionary_struct*, char, char, char) at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/interp.dd:178
49# readeval3(parse_TokenFile_struct*, char, parse_DictionaryClosure_struct*, char, char, char) at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/interp.dd:272
50# readeval(parse_TokenFile_struct*, char, char) at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/interp.dd:284
51# interp_process at /home/mahrud/Projects/M2/M2/M2/Macaulay2/d/interp.dd:600
52# interpFunc(ArgCell*) at /home/mahrud/Projects/M2/M2/M2/BUILD/clang/../../Macaulay2/bin/main.cpp:193
53# ThreadTask::run(SupervisorThread*) at /home/mahrud/Projects/M2/M2/M2/BUILD/clang/../../Macaulay2/system/supervisor.cpp:377
54# SupervisorThread::threadEntryPoint() at /home/mahrud/Projects/M2/M2/M2/BUILD/clang/../../Macaulay2/system/supervisor.cpp:426
55# SupervisorThread::threadEntryPoint(void*) at /home/mahrud/Projects/M2/M2/M2/BUILD/clang/../../Macaulay2/system/supervisor.hpp:100
56# GC_inner_start_routine in /home/mahrud/.linuxbrew/lib/libgc.so.1
57# GC_call_with_stack_base in /home/mahrud/.linuxbrew/lib/libgc.so.1
58# 0x00007F538B315432 in /lib64/libpthread.so.0
59# clone in /lib64/libc.so.6
-- end stack trace *-
exiting

This happened in v1.15 too:

i1 : capture format concatenate(28:"a"); exit 0
^C^C
Exit (y=yes/n=no/b=backtrace)? b
-- stack trace, pid 2516279:
level 0 -- return addr: 0x0x555555669dd5 -- frame: 0x0x7fffe42b3d00
level 1 -- return addr: 0x(nil) -- frame: 0x0x7fffe42b3d16
-- end stack trace
exiting
mahrud commented 4 years ago

Correction: it doesn't crash, just gets stuck in an infinite loop here: https://github.com/Macaulay2/M2/blob/65332ed07101a51255182fb5a16e4e6e20809292/M2/Macaulay2/d/stdio.d#L414-L425

DanGrayson commented 4 years ago

And this one works (??):


i5 : capture "aaaaaaaaaaaaaaaaaaaaaaaaaaaa"

o5 = (false,                                  , {i1 : aaaaaaaaaaaaaaaaaaaaaaaaaaaa, i2 : })
             i1 : aaaaaaaaaaaaaaaaaaaaaaaaaaaa
                                                 o1 = aaaaaaaaaaaaaaaaaaaaaaaaaaaa
             o1 = aaaaaaaaaaaaaaaaaaaaaaaaaaaa
                                                 o1 : Symbol
             o1 : Symbol

             i2 : 

o5 : Sequence
mahrud commented 4 years ago

I can't figure out what makes it go wrong. That string was the shortest thing that worked, but for a more realistic thing take any documentation node and try:

capture toString examples (sum, List)
mahrud commented 4 years ago

@DanGrayson here's the patch for the error above:

diff --git a/M2/Macaulay2/d/stdio.d b/M2/Macaulay2/d/stdio.d
index dc67ff295..003cdac40 100644
--- a/M2/Macaulay2/d/stdio.d
+++ b/M2/Macaulay2/d/stdio.d
@@ -415,6 +415,7 @@ simpleout(o:file,x:string):int := (
          if j == n then (
               if simpleflush(o) == ERROR then (releaseFileFOSS(o); return ERROR);
               j = foss.outindex;
+              n = length(foss.outbuffer);
               );
          b := m-i;                                         -- number of bytes to transfer this time
          if b > n-j then b = n-j;
@@ -554,7 +555,6 @@ export (o:file) << (c:char) : file := (
      foss := getFileFOSS(o);
      if o.output then (
          if foss.hadNet then (
-              foss.hadNet = true;
               foss.nets = NetList(foss.nets,toNet(c));
               )
          else (

However, another error remains in printing nets:

i3 : (capture "1\n2\n3")#1

o3 =       123
     i1 : 1

     o1 = 

     i2 : 2

     o2 = 

     i3 : 3

     o3 = 

     i4 : 
mahrud commented 4 years ago

Commit 1fe22f49059b474de733ec2579ca4fbd411cf41e almost fixed capture, but:

print last capture "4*4"

 *** out of memory trying to allocate -10 bytes, exiting ***
DanGrayson commented 4 years ago

Thanks, that looks similar to the crash that happens when printing in threads, so maybe we'll fix it now! I'll make a debug build and track it down.

DanGrayson commented 4 years ago

Fixed in commit 0e76993ac56d94ca81131971b10f947d3bef125d on the development branch now.

mahrud commented 4 years ago

Thanks, but i is missing:

./M2 -q --silent --no-preload -e 'print last capture "4*4"; exit 0'

1 : 4*4

1 = 16

2 : 

Could you make a branch and open a PR for fixing this instead of commit by commit? That way I could also help.

DanGrayson commented 4 years ago

Thanks, but i is missing:

./M2 -q --silent --no-preload -e 'print last capture "4*4"; exit 0'

1 : 4*4

1 = 16

2 : 

Fixed, by setting the interpreterDepth to 1 when capturing and making the variable thread local.

Could you make a branch and open a PR for fixing this instead of commit by commit? That way I could also help.

Sure, done. See https://github.com/Macaulay2/M2/pull/1481 -- I'll do other things for a while.

mahrud commented 4 years ago
i1 : a = 12

o1 = 12

i2 : capture "a"

o2 = (false,        )
             i1 : a

             o1 = 12

             i2 : 

How would one make capture work in an empty frame? (is that the right term?)

mahrud commented 4 years ago

I'm guessing the dictionaryPath needs to be truncated, but I'm not sure what should remain in it.

DanGrayson commented 4 years ago

Ah, good point. I wouldn't call it a "frame" -- what we want, for the sake of running examples, is to have no previously set global variables around, and to have a place for putting new ones that none of the other threads are using.

All dictionary lookups happen by following the dictionary path:

Macaulay2, version 1.16.0.2
with packages: ConwayPolynomials, Elimination, IntegralClosure, InverseSystems, LLLBases, MinimalPrimes, PrimaryDecomposition, ReesAlgebra, TangentCone, Truncations

i1 : dictionaryPath

o1 = {Truncations.Dictionary, SimpleDoc.Dictionary, InverseSystems.Dictionary, ConwayPolynomials.Dictionary, ReesAlgebra.Dictionary, TangentCone.Dictionary, Classic.Dictionary,
     ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
     PrimaryDecomposition.Dictionary, IntegralClosure.Dictionary, MinimalPrimes.Dictionary, LLLBases.Dictionary, Elimination.Dictionary, User#"private dictionary", User.Dictionary,
     ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
     Core.Dictionary, OutputDictionary, PackageDictionary}

o1 : List

i2 : select(dictionaryPath,mutable)

o2 = {User#"private dictionary", User.Dictionary, OutputDictionary, PackageDictionary}

o2 : List

We want to set dictionaryPath to a new value that includes the dictionaries of just the needed packages, at least one mutable dictionary where new symbols can go, the OutputDictionary, where o1, o2, ... go, and the PackageDictionary, which has to be there for referring to loaded packages. Get rid of the two User dictionaries. A new mutable dictionary can be made with new Dictionary. Make two, put one on the list and assign one to OutputDictionary. To run examples in threads, we have to make the symbol OutputDictionary thread local. Some variables are already thread local: debugLevel engineDebugLevel debuggingMode allowableThreads loadDepth stopIfError. Use those as a model. The code for them all is in the d directory.

mahrud commented 4 years ago

I tried variations of this suggestion, but ran into mysterious issues, for instance with QQ[a..c] or things of this sort. I'll try again later today.

DanGrayson commented 4 years ago

Oh, yes. The problem is that "getSymbol" refers to "User":


i23 : code getSymbol

o23 = ../../../../../../Macaulay2/m2/setup.m2:61:15-63:49: --source code:
      getSymbol = s -> (
           if instance(User,Symbol) then error "getSymbol used before package User created";
           getGlobalSymbol(User#"private dictionary", s))

So we have to make User thread local and give it a suitable value.

mahrud commented 4 years ago

Sure, I figured I have to empty the User dictionaries, but errors persisted:

i1 : R = QQ[a..d,MonomialOrder=>GLex]

o1 = R

o1 : PolynomialRing

i2 : a^3*b+a^4+b^100
../../../../Macaulay2/m2/robust.m2:98:26:(1):[14]: error: no method for binary operator ^ applied to objects:
--            a (of class Symbol)
--      ^     3 (of class ZZ)
currentString:2:2:(3):[13]: --back trace--
mahrud commented 4 years ago

And this:

i1 : R = ZZ/101[a .. o];

i2 : f = genericMatrix(R, a, 3, 5)
../../../../Macaulay2/m2/methods.m2:37:35:(1):[15]: error: no method found for applying genericMatrix to:
     argument 1 :  R (of class PolynomialRing)
     argument 2 :  a (of class Symbol)
     argument 3 :  3 (of class ZZ)
     argument 4 :  5 (of class ZZ)
../../../../Macaulay2/m2/methods.m2:133:19:(1):[14]: --back trace--
currentString:2:5:(3):[13]: --back trace--
DanGrayson commented 4 years ago

Now it's a mystery. What if you insert use R?

mahrud commented 3 years ago

Since capture doesn't crash anymore, I'll close this and open another one with a list of remaining issues.