[I7-1937] [Mantis 1973] Can get out of memory error before the compiler is really out of memory

curiousdannii-testing commented 2 years ago

Reported by : nuku

Description :

Translating the Source - Failed
The application ran your source text through the Inform 7 compiler, as usual, but the compiler unexpectedly failed. This should not happen even if your source text is gibberish, so you may have uncovered a bug in the program.

When a program like the I7 compiler fails, it typically returns an error number: this time, the number was 11, and that probably indicates that the compiler failed to manage its data structures properly. Perhaps you created a complicated situation on which it has not been fully tested.

The best option now is probably to reword whatever was last changed and to try again. Subsequent attempts will not be affected by the failure of this one, so there is nothing to be gained by restarting the application or the computer. A failed run should never damage the source text, so your work cannot be lost.

If you think it likely that the Inform 7 compiler is at fault, please check that you have the currently distributed build of the system: if you have, please consider taking the time to fill out a bug report at the Inform bug tracker (www.inform7.com/bugs). If you think the fault may be due to a problem in an extension you're using, then please contact the extension's author directly.

Sorry for the inconvenience

Steps to reproduce :

https://github.com/Nuku/Flexible-Survival

Additional information :

imported from: [Mantis 1973] Can get out of memory error before the compiler is really out of memory
status: Closed
resolution: Resolved
resolved: 2022-04-12T08:45:31+10:00
imported: 2022/01/10

curiousdannii-testing commented 2 years ago

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by DavidK :
Do you get the same stack trace on Windows? The latest Windows build should also give stack traces on NI failure on the Console sub-tab, but only for the latest June 2016 build - you might need to re-download it to be sure you've got the most up-to-date Windows build of 6M62.

curiousdannii-testing commented 2 years ago

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by DavidK :
Good call: an internal error is generated at 10,000 memory blocks (about 2Gb). Shame it doesn't actually manage to show the error. When I get the chance I'll up this limit and rebuild on Windows as a test.

curiousdannii-testing commented 2 years ago

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by DavidK :
The core problem here is that the internal constant MAX_BLOCKS_ALLOWED should be settable in some way.

However, for the moment, I've just recompiled ni with MAX_BLOCKS_ALLOWED = 15000 and put the result here: http://davidkinder.co.uk/ni.zip

When I try compiling the game, it now eventually fails with a "Too much text in quotation marks" error, and then crashes out. This is unfortunate, but I suspect that the "too much text" error is real and needs to be fixed in the game source.

curiousdannii-testing commented 2 years ago

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by DavidK :
Yes, it looks like I'm getting http://inform7.com/mantis/view.php?id=1830 with the patched ni.

curiousdannii-testing commented 2 years ago

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by DavidK :
Anyway, nuku, would you try the patched ni?

curiousdannii-testing commented 2 years ago

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by DavidK :
Great. I'll leave this bug open so that Graham can look at making a proper fix for the memory limit in the next version.

curiousdannii-testing commented 2 years ago

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by dfremont :
From the backtrace, it looks to me like Inform is trying to issue an error about running out of memory (probably the "generating endless structures" internal error). But when preparing the error screen, Inform tries to allocate a new structure, which triggers the out of memory error again. So we have an infinite recursion that eventually exhausts the stack.

Partial backtrace:

Exception Type:EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_PROTECTION_FAILURE at 0x00000000bf7ffd5c
Exception Note:EXC_CORPSE_NOTIFY

VM Regions Near 0xbf7ffd5c:
Stack 00000000b0083000-00000000b0094000 [ 68K] rw-/rwx SM=COW
--> Stack Guard 00000000bc000000-00000000bf800000 [ 56.0M] ---/rwx SM=NUL
Stack 00000000bf800000-00000000c0000000 [ 8192K] rw-/rwx SM=PRV

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_c.dylib 0x99bc286e __vfprintf + 12
1 libsystem_c.dylib 0x99beb1aa __v2printf + 670
2 libsystem_c.dylib 0x99bd0469 _vsnprintf + 832
3 libsystem_c.dylib 0x99bd04cc vsnprintf_l + 63
4 libsystem_c.dylib 0x99bc1292 snprintf + 82
5 ni 0x00001f90 Streams__printf + 608
6 ni 0x0003e977 HTML__begin_html_table + 135
7 ni 0x0003e5fb HTML__html_outcome_image + 1051
8 ni 0x000581b2 Problems__show_problem_location + 98
9 ni 0x00059550 Problems__issue_problem_begin + 352
10 ni 0x00005797 Problems_Issue_internal_error_fn + 215
11 ni 0x0000550b Memory__allocate_another_block + 91
12 ni 0x00006dd8 Memory__allocate + 120
13 ni 0x00027e50 allocate_parse_node_annotation_array + 80
14 ni 0x0002805b allocate_parse_node_annotation + 75
15 ni 0x000c2001 ParseTree__pna_new + 17
16 ni 0x000c2234 ParseTree__pn_annotate_pointer + 212
17 ni 0x00058101 ParseTree__set_problem_falls_under + 81
18 ni 0x00057fac Problems__visit_for_headings + 108
19 ni 0x000c6b9e ParseTree__traverse_from_ppn_nocs + 94
20 ni 0x000c6bf6 ParseTree__traverse_from_ppn_nocs + 182
21 ni 0x000c6bf6 ParseTree__traverse_from_ppn_nocs + 182
22 ni 0x000c6bf6 ParseTree__traverse_from_ppn_nocs + 182
23 ni 0x000c6bf6 ParseTree__traverse_from_ppn_nocs + 182
24 ni 0x00057f32 ParseTree__traverse_ppn_nocs + 66
25 ni 0x00057e86 Problems__find_headings_at + 118
26 ni 0x00058248 Problems__show_problem_location + 248
27 ni 0x00059550 Problems__issue_problem_begin + 352
28 ni 0x00005797 Problems_Issue_internal_error_fn + 215
29 ni 0x0000550b Memory__allocate_another_block + 91
30 ni 0x00006dd8 Memory__allocate + 120
31 ni 0x00027e50 allocate_parse_node_annotation_array + 80
32 ni 0x0002805b allocate_parse_node_annotation + 75
33 ni 0x000c2001 ParseTree__pna_new + 17
34 ni 0x000c2234 ParseTree__pn_annotate_pointer + 212
35 ni 0x00058101 ParseTree__set_problem_falls_under + 81
36 ni 0x00057fac Problems__visit_for_headings + 108
37 ni 0x000c6b9e ParseTree__traverse_from_ppn_nocs + 94
38 ni 0x000c6bf6 ParseTree__traverse_from_ppn_nocs + 182
39 ni 0x000c6bf6 ParseTree__traverse_from_ppn_nocs + 182
40 ni 0x000c6bf6 ParseTree__traverse_from_ppn_nocs + 182
41 ni 0x000c6bf6 ParseTree__traverse_from_ppn_nocs + 182
42 ni 0x00057f32 ParseTree__traverse_ppn_nocs + 66
43 ni 0x00057e86 Problems__find_headings_at + 118
44 ni 0x00058248 Problems__show_problem_location + 248
45 ni 0x00059550 Problems__issue_problem_begin + 352
46 ni 0x00005797 Problems_Issue_internal_error_fn + 215
47 ni 0x0000550b Memory__allocate_another_block + 91
48 ni 0x00006dd8 Memory__allocate + 120

(continues identically for a long time)

curiousdannii-testing commented 2 years ago

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by dfremont :
Looking at the debugging log, compilation appears to go smoothly until exactly 10,000 memory blocks have been allocated, when the crash happens. So you may just be hitting the threshold used by Inform to detect when runaway memory allocation is occurring. If that's the case, there probably isn't anything you can do except make the project smaller and wait until the limit is raised (or Inform's memory usage reduced).

An experiment you can try: if the explanation above is right, then putting Korvin back and removing a sufficient amount of other material (how much is hard to say) should allow a successful build. If the error really depends on that specific extension, then something else is going on. But I suspect it's just that Korvin is one of the last extensions included, and pushes you over the edge.

curiousdannii-testing commented 2 years ago

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by dfremont :
Has #0001830 been fixed? It might be the further problem here, as it was producing a "too much text in quotation marks" error and crash before for this project - see the notes on #0001816 (at least at that time, you could fix this by deleting the extensions by "Auto Translator").

The "too much text in quotation marks" error means there's a quoted string that's too long (it's not about the total amount of text in your story). You can easily get around it by splitting the string into two or more parts and saying them separately, or by factoring out part of the string into a text substitution.

curiousdannii-testing commented 2 years ago

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by dfremont :
To be clear, #0001830 happens when there's a faulty extension that you're not including in your story. You need to either fix the error in the extension (i.e. shorten that overlong quoted text), or delete the extension file completely.

curiousdannii-testing commented 2 years ago

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by nuku :
The extension Korvin by Stripes is the culprit, but I have no idea why. Commenting it out and making dummy variables that other code looks for allows for compiling.

curiousdannii-testing commented 2 years ago

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by nuku :
Oof, anything I can do on my end to slip by this?

curiousdannii-testing commented 2 years ago

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by nuku :
We are the largest Inform game that I'm aware of, stressing the engine in new and novel ways. Might that ceiling become something you can define in the game? That is, with a 'use' statement.

curiousdannii-testing commented 2 years ago

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by nuku :
The console ends with:

I've also read Glulx Entry Points by Emily Short, which is 1138 words long.

Compiler finished with code 11

curiousdannii-testing commented 2 years ago

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by nuku :
Too much text? That's a thing? We need our text. Suggestions?

curiousdannii-testing commented 2 years ago

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by nuku :
Curious, since those have been there a while. Still, if it helps?

curiousdannii-testing commented 2 years ago