[I7-1780] [Mantis 1816] Malloc error on Flexible Survival (Windows ni)

[curiousdannii-testing]

Reported by : nuku

Description :

I worked out all the errors and..

The application ran your source text through the Inform 7 compiler, as usual, but the compiler unexpectedly failed. This should not happen even if your source text is gibberish, so you may have uncovered a bug in the program.

When a program like the I7 compiler fails, it typically returns an error number: this time, the number was 2, so I7 probably stopped because of a fatal file-system error.

It is very unlikely that your computer is at fault. More likely causes are:

•disc space running out on the volume holding the project;
•trying to run a project from a read-only volume, such as a burned CD or DVD;
•trying to run a project which belongs to another user, whose files you have no permission to alter.
However, if you think it more likely that the Inform 7 compiler is at fault, please check that you have the currently distributed build of the system: if you have, please consider taking the time to fill out a bug report at the Inform bug tracker (www.inform7.com/bugs).

Looking in the console:

Run out of memory: malloc failed

Compiler finished with code 2

Additionally: found the memory used is 1.9 Gigs of ram. I have plenty more, but it crashes right there. 1973.3 MB

Steps to reproduce :

This is Flexible Survival, what's the best way to get you the behemoth I'm currently working on?

Additional information :

imported from: [Mantis 1816] Malloc error on Flexible Survival (Windows ni)

• status: Closed
• resolution: Resolved
• resolved: 2022-04-07T05:00:56+10:00
• imported: 2022/01/10

[curiousdannii-testing]

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by DavidK :
If it were anyone else, I would assume a bug, but you may well be actually using up all the 2Gb address space of the ni compiler process.

The first thing to do is see if there is really a memory leak, or if your project is just too vast for the address space. Would you package up the Inform project and put it somewhere so I can compile it?

[curiousdannii-testing]

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by DavidK :
When I have a chance I will modify the NI executable to have the Large Address Aware flag set in its Win32 header, which means that the process gets the full 4Gb address space on 64-bit Windows (https://blogs.msdn.microsoft.com/oldnewthing/20050601-24) Provided there aren't any coding errors in NI related to pointers and signed arithmetic, that should give enough room for compiling.

If that doesn't work I'll have to look at bundling 32-bit and 64-bit builds of NI and have the front-end choose the right one for the OS.

[curiousdannii-testing]

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by DavidK :
Turning on the GNU ld linker flag "--large-address-aware" doesn't cause any test harness failures, and allows compilation of FS to succeed. The Windows build of 6M62 has been updated to include the rebuilt ni.

[curiousdannii-testing]

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by zarf :
I gave it a go on the Mac.

(Quick summary: create an I7 project. Download all of https://github.com/Nuku/Flexible-Survival and move it to project.materials/Extensions. Copy Inform/story.ni from that directory to project.inform/Source/ .)

On my iMac (2.7 GHz Intel Core i5), the compile ran for ten minutes. The process size grew to 2.59 GB virtual. It then failed with a simple error:

In the line '"[Yamato Dragoness Desc]"' , I was expecting that 'Yamato Dragoness Desc' would be something to 'say', but it didn't look like any form of 'say' that I know.

I'll try fixing that error. But it does not look like memory is leaking uncontrollably. This is just a test case that requires more than two and a half gigabytes of memory. The Mac compiler is 64-bit and seems to cope with this, as long as you're okay with a ten-minute compile time.

[curiousdannii-testing]

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by zarf :
I changed the capitalization on a couple of say phrases in "Blue Bishop/Yamato Dragoness for FS.i7x". (I didn't think capitalization mattered, but...?)

On this try I got a more serious error:

I've also read Chinchilla Cuntboy for FS by Auto Translator, which is 1528 words long.

  >--> Too much text in quotation marks: "[if you is female and Female Widow

    Convert is male...

Contra the error message, the quoted text is actually from "Auto Translator/Female Widow Convert for FS.i7x".

The ni compiler crashed after displaying this error:

Exception Type:EXC_BAD_ACCESS (SIGSEGV)

Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000004

Exception Note:EXC_CORPSE_NOTIFY

VM Regions Near 0x4:

--> 

    __TEXT 0000000000001000-000000000026a000 [ 2468K] r-x/rwx SM=COW  /Applications/Local Apps/*/Inform-6M62.app/Contents/MacOS/ni
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread

0   ni      0x0005a7fd SourceFiles__total_word_count + 13

1   ni      0x000e0572 Extensions_Dictionary_time_stamp + 290

2   ni      0x000e2704 Extensions_Documentation_write_extension_documentation_page + 2516

3   ni      0x000e1ce4 Extensions_Documentation_write_extension_documentation + 52

4   ni      0x000e21eb Extensions_Documentation_write_extension_documentation_page + 1211

5   ni      0x000e1ce4 Extensions_Documentation_write_extension_documentation + 52

6   ni      0x000db3c4 Extensions_Documentation_write_sketchy + 36

7   ni      0x000db073 Extensions_Files_write_sketchy_documentation_for_extensions_found + 51

8   ni      0x0004884e Extensions_Files_update_census + 94

9   ni      0x000479f5 Index__index_actual_element + 69

10  ni      0x00047697 Index__open_file + 2375

11  ni      0x001a12c8 TemplateFiles__interpret + 25064

12  ni      0x001fa820 Main__core_inform_main + 1568

13  ni      0x001fa1fb main + 43

14  ni      0x00001bfd _start + 212

15  ni      0x00001b28 start + 40

I'm going to call that a separate bug. The problem nuku is running into here is simply that the Windows ni.exe cannot address enough memory.

[curiousdannii-testing]

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by zarf :
"If that doesn't work I'll have to look at bundling 32-bit and 64-bit builds of NI and have the front-end choose the right one for the OS."

That sounds like the right solution long-term.

[curiousdannii-testing]

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by dfremont :
I get the same crash on my Mac. However, it occurs during the census of unused extensions that happens after Inform has successfully compiled the story, and is easily avoided: just delete the "Auto Translator" folder (none of those extensions are used). Then after making the aforementioned changes to "Yamato Dragoness for FS", the story compiles successfully on my machine (running Mac OS 10.10.5).

[curiousdannii-testing]

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by dfremont :
I have submitted a new report, with a simplified test case, for the crash: #0001830.

[curiousdannii-testing]

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by nuku :
The version is 6M62, to be official. I didn't see it in the dropdown.

[curiousdannii-testing]

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by nuku :
I could be wrong, but it does this before it finishes tallying up all the error messages I can see in the console and may (or may not) be related to that. I'm slowly hacking through the code, pruning these errors.

Inform became much more strict about a few things that were used a LOT in this code.

[curiousdannii-testing]

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by nuku :
Nope, fixed every error it reported, compiler error remains the same.

[curiousdannii-testing]

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by nuku :
Sure, here you are: https://github.com/Nuku/Flexible-Survival

[curiousdannii-testing]

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by nuku :
The .ni is in the 'inform' directory.

[curiousdannii-testing]

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by nuku :
A 10 minute compile is fine by me, but I think the windows version is 32 bit? How do we fix?

[curiousdannii-testing]

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by nuku :
How do I get inform to add the '/LARGEADDRESSAWARE' to its call to .ni?

[curiousdannii-testing]

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by nuku :
Worked! Compiled! We are in business!

[curiousdannii-testing]

557058:4c095ffd-6d6f-47ce-9e73-77c613347b86:

Comment by TigerStripes2 :
Another FS teammate here. To further the testing on this bug, I applied Nuku's changes to my local copy and did several compile attempts. It is also important to note that we've been awaiting this compiler update because we'd previously reached an error in the compiler which prevented us from adding new material w/o removing old material - see http://inform7.com/mantis/view.php?id=1610

Old version of Inform (6L38):

• Compiled with version of FS code Nuku tweaked for 6M62 the w/o reintegration of removed extensions, resulting in successful compile of this restricted version.
• Attempting to add back even one of the smaller extensions removed resulted in the previous problem recurring.
• This confirms we were very near 6L38's limit, but still able to functionally compile and run.

New version of Inform:

• Installed new version of Inform (6M62).
• Attempted to compile the same version w/o reintegration of removed extensions, resulting in a failure to compile identical to the reported new error. This failure occurred at the very near or at the end of compiling.