RTS returns to wrong address when using ≥ 18MB RAM

[ryandesign]

I'm having this error from CodeWarrior Pro 5's MWC68k/MWCPPC when compiling one particular file:

### MWC68K Compiler Error:
# ### Error: Out of memory ###
# errors caused tool to abort

Many smaller files compiled fine but this file is over 3,000 lines long so maybe that's to be expected. "No problem," I thought. "I'll just give mpw more RAM." Since the default RAM is 16MB I re-ran mpw with --ram=48M. To my surprise, mpw exited with code 0 but did not compile the file. No files compile with this setting, not even tiny ones. In fact --ram=17M is the highest I can go and still have it work, but 17MB is still not enough to compile this file.

Here's where it goes off the rails with --ram=48M:

⋮ 
$02C20B1C   BSR.L      $02C2172C                                ; 61FF 00000C0E
$02C2172C   SUBQ.W     #$2,A7                                   ; 554F
$02C2172E   SUBQ.W     #$2,A7                                   ; 554F
$02C21730   MOVE.L     $0008(A7),-(A7)                          ; 2F2F 0008
$02C21734   MOVE.B     #$01,-(A7)                               ; 1F3C 0001
$02C21738   PEA        $0009(A7)                                ; 486F 0009
$02C2173C   PEA        $000C(A7)                                ; 486F 000C
$02C21740   MOVEQ.L    #$0000000C,D0                            ; 700C
$02C21742   ILLEGAL                                             ; A823
a823 AliasDispatch($000c)
     ResolveAliasFile(/Volumes/Shared/...setup.h)
$02C21744   MOVE.W     (A7)+,D0                                 ; 301F
$02C21746   ADDQ.W     #$2,A7                                   ; 544F
$02C21748   RTS                                                 ; 4E75
$00C20B22   ORI.B      #$00,D0                                  ; 0000 0000
$00C20B26   ORI.B      #$00,D0                                  ; 0000 0000
$00C20B2A   ORI.B      #$00,D0                                  ; 0000 0000
⋮ 
$0200FFFA   ORI.B      #$00,D0                                  ; 0000 0000
$0200FFFE   ORI.B      #$FF,D0                                  ; 0000 FFFF
$02010002   ILLEGAL                                             ; FFFF
⋮ 

I've abbreviated the file path in the logs here for the sake of ... brevity.

The RTS should be going back to 02C20B22 but for some reason it's returning to 00C20B22 instead, where there isn't any code, like it's truncating the return address to 24-bit.

There are many other subroutines before this point that returned just fine. ResolveAliasFile was called 11 times successfully before this failure on the 12th time.

Here's a successful run of mpw with --ram=16M for comparison:

⋮
$00E20B1C   BSR.L      $00E2172C                                ; 61FF 00000C0E
$00E2172C   SUBQ.W     #$2,A7                                   ; 554F
$00E2172E   SUBQ.W     #$2,A7                                   ; 554F
$00E21730   MOVE.L     $0008(A7),-(A7)                          ; 2F2F 0008
$00E21734   MOVE.B     #$01,-(A7)                               ; 1F3C 0001
$00E21738   PEA        $0009(A7)                                ; 486F 0009
$00E2173C   PEA        $000C(A7)                                ; 486F 000C
$00E21740   MOVEQ.L    #$0000000C,D0                            ; 700C
$00E21742   _AliasDispatch                                      ; A823
a823 AliasDispatch($000c)
     ResolveAliasFile(/Volumes/Shared/...setup.h)
$00E21744   MOVE.W     (A7)+,D0                                 ; 301F
$00E21746   ADDQ.W     #$2,A7                                   ; 544F
$00E21748   RTS                                                 ; 4E75
$00E20B22   MOVE.W     D0,D3                                    ; 3600
$00E20B24   MOVE.W     D3,D0                                    ; 3003
$00E20B26   EXT.L      D0                                       ; 48C0
⋮

[ryandesign]

I was compiling with -opt all (all optimizations). If I change -opt all to -opt off (no optimizations) then it is able to compile the large file in 16MB of memory, and there are other levels of optimization I can try, so that's a workaround at least.

[jduerstock]

As a complete shot in the dark, you could try commenting out the

        if (MemorySize <= 0x00ffffff)
            address &= 0x00ffffff;  

lines in mm.cpp.

[ryandesign]

Yeah that does seem like an obvious place, only the toolbox trace log doesn't show StripAddress ever being invoked.

[jduerstock]

Comment out

    memorySetAddress32Bit(FALSE);

in cpu/fmem.c, perhaps?

[ksherlock]

Can you provide some sample code for triggering it?

[ksherlock]

I recreated with a small file. _ResolveAliasFile was clobbering the stack by writing 2 bytes (instead of 1) for the targetIsFolder/isAliased ptrs (which were local variables and therefore on the stack). It should be fixed now 2dcb161f5e527f9f8a83cba73a72edc984d89588

[ryandesign]

_ResolveAliasFile was clobbering the stack

That would certainly do it! Thanks for finding and fixing that. The compile now succeeds with optimizations in 22MB RAM. Thanks @jduerstock for your suggestions too.