emxomf: Index too large

[dmik]

There is a problem with emxomf when it comes to a source file containing too much debug info (i.e. compiled with the -g' command line switch): it will fail with theIndex too large` error on such a file.

[dmik]

Here is the test case: http://rpm.netlabs.org/test/gcc-issue-7-test.zip. It only involves as and emxomf from GCC, nothing more (there is a source .S file within). If you build it with no -Zomf option it works fine (no emxomf involved).

The problem is quite annoying. I started to see it often with the recent Firefox (24esr), more info here: https://github.com/bitwiseworks/mozilla-os2/issues/24#issuecomment-40694706

[StevenLevine]

This error occurs when there are more than 32K names (LNAMES), segments (SEGDEF) or groups (GRPDEF) in a single object file. See put_idx() at emxomf.c:942. Either the .o file is really massive or there's a variable that's not getting reset.

[StevenLevine]

A bit of time with the debugger implies that the test case has something on the order of 70K items that need index entries. This far exceeds the OMF 32K index entry limit. I suspect most of these are for debug data so I am investigating how best to prune the list of items that need indexing.

[dmik]

Aha, I see. By 'prune' you mean dropping them? We should think on how to keep them as it makes no sense to build with -g otherwise. May be we could automatically split it in several .obj files when the 32K limit is reached? We could then join the resulting parts in an OMF lib so that one file can still be used per one input .S file.

[StevenLevine]

Yes, prune would drop what did not fit, which perhaps some heuristics to guess which are most likely to be candidates for dropping. In your testcase, you would only lose about 20% of the debug data. This is relatively easy solution to implement.

Like you I already thought of splitting the output to multiple OMF files. This is a viable solution, but it is non-trivial to implement. The code needs to write the correct index entries into each OMF file and some index entries will need to go into more than one OMF file. This will require a bit of record keeping. It's probably worth doing, but not right now.

[StevenLevine]

We are making progress. Oddly the testcase only fails for one item (i.e. sym_ptr[42560]. If I skip over this failure in the debugger, emxomf runs to completion with no errors. I don't yet understand this item. It is n_type 9 and I've yet to find the definition for n_type 9. This is starting look like something other than a simple overflow. More later.

[StevenLevine]

Brain crap. Type is N_BSS | N_EXT, which I do understand. Back to the bugger.

[StevenLevine]

Brain cramp. Type is N_BSS | N_EXT, which I do understand. Back to the bugger.

[StevenLevine]

Try

http://www.warpcave.com/betas/emxomf_exe-20140420.zip

This is a debug build which bypasses the index abort with minimal effect on HLL debug data.

[dmik]

Wow, that was fast again. Works here:

emxomf warning: Unknown range type: 16..0
emxomf info: parsing sym no.3 type=128 at char ' ' in position 22:
__float128:t1=r0;16;0;
emxomf warning: syntax error in stabs:   (off 22)
 stabs: __float128:t1=r0;16;0;
emxomf info: parsing sym no.3 type=128 at char ' ' in position 22:
__float128:t1=r0;16;0;
emxomf warning: HLL debug suppressed for symbol '_gPendingPointerLockRequest' - index 35332 too large

The last message is from your fix, isn't it?

Anyway, I will try to build all the full stuff in -g again on my spare PC that I now have.

[dmik]

No, I can't fully test it ATM due to https://github.com/bitwiseworks/mozilla-os2/issues/29#issuecomment-40875596. Which means that even if all builds with the fixed emxomf there is no working tool to link it together yet.

[StevenLevine]

Yes. The last message is mine. It is interesting that of all the debug data, this is the only item with an index overflow.

It might make sense to revisit the other stabs warnings in the future. They are debug data types unknown to emxomf.

I'm a tad confused. Can't you get a good -g build with Knut's v1.7 wlink? If not, my wlink build, when it arrives, will probably fail the same way.

[dryeo]

On 04/21/14 12:50 PM, StevenLevine wrote:

I'm a tad confused. Can't you get a good -g build with Knut's v1.7 wlink? If not, my wlink build, when it arrives, will probably fail the same way.

Knuts build of wlink stopped working some time back, Rich patched it to more aggressively use high memory and that has been working for regular builds. Now we're back to "Error! E3009: dynamic memory exhausted" for debug builds testing with 17esr there are about 3179 object files getting linked and it is using about 600 MBs of memory before dieing here. btw with 17ESR and Pauls build of 4.4.6 I don't get any emxomf errors

[StevenLevine]

Too many moving parts. This needs a separate ticket.

I guess, I need to finish implementing upper memory heap support for OpenWatcom. What Rich did was a partial solution, but OpenWatcom really needs generic upper memory heap support similar to what kLIBC provides. The first version will probably not include Knut's os2safe.h features.

FWIW, 600MB seems lowish. It implies a lot of the wlink control data is still in low memory.

[dmik]

This task actually doesn't seem to belong here; emxomf belongs to the LIBC tool chain. There is a LIBC ticket http://trac.netlabs.org/libc/ticket/293 that contains the Steven's patch to this problem. So closing this one.