@rspfile

[SilvanScherrer]

When building something with gcc -Zomf -Zexe -Zlinker "DISABLE 1121" test.c all works w/o issues. But when we use a rspfile like the following: gcc -Zomf -Zexe -Zlinker "DISABLE 1121" @object1.rsp we get an error Error! E30033: file ld6JgbBQ.: lie(10): directive error near 'DISABLE\'

[SilvanScherrer]

with gcc4 all works as it should. we just verified that.

[komh]

Split it. That is,

-Zlinker DISABLE -Zlinker 1121

See https://github.com/komh/kdllar/issues/3.

[dmik]

Nothing has been changed in GCC itself in this regard. Tthe gcc.exe frontend itself and all other relative executables uses expandargv from libiberty to process response files. This is the documentation for it:

@deftypefn Extension void expandargv (int *@var{argcp}, char ***@var{argvp})

The @var{argcp} and @code{argvp} arguments are pointers to the usual
@code{argc} and @code{argv} arguments to @code{main}.  This function
looks for arguments that begin with the character @samp{@@}.  Any such
arguments are interpreted as ``response files''.  The contents of the
response file are interpreted as additional command line options.  In
particular, the file is separated into whitespace-separated strings;
each such string is taken as a command-line option.  The new options
are inserted in place of the option naming the response file, and
@code{*argcp} and @code{*argvp} will be updated.  If the value of
@code{*argvp} is modified by this function, then the new value has
been dynamically allocated and can be deallocated by the caller with
@code{freeargv}.  However, most callers will simply call
@code{expandargv} near the beginning of @code{main} and allow the
operating system to free the memory when the program exits.

@end deftypefn

It clearly states that arguments in the response file are space-separated. EMX (LIBC), on the contrary, uses line endings as argument separators. Of course, this means that the libiberty approach can't handle options with spaces the way LIBC does them.

Libiberty hasn't changed in this regard between what's used in GCC4 and GCC9. This means that the only way this can work in GCC4 is that somehow EMX code is used (via a direct call to _response at the beginning of main) to handle response files instead of libiberty. What I don't understand is that from what I see in the GCC4 code, _response is called after expandargv so it should be the same.. But given there is a huge mess of everything in there (e.g. some macros used in EMX contain calls to _response), there must be an explanation to that. I just don't want to spend time digging it. I will try to fix itn in GCC9 by using _response directly there as well.

[dmik]

What I don't like here is that _response is not compatible with libiberty (used everywhere in gcc and everywhere else in the Unix world) per se. So it's a potential source of future problems due to incompatibility. But in this particular case I think that keeping GCC for OS/2 compatible to its own older OS/2 builds is more important.

[dmik]

OTOH, I see that libiberty allows preserving white space with either double or single quotes (as well as with backslashes but that's disabled on OS/2 because they are valid path separators). This means that having -Zlinker "DISABLE 1121" in a response file or on the command line should work even with libiberty. Somehow OS/2 bits in gcc's main break this. This needs to be found out.

Note that things get screwed up only when @file is present on the gcc command line. This means, that this works:

gcc -Zomf hello.c -Zlinker "DISABLE 1121"

while this doesn't:

gcc -Zomf hello.c -Zlinker "DISABLE 1121" @hello.rsp

even if hello.rsp itself doesn't contain -Zlinker "DISABLE 1121" or such.

[dmik]

I think I know what it is. It's quite complicated.

1. GCC uses libiberty not only to parse response files via expandargv but also to create them via writeargv.
2. GCC uses response files to pass some options to tools it runs (including the linker) if and only if a response file was used in its own invocation (as I guess, assuming that if it's needed for gcc, it is needed for tools as well). This is done for input files and for output files (%i and %o in GCC SPEC).
3. Linker options (including -Zlinker) somehow appear as part of the %o option substitution.
4. The difference between GCC 4 and GCC 9 is that the former only uses response files for %o options if HAVE_GNU_LD is defined. In GCC 9, they removed a check for HAVE_GNU_LD.
5. OS/2 builds have HAVE_GNU_LD defined (as configure somehow passes this test) but GCC 4 builds explicitly disabled HAVE_GNU_LD in gcc.c — in order to specifically stop from using response files for linker options, I suppose.
6. Since GCC 9 removed HAVE_GNU_LD usage from %o processing, we got response files for linker options with this ticket's problem. Note that its OS/2 override was also removed from gcc.c (as not used anywwhere).

An obvious fix is to bring HAVE_GNU_LD back into gcc.c. I really wonder why GCC devs removed. But in order to know that, I have to make a full clone of the GCC repo which is enormously big. Sigh...

From what I see, HAVE_GNU_LD is used in other GCC sources specifically to guard places dealing with response files. From what I get their logic is as follows: If the linker is not ld from binutils, it will most likely know nothing about libiberty response files so better not use them. This is a fair assumption and it worked really well in case of emxomfld which tries to mimic ld but doesn't use libiberty response files. In fact, it uses LIBC own response files (_response/-Zargs-resp) which, as pointed out above, is not compatible with libiberty ones.

Another fix to that problem would be to use libiberty instead of LIBC for response files in all EMX tools (emxomfld, emxomf, emxexp and alike) but this may break some makefiles so I'm not sure if we should do that. I will create a ticket though.

As a matter of fact, GCC for OS/2 itself has been always using libiberty response files so making it use LIBC ones may break some makefiles too. This is really so weird that gcc and the rest of EMX disagree here. Was certainly NOT a wise decision back there to create such a terrible mix.

Another problem is that the GCC version of libiberty disables using of backslash as an escape character when reading response files — for obvious reasons of it being a valid path separator under OS/2. However, it doesn't do so when writing response files. I can actually make it use quotes instead but this will not fix the problem as LIBC response file processing doesn't give any special treating to quotes.

So complicated.

[dryeo]

At some point, response files had to be added to calling emxomfld IIRC (might have been emxomfar) due to Mozilla builds breaking as using a pipe was running into a 64kb limit. IIRC, it was Yuri or Paul who added the support.

[dryeo]

The related bug, http://trac.netlabs.org/libc/ticket/216 though it doesn't show Yuri's work.

[dmik]

Thanks for the feedback, Dave. Although I really doubt that using response files in GCC when running subprograms was added because of this ticket (or because of the OS/2 limitation). Most likely the GCC folks needed it on Windows (mingw) or such. Anyway, this sheds some light on why they could do that.

But the disagreement between EMX tools and GCC wrt response files I mention above has nothing to do with this particular problem I suppose. I believe EMX tools have been using LIBC response files since the very beginning (and GCC has been using libiberty since its very beginning too).

However, I now think that changing EMX tools to use libiberty instead is pretty safe. I doubt there are any makefiles that manually create response files for those tools. And GCC will handle it nicely anyway. I will think on that more.

Just for the record, HAVE_GNU_LD was removed by this original commit https://github.com/gcc-mirror/gcc/commit/e7208389c8381feaf8c7c60d975b06c446978006 (original gitweb). No explanation in the commit itself on why it was done so. Perhaps, the commit author simply didn't know why it originally appeared there.

[dmik]

I'm reopening this as I've found the source of the problem. It's the following libiberty code:

https://github.com/bitwiseworks/gcc-os2/blob/7d9b429416ea417d73c6a5b1d472a09bea0681ee/libiberty/pex-djgpp.c#L218

In short: GCC starts driver processes (including emxomfld.exe) with the help of libiberty's pex_run_in_environment. And on EMX its worker detects if the size of arguments is going to be larger than 30K (DosExecPgm has a limit of 32K), and if so, creates a response file on its own.

However, GCC doesn't know about that and decides to create its own response file under some circumstances (see the comments above). As a result, emxomfld ends up with two response files: one created by libiberty and passed to it as its only argument (it has a name $TEMP/grspXXXX.tmp btw) and the other one created by GCC ($TEMP/xxxxxxxx) which is put, among all other arguments, into the libiberty's response file (i.e. a nested response file). Nested response files are not supported by LIBC (and by GCC/libiberty too IIRC). Hence, after calling LIBC's _response to resolve any possible response files, emxomfld ends up with @$/TMP/xxxxxxxx among its arguments (note the leading @!) which it treats as an input file. And, of course, _abspath fails on that file with "File not found" because it's in fact a non-existing file.

The solution here is to disable response file creation by GCC completely on EMX since this is done as needed by libiberty when starting child driver processes.