ld seems to have an old PE stub

[stsp]

Hi.

I wasn't yet successful in creating PE executable with raw binutils (rather than with djgpp), but, unless I am mistaken, ld still embeds the old stub (the one w/o my patches). Only stubify tool has the new stub, so when using djgpp, the stub is good. But I believe creating PE with raw binutils will yield the old stub. I looked into an ld binary and I can see the old stub there because the error msg that I added, is not there.

How does ld embed the PE stub? Is it hard-coded somewhere in the sources? I mean, the one doesn't build djgpp libc when building the raw binutils. But the stub is there.

[jwt27]

Yes, the stub is hard-coded in ld. See binutils-gdb/bfd/go32stub.h. It's normally not a problem since all executables created by gcc go through stubify.

Not sure what you mean though - PE executables are for windows (or HX) and have no djgpp stub at all.

[stsp]

OK, it you think updating the stub is not needed, then lets close. I am just pondering around these tools and am trying to find the minimal set need to create the dos coff executable from an asm file, without involving djgpp gcc.

[stsp]

Btw how feasible is to convert an elf object to coff object? Current objcopy doesn't seem to be able to do that...

[stsp]

Not sure what you mean though - PE executables are for windows (or HX) and have no djgpp stub at all.

Indeed but the object files produced with djgpp are very similar to the object files produced with ld -mi386pe or with objcopy from elf to PE. I can even use nm from casual binutils with djgpp-created object files. Can I assume that only executables would be different, but the object files can be created from asm with raw binutils?

[stsp]

djgpp.tar.gz

Well, no, it is a bug. Attached is the test-case. Just run make and you'll get a perfectly working binary but with an old stub.

[jwt27]

Can I assume that only executables would be different, but the object files can be created from asm with raw binutils?

No, I don't think that's safe. I don't have an exact list of all the differences, but at least the section names and flags are incompatible. Compare the linker script templates for PE and go32. It may happen to work for a basic hello world with only .text/.data/.bss, but with anything more complicated you'll likely run into trouble.

Well, no, it is a bug. Attached is the test-case. Just run make and you'll get a perfectly working binary but with an old stub.

You'll get the new stub if you use:

DOS_LD ?= i386-pc-msdosdjgpp-gcc

That way you also don't need to link with crt0.o and libc.a manually.

You can even assemble prnt.S via gcc:

%.o : %.S
    $(DOS_CC) -c $< -o $@

It will do the right thing based on the file extension.

[stsp]

The idea was to avoid djgpp-gcc and build basic asm stuff with just djgpp-binutils that are in both debian and ubuntu.

[jwt27]

(moved to build-gcc)

The idea was to avoid djgpp-gcc and build basic asm stuff with just djgpp-binutils that are in both debian and ubuntu.

Okay, I see. I can patch the stub in the build scripts here if you want, but we can't send that patch to upstream binutils unless it is also merged in upstream libc. And since that doesn't seem to be happening (or maybe you could bump it again?), you'll have to ask @skitt to patch the Debian package too.

[stsp]

Can upstream ld be patched to load stub from a file? Seems much better than hard-coding.

[stsp]

Oh I guess this is even possible to do in a linker script?

[jwt27]

Can upstream ld be patched to load stub from a file? Seems much better than hard-coding.

It would need to figure out the path to stubify. Not sure how that's done, maybe you could see how gcc does it.

Oh I guess this is even possible to do in a linker script?

The stub isn't like a regular "section", so I don't think that's possible.

[stsp]

https://www.mail-archive.com/gcc-patches@gcc.gnu.org/msg125197.html gcc does it this way, which was patched in by djgpp devs. Finding path is not a problem, as it is sys-root/bin, but someone will need to add such a post-link step to an ld... I suspect it had to be added to ld initially, not to gcc.

[stsp]

Also I wonder if @tkchia can give us any hint, as I think his linker script details an exe file very verbosely. Is there really no way to embed a stub with a linker script?

[jwt27]

ia16-elf and coff-go32 use very different formats, so that's not comparable. The stub is only visible in the bfd target hooks, ld doesn't know it exists. It only sees a plain COFF image.

[stsp]

Well, for now I can run stubify by hands, as it seems to be correctly updating any existing stub. But definitely some solution on an ld side would be good.

[stsp]

Well, stubify is not in ubuntu, its in PPA. How about this: it might be possible to move stub.asm into binutils, and build it there instead of hard-coding a header. The excuse can always be that djgpp uses stubify so it won't be affected by any change. :)

[stsp]

Btw, why was stubify ever needed if ld already adds the stub? I don't understand.

[jwt27]

Well, stubify is not in ubuntu, its in PPA. How about this: it might be possible to move stub.asm into binutils, and build it there instead of hard-coding a header. The excuse can always be that djgpp uses stubify so it won't be affected by any change. :)

But the stub is assembled with djasm, and that's not part of binutils...

Btw, why was stubify ever needed if ld already adds the stub? I don't understand.

I don't know the exact history, but I expect ld originally produced only coff images, and they had to be stubified manually or by gcc.

[stsp]

stub.S.gz Here is a quick conversion of stub.asm to gas. Are you interested to integrate it into your tree?

[stsp]

The following things were uncertain during conversion: Commented out directives .copyright, .id, .start, .stack since I don't know what they do. .addrsize was replaced with addr16, .opsize was replaced with data16. .code16 was put to the beginning of the file, and .code32 was put after switch to PM, not sure if that was right or needed.

The rest was quite simple to convert.

[jwt27]

No, I don't think that's a good solution.

Either we patch go32stub.h here in the build scripts, and then the Debian package would have to be updated similarly. Or, you try again to get the stub patch accepted in djgpp, then we can update go32stub.h upstream. Or, we find a way to call stubify from bfd to generate a stub for ld. Or, just don't bother, and link with gcc or call stubify manually. Is there any realistic scenario where you have only binutils available and not the rest of the toolchain?

[stsp]

Calling stubify from ld is not an option: you'll need then stubify in a binutils package, so how to build the stub for it, will then be the open question. Does ld hard-code all other stubs in a blob header?

Is there any realistic scenario where you have only binutils available and not the rest of the toolchain?

Current debian/ubuntu have only binutils, yes.

[stsp]

Maybe having an extra package like binutils-stub-djgpp and having binutils-djgpp to depend on it, would be a possibility to load the stub from file.

[tkchia]

Hello @jwt27, hello @stsp,

Perhaps for now we can update go32stub.h manually, but also add makefile rules to allow developers to regenerate it,

• if they want to (further) change it
• and they happen to have the needed tools (djasm and whatnot).

It seems that a lot of files in the rest of the Binutils source tree itself are already handled in this way anyway. E.g. bfd/bfd-in2.h says

/* DO NOT EDIT!  -*- buffer-read-only: t -*-  This file is automatically
   generated from "bfd-in.h", "init.c", "opncls.c", "libbfd.c",
   "bfdio.c", "bfdwin.c", "section.c", "archures.c", "reloc.c",
   "syms.c", "bfd.c", "archive.c", "corefile.c", "targets.c", "format.c",
   "linker.c", "simple.c" and "compress.c".
   Run "make headers" in your build bfd/ to regenerate.  */

Thank you!

[stsp]

Perhaps for now we can update go32stub.h manually

This is exactly the problem because stub sources in various djlibc forks diverged. And you can't easily submit any patch to an official repo as they are no longer accepted (or, more precisely, there are trolls in an ML who do everything for shutting down every possible contributor). This is why some "simplified" strategy is needed, after having which, we could eg just ask @skitt to include some patch and regenerate/rebuild.

I fully agree with what you wrote about the possibility of re-generating. This means my gas conversion is not a loose-end, but rather is one of the possible solutions.

[stsp]

Of course we could as well ask @skitt to change the blob, but that's a bit "impolite". No one wants to take the blob patches, I am afraid. Even if we could get our patches into the main djlibc tree, it would likely still be problematic to submit a new binary blob to binutils. They will likely only take one from DJ himself. Overall, I fear that blobs are exceptionally problematic to update. So its better to avoid.

[skitt]

I wouldn’t mind patching the blob in binutils-djgpp, assuming the old stub is never needed (which I think is correct).

[stsp]

OK, lets do just that then. @jwt27 indicated that the patch needs to be upstreamed to djlibc first, in order to get into binutils upstream. But since you are not binutils upstream, I suppose upstreaming to djlibc is not a requirement here.

[skitt]

I just need to know which version of stub.asm is the correct one, I can take care of the rest. I’ve updated binutils locally with the stub.asm from djlsr205.zip, is that the desired version?

[stsp]

I think this is the good one: https://github.com/jwt27/djgpp-cvs/blob/jwt27/src/stub/stub.asm But we can wait for the repo owner to tell.

[skitt]

Ah yes, with your empty environment patch.

[jwt27]

I think this is the good one: https://github.com/jwt27/djgpp-cvs/blob/jwt27/src/stub/stub.asm But we can wait for the repo owner to tell.

Yep, that is the one we want. Just pushed a commit to add the patch (not yet tested).

@skitt, you can use the following:

https://github.com/jwt27/build-gcc/blob/master/patch/binutils-2.39/go32stub.patch

This is exactly the problem because stub sources in various djlibc forks diverged. And you can't easily submit any patch to an official repo as they are no longer accepted (or, more precisely, there are trolls in an ML who do everything for shutting down every possible contributor).

I think you're exaggerating the problem a bit here. There is only one fork where the stub code has diverged from mainline, and it's ours. Also I don't recall any serious objections to your patch on the djgpp list, more likely it was just forgotten about.

Perhaps for now we can update go32stub.h manually, but also add makefile rules to allow developers to regenerate it,

• if they want to (further) change it
• and they happen to have the needed tools (djasm and whatnot).

It seems that a lot of files in the rest of the Binutils source tree itself are already handled in this way anyway. E.g. bfd/bfd-in2.h says

Hi @tkchia, thanks for chiming in.

I think this solution is more trouble than it's worth though. The stub code very rarely changes - the blob in bfd was last updated 22 years ago, and the source in libc has only seen version number bumps and one minor size optimization since then.

Regenerating the blob from the binutils source tree would introduce a dependency on the libc code and utilities (djasm) for everyone who wants to use --enable-targets=all. I don't think that will be accepted there.

[stsp]

Also I don't recall any serious objections to your patch on the djgpp list, more likely it was just forgotten about.

You are right. I refreshed my memory by digging the archives, and it appears I sent 2 patches, and was trolled out with another one. So indeed there is a possibility to remind about this one, and maybe eventually I'll re-subscribe to do so. Do they still post their FAQ weekly? And do they accept any changes, eg found this report: https://www.delorie.com/djgpp/mail-archives/browse.cgi?p=djgpp-workers/2022/07/31 Did it generate some patch?

Regenerating the blob from the binutils source tree would introduce a dependency on the libc code and utilities (djasm)

Which is why I will keep playing around my gas port as time permits. Eventually I think that should became the right solution to this problem.

[jwt27]

Do they still post their FAQ weekly?

It's monthly now, I believe you suggested to change it.

And do they accept any changes, eg found this report: https://www.delorie.com/djgpp/mail-archives/browse.cgi?p=djgpp-workers/2022/07/31 Did it generate some patch?

Yes, that was fixed within a few days. (The rest of that discussion is on the djgpp@ list)

Which is why I will keep playing around my gas port as time permits. Eventually I think that should became the right solution to this problem.

Ah, so you'd propose to move the stub entirely to binutils, and make stubify obsolete. That makes sense.

[stsp]

It's monthly now, I believe you suggested to change it.

Yeah but back then it didn't yield any useful results. Good that it eventually did.

Yes, that was fixed within a few days. (The rest of that discussion is on the djgpp@ list)

OK good to know, so I'll need to re-subscribe eventually.

Ah, so you'd propose to move the stub entirely to binutils, and make stubify obsolete. That makes sense.

Cool so would you like to participate and take up my stub.S mod? :) Its not like I am going to do the upstreaming work very soon, but the conversion is here. So you could just pick it up. :) I see it in 4 steps: first switch in djlibc, then then upstream into binutils, then remove from djlibc and make stubify obsolete.

[jwt27]

I can't write the whole patch for you, but I'm willing to help out if you have some proof of concept.

I do see an issue though. You need an x86 assembler to produce the stub in bfd, but the assembler depends on bfd to build.

[stsp]

What do you mean by the "whole patch"? For now I am talking about just a first step, i.e. put it into djlibc. For that, not much of a patch is needed but to remove djasm from a makefile.

You need an x86 assembler to produce the stub in bfd, but the assembler depends on bfd to build.

According to what @tkchia wrote above, this shouldn't be a problem. We just need a script to re-generate the stub header. The stub header itself is also in a repo, so it can be used w/o working as.

[skitt]

I’ve uploaded a new binutils-djgpp package with the fixed stub. It should be in Ubuntu devel within a few days.

[stsp]

Thanks.

[jwt27]

What do you mean by the "whole patch"? For now I am talking about just a first step, i.e. put it into djlibc. For that, not much of a patch is needed but to remove djasm from a makefile.

I think you should first see if your idea is even feasible on the binutils side, before removing stuff in libc...

According to what @tkchia wrote above, this shouldn't be a problem. We just need a script to re-generate the stub header. The stub header itself is also in a repo, so it can be used w/o working as.

So you intend to build the assembler twice, just to generate one header?

[stsp]

before removing stuff in libc...

I do not propose to remove anything from libc. I meant, remove djasm from a makefile, replacing it with gas. Not much of a removal. :)

So you intend to build the assembler twice, just to generate one header?

No. @tkchia says this:

Perhaps for now we can update go32stub.h manually, but also add makefile rules to allow developers to regenerate it,

Obviously for such rule to exist, you need an ability to build stub with gas. And obviously when you execute such rule, gas is built. But you don't execute that rule when building binutils.

[jwt27]

before removing stuff in libc...

I do not propose to remove anything from libc. I meant, remove djasm from a makefile, replacing it with gas. Not much of a removal. :)

Have you confirmed that your port produces the same code as djasm?

So you intend to build the assembler twice, just to generate one header?

No. @tkchia says this:

Perhaps for now we can update go32stub.h manually, but also add makefile rules to allow developers to regenerate it,

Obviously for such rule to exist, you need an ability to build stub with gas. And obviously when you execute such rule, gas is built. But you don't execute that rule when building binutils.

In that case you might as well assume that the developer has djasm installed, and use that. But frankly, I don't really see a problem with keeping the stub in libc, and updating the one in binutils once every 20+ years.

[stsp]

Have you confirmed that your port produces the same code as djasm?

Unfortunately not yet. I am eventually planning to get to this, but that's where I could get some help.

In that case you might as well assume that the developer has djasm installed, and use that.

This is very unlikely as it is only available from your PPA and a few other (unpopular) sources. The last thing binutils devs need to install, is a djgpp suit. :) Given that gas port already exist (in yet an unverified state, yes), I wouldn't bother proposing them a djasm-based solution.

But frankly, I don't really see a problem with keeping the stub in libc, and updating the one in binutils once every 20+ years.

Well, you don't see the problem, but the fact is, this isn't done. Patch not upstreamed to libc, stub not updated in binutils. Yes, good that @skitt agreed to work as a "short-cut" for that problem. That's at least a partial relief.

[stsp]

stub.S.gz

Have you confirmed that your port produces the same code as djasm?

Actually it doesn't work at all. Attached an updated version. It appears i586-pc-msdosdjgpp-as wrongly handles .align directive. While in gas it means "align to N amount of bytes", in djgpp-as it means "align to 2^N amount of bytes", which is more than strange. You can see that by compiling this newly attached stub with as -m32 and with i586-pc-msdosdjgpp-as. In the later case you'll get this:

stub.S: Assembler messages:
stub.S:847: Warning: alignment too large: 31 assumed
stub.S:142: Error: value of 00010000 too large for field of 2 bytes at 0000008d

Error is because of .align 16 at line 971. Warning is because of .align 512 at the line it points to. So... what's that? Its obviously not a "bug" because it even truncates the value to 31, so it perfectly knows its going to use the power of 2. Is the djgpp-gas somehow (mis)configured to treat .align in this incompatible way?

[jwt27]

The behavior of .align is platform dependent, and for djgpp it happens to align to a power of 2 (same as .p2align). To align to an arbitrary boundary, there is .balign.

btw so you're aware, I sent a stub patch to the djgpp list yesterday (unrelated to yours). If that gets accepted you may have to rebase your patches.

[stsp]

https://www.delorie.com/djgpp/mail-archives/browse.cgi?p=djgpp/2022/11/23/14:50:26 You mean this? But how does this happen that it runs only per every even start? Does hdpmi returns different statuses?

[jwt27]

It loads hdpmi, returns non-zero, then the stub thinks it's failed. Then on the next run, hdpmi is still loaded, so int 2f/1687 succeeds.

[stsp]

And then hdpmi unloads?

[jwt27]

Yes, it only stays resident if you run it with -r.

[stsp]

Then on the next run, hdpmi is still loaded

So w/o -r it stays resident, but only until last client terminates? Convoluted. :)