Currently ld uses mmap to read in the contents of the response file (presumably for performance reasons), but mmap sometimes doesn't work on special files like pipes. For example, if you do something like:
$ ld @<(echo --help)
… that will currently fail with a segmentation fault.
You might wonder why one would want to generate a response file from process substitution. The rationale behind this is that I'm currently in the process of fixing a longstanding issue with the linker failing in Nixpkgs on macOS due to hitting command-line length limits and the fix entails the use of process substitution to generate the process file.
Specifically, what I was doing was building upon the work from this PR:
… which modified the cc-wrapper in Nixpkgs to use a response file generate from process substitution. I was going to do essentially the same for the ld-wrapper in Nixpkgs, but that failed with a segmentation fault (for the reasons outlined above).
There are other possible ways to work around that, but using process substitution is the "leanest" way of generating the response file for ld in Nixpkgs, so I wanted to push on getting that working here instead of working around the problem downstream.
So the way I fixed it was to fall back to using read instead of mmap if the mmap failed. After this change, the above sample command now works correctly.
This also fixes another small issue along the way: this now correctly detects when the mmap fails. Previously, the mmap logic was detecting failure by looking for a NULL/0 return value, but that is not the correct error-handling behavior. mmap returns MAP_FAILED on failure, which is -1 in practice, and not 0. That's the reason why the code was failing with a segmentation fault before because it wasn't detecting the failure and proceeding to read from the invalid buffer anyway.
Currently
ldusesmmapto read in the contents of the response file (presumably for performance reasons), butmmapsometimes doesn't work on special files like pipes. For example, if you do something like:… that will currently fail with a segmentation fault.
You might wonder why one would want to generate a response file from process substitution. The rationale behind this is that I'm currently in the process of fixing a longstanding issue with the linker failing in Nixpkgs on macOS due to hitting command-line length limits and the fix entails the use of process substitution to generate the process file.
Specifically, what I was doing was building upon the work from this PR:
https://github.com/NixOS/nixpkgs/pull/112449
… which modified the
cc-wrapperin Nixpkgs to use a response file generate from process substitution. I was going to do essentially the same for theld-wrapperin Nixpkgs, but that failed with a segmentation fault (for the reasons outlined above).There are other possible ways to work around that, but using process substitution is the "leanest" way of generating the response file for
ldin Nixpkgs, so I wanted to push on getting that working here instead of working around the problem downstream.So the way I fixed it was to fall back to using
readinstead ofmmapif themmapfailed. After this change, the above sample command now works correctly.This also fixes another small issue along the way: this now correctly detects when the
mmapfails. Previously, themmaplogic was detecting failure by looking for aNULL/0return value, but that is not the correct error-handling behavior.mmapreturnsMAP_FAILEDon failure, which is-1in practice, and not0. That's the reason why the code was failing with a segmentation fault before because it wasn't detecting the failure and proceeding to read from the invalid buffer anyway.