Make PDB builds reproducible

[jasonwhite]

Currently, PDB files are not patched to remove non-determinism.

This presents a larger challenge than just patching the PE file alone, because:

1. The PDB file format is not well documented.
2. The size of the PDB file seems to change with every build. Thus, we cannot use a memory map to make modifications to the PDB file as we will need to remove data.

References:

1. https://github.com/Microsoft/microsoft-pdb

[jasonwhite]

Fixed with f4241a14ad593ebc70de9311036255e6cd07a795.

[l0calh05t]

I believe this issue has to be reopened, because even the most trivial program (empty main) produces non-deterministic PDBs (with VS2015):

@setlocal
@echo off

call "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" amd64 || exit /b 1
cl
echo.
ducible --version
echo.

echo int main(int, char**) { return 0; }> main.cpp

for %%i in (1,2) do (
    cl /nologo /Zi main.cpp /link /incremental:no >NUL || exit /b 1
    ducible main.exe main.pdb >NUL || exit /b 1
    move main.exe main.%%i.exe >NUL || exit /b 1
    move main.pdb main.%%i.pdb >NUL || exit /b 1
    del vc140.pdb || exit /b 1
    del main.obj || exit /b 1
)

fc main.1.exe main.2.exe >NUL 2>&1 && echo EXE identical || ( echo EXE different && exit /b 1)
fc main.1.pdb main.2.pdb >NUL 2>&1 && echo PDB identical || ( echo PDB different && exit /b 1)

Results in the following output:

Microsoft (R) C/C++ Optimizing Compiler Version 19.00.24215.1 for x64 Copyright (C) Microsoft Corporation. All rights reserved.

usage: cl [ option... ] filename... [ /link linkoption... ]

ducible version v1.2.1-2535ce5

EXE identical PDB different

[jasonwhite]

@l0calh05t I can reproduce that problem with the test suite with VS2017 (I don't have VS2015 installed currently). I guess getting reproducible PDBs is a moving target. It looks like the /Zi flag is causing this. Using /Z7 instead appears to be fine.

I probably won't have time to dig into this in the near future, but from glancing at the hexdump it looks like there are some source file paths in the PDB that aren't in sorted order. If anyone else has the gusto to fix this, I'll happily take a PR for it. :wink:

[l0calh05t]

I‘m fairly certain that I also tried /Z7 to no avail. I’ll retest tomorrow.

[l0calh05t]

I just re-checked. And /Z7 also results in differing PDBs with VS2015. However, if I compare the pdbdump --verbose outputs of the two builds they are identical for /Z7 even though the binaries differ. This isn't the case for /Zi where both binary and dump differ.