Binlog "censoring" rewriter that strips secrets

[KirillOsenkov]

It should be relatively easy to write a censoring tool that rewrites a binlog to omit certain strings.

Need to think about whether to manually specify strings to exclude or use some heuristic, such as Environment variables.

Related: https://github.com/dotnet/msbuild/issues/3432 - only include environment variables actually used during the build

[KirillOsenkov]

Perhaps we should go over all environment variables and delete the ones that were never mentioned anywhere

[KirillOsenkov]

See also: https://github.com/dotnet/msbuild/issues/3603

[aolszowka]

I assume this is related to #191 or is it slightly different?

[KirillOsenkov]

Oh, yeah ;) I forgot we have a dupe already

[MeikTranel]

we should be very careful about stripping things that dont show up in binary logger events - example: Embarcadero Delphi uses BDS as an environment variable to resolve the compiler dll. It does this internally inside the MSBuild task dll - of course this is a trainwreck for many reasons in of it self but its something i cannot change. If we were to silently remove BDS because its never mentioned inside a task or reused for constructing another value i would not know what version of the compiler was used. In reality BDS is used as a property as well but the general point still stands - i think we should make sure that if something like this would drop we do our best to offer a frontend that does two things:

• Provide a summary of removed entries
• Offer to clean in manual mode where one provides a filter themselves

[KirillOsenkov]

Consider running through a bunch of regexes like here: https://github.com/rodrigoramosrs/fileleaks/blob/main/src/FileLeaks.CLI/regex/all.json

[KirillOsenkov]

A prototype was implemented by @JanKrivanek: https://github.com/KirillOsenkov/MSBuildStructuredLog/pull/711

[KirillOsenkov]

I’ll close this issue as it’s now implemented and we can file new bugs for remaining issues