search

KirillOsenkov / MSBuildStructuredLog

A logger for MSBuild that records a structured representation of executed targets, tasks, property and item values.
MIT License
1.47k stars 198 forks source link

Parallelism for secret search #832

Open KirillOsenkov opened 2 weeks ago

KirillOsenkov commented 2 weeks ago

One todo would be to investigate adding parallelism to secret detection because right now it's really slow on real-life binlogs, even moderately sized. Takes over two minutes on a 5 MB binlog I use often. I think a judicious use of Task.Run() will really make things better here. You can maybe spawn one task per string, and one task per file, then just foreach the tasks and do task.Wait() on each of them. No need for async as only one thread will be blocked (the one where the computation happens)

JanKrivanek commented 2 weeks ago

Once improved - what would you think about spinning it on background for all opens and then showing some form of info (node in the tree, goldbar, ...) if sensitive data found (except probably username)? It might feel too heavy - but preventing spilling secrets unintentionaly feels worth it.

KirillOsenkov commented 2 weeks ago

definitely too heavy, can't have it burn CPU and battery and interfere with the investigation. Needs to be an explicit gesture from user.

JanKrivanek commented 2 weeks ago

I'm sad about it - but it makes perfect sense

KirillOsenkov commented 2 weeks ago

also from looking at the current perf this seems like it will be very expensive computationally, we should get some data for larger binlogs (and I can share some large binlogs if you need them)