search

chainguard-dev / malcontent

#supply #chain #attack #detection
Apache License 2.0
446 stars 31 forks source link

Add dynamic progress bar to improve scanning feedback #609

Closed egibs closed 5 days ago

egibs commented 1 week ago

I was thinking about how to address the feedback void between pressing Return and seeing the rendered file report(s). This void is especially long when scanning directories with large numbers of files.

Since rendering the file reports in a live manner such that they remain in a deterministic order is tricky due to concurrency, I instead landed on adding a progress bar which will update as a scan progresses.

This PR adds a simple, thread-safe progress bar that will update as each file or archive in a scan path is processed when running mal analyze or mal scan:

$ go run cmd/mal/mal.go analyze /usr/bin/
🔎 Scanning "/usr/bin/"
Processing discovered file(s)...  27% ▐▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒                    ▐ (441/1586) [6s:6s]

It's important to note that the bar will update across multiple lines if additional lines are written to stdout, so I moved several of our logger.Errorf statements in scan.go to .Warnf since we're returning the critical errors most of the time anyway. I also made this feature opt-out for use-cases where the progress bar will be rendered incorrectly (CI mostly).

tstromberg commented 1 week ago

I like this conceptually, but we should ensure that the solution doesn't impact our ability to address https://github.com/chainguard-dev/malcontent/issues/489 - which is a root cause for the long delay for results. The API was designed to stream results as soon as they were available - I think we lost that in the move to parallelism.

That said, I do think the progress bar is a great idea.

egibs commented 5 days ago

This is at odds with the re-adding of streaming output in the terminal for analyze/scan in #617.

It's possible that we can use third-party UI library to store the scrolling output in one container and then the progress bar in another container but that seems like a lot of effort for a simple progress bar.