Consider integrating Chrome DevTools protocol

[hamiltont]

Love the tool, but I'm running into an edge case that might only be solvable by a big hammer. Might be overkill for this project, but I wanted to propose just in case :-)

I'm trying to capture screenshots from go tool trace -http. Problem is, these pages can take up to 15 seconds to properly load. I've tried using --timeout 15 --chrome-time-budget 15 --chrome-timeout 15. None of those help - Chrome considered the page loaded, and AFAIK there is no method to simply say "please wait 10 seconds before you take the screenshot, even if the page reports it is loaded before then".

A google search leads me here, where a user has a similar issue. I looked into the two recommended solutions. Both internally utilize the Chrome DevTools Protocol, which allows you to connect to a headless instance and communicate with it. As would be expected, this allows a lot more power than passing flags, e.g. you wait for DOM events, or screenshot specific page elements , or otherwise engage with the page. The more developed project is https://github.com/puppeteer/puppeteer, which is a Google project for programmatically interacting with headless Chromium.

There is already a golang project that speaks the DevTools protocol - https://github.com/chromedp/chromedp. However, it seems focused on being a golang version of puppeteer - intended for programmatic usage. For example, here is their screenshot source code. This project is clean and simply focused on getting a screenshot.

Would you be open to PRs that attempt to utilize chromedp in some circumstances? Not quite sure the proper combination of flags to indicate when it should be used, that would need to be discussed. Perhaps some "wait X seconds after specified DOM event" approach?

Misc note: The API call for taking a screenshot using DevTools is in the 1.3 release candidate, whereas 1.2 is the current stable. Doubt it's an issue, as clearly puppeteer finds it OK to expose features from the RC protocol version, but thought it was worth mentioning regardless.

[leonjza]

Exploring more avenues for taking screenshots would absolutely be awesome! Lets experiment :)

[hamiltont]

Great! FWIW, I was able to prototype my use case (auto-capture screenshot of execution trace during a CI/CD pipeline) in about 15 minutes. Relevant code at https://github.com/hamiltont/trace-grok/blob/master/harness.go#L121 shows that IMO it's really easy to use this chromedp project. The project documentation is really good. It does require the google-chrome binary to be available on the PATH (not sure if gowitness requires that already)

The added power of this tool would also allow a lot of new capabilities. In addition to my requested capability (a timer that only starts when the page is rendered), we could add support for partial screenshots (e..g --screenshot-id "#sidebar" --screenshot-query "body > overlay .buynow" --screenshot-frame <some frame id>) which is kind of neat.

[randomactsofsecurity]

Going to experiment with using chromedp in gowitness and will let you know how it goes, hopefully it'll clear up the blank screenshot issues we run across

[hamiltont]

@randomactsofsecurity awesome! Please feel free to reuse any code of mine in your attempt, I've no claim to it, just never got around to making a proper PR

[leonjza]

Hah! @randomactsofsecurity I have ported the screenshot logic to use chromedp this week and it seems to work fine. I am actually busy refactoring large parts of the code for what might end up being v2 of gowitness. Should see some commits land soon™

[randomactsofsecurity]

Sweet sounds great, looking forward to testing it out!

[leonjza]

V2 just landed using chromedp!