search

buildkite / terminal-to-html

Converts arbitrary shell output (with ANSI) into beautifully rendered HTML
http://buildkite.github.io/terminal-to-html
MIT License
642 stars 45 forks source link

Garbled output from Gradle #143

Closed bpoland closed 1 month ago

bpoland commented 2 months ago

Hey, we're seeing very infrequently that Gradle will mangle output, I assume due to some weird control characters. Here's an example:

image

It seems like this could be related to some previous discussions:

I also found this Gradle issue where the output looks similar but I'm not sure if it's related https://github.com/gradle/gradle/issues/8204

One of the more annoying parts of this issue is that when it happens, it corrupts all the output in the step/job and isn't contained to a single log group. Is there a way that terminal-to-html could prevent control characters from jumping across log groups above it somehow? It seems unlikely/unexpected that you'd want to go backwards and change log groups above.

Also interested in anyone's thoughts about what could be causing this. I have only seen it happen a few times out of hundreds of builds and I don't know what's different when it does happen...

DrJosh9000 commented 2 months ago

Hi again @bpoland ! That console output does look funky. Have you tried gradle --console=plain as a workaround?

Is there a way that terminal-to-html could prevent control characters from jumping across log groups above it somehow? It seems unlikely/unexpected that you'd want to go backwards and change log groups above.

That's a good idea. terminal-to-html itself isn't aware of log groupings, but perhaps we could make it aware.

Also interested in anyone's thoughts about what could be causing this

Looking through the Gradle bug you linked, it could possibly be related to how the agent doesn't set a window size on the PTY. It'll need more investigating.

bpoland commented 2 months ago

Hi again @bpoland ! That console output does look funky. Have you tried gradle --console=plain as a workaround?

Yeah console=plain does work but the richer output is a lot more useful when it does work. It seems like a shame to turn it off just for the 1% or less of runs where the output gets mangled 😢

DrJosh9000 commented 1 month ago

I have a theory why this is happening only very occasionally, and it relates to some other weird output in (other customers') jobs. There's a chance this is an agent bug (that I added 😣) to do with timestamp insertion.

DrJosh9000 commented 1 month ago

I've just released v3.73.0 of the agent (and v0.11.0 of the k8s stack) with that fix - give it a whirl and let me know if you see more garbled logs!

bpoland commented 1 month ago

Thanks, just to follow up here -- I haven't seen any garbled output since updating, but it was also quite rare anyway :)

I'm good to close this and reopen later if I see it again?

DrJosh9000 commented 1 month ago

Absolutely, let us know if you see it again!

bpoland commented 1 month ago

@DrJosh9000 I just came across a small garble in the output, much less serious than I've seen previously. This is the only time I've seen anything weird since updating. The 77% CONFIGURING [59s] and > IDLE lines are left over from when Gradle was running, and don't normally show up after it finishes.

image

So it's possible there is still something a little off, or could very well be a Gradle bug too haha. I don't think this is worthy of reopening this issue since it's pretty minor but just thought I'd share -- thanks again for the help here!

bpoland commented 5 days ago

Just to circle back here again, unfortunately we are still seeing occasional issues with Gradle output getting messed up, similar to my last screenshot :(

I think the fix here definitely improved things as it no longer seems to jump backwards and trample output from previous log groups though! So a minor issue now, but still a bit annoying...