Strip or honor ASCII color coding

jessehouwing commented 6 years ago

Agent Version and Platform

2.133.3 Windows | Linux

VSTS Type and Version

VisualStudio.com

If VisualStudio.com, what is your account name? http://psd30.visualstudio.com

What's not working?

When a commandline tool issues ASCII colour coding information (e.g. [?25l) they are shown in the logs and make it harder to read.

2018-05-19T15:43:16.7926995Z It looks like this is your first time using Cypress: 2.1.0
2018-05-19T15:43:16.7938560Z 
2018-05-19T15:43:16.7958404Z [?25l[15:43:05]  Verifying Cypress can run /opt/vsts/_work/r1/a/xxx-CI/src/node_modules/cypress/dist/Cypress [started]
2018-05-19T15:43:16.7980465Z [15:43:06]  Verifying Cypress can run /opt/vsts/_work/r1/a/xxx-CI/src/node_modules/cypress/dist/Cypress [completed]
2018-05-19T15:43:16.8000804Z [?25h
2018-05-19T15:43:16.8019478Z Opening Cypress...

It would be useful to either honor the color coding, somehow instruct/configure the terminal as monochrome or stripping out the nonsense characters.

bryanmacfarlane commented 6 years ago

Yeah, this is on our backlog. labelling as an enhancement.

adamralph commented 5 years ago

Any update? As the maintainer of a build package which uses ANSI colours, I'm starting to get questions.

TingluoHuang commented 5 years ago

we are fixing this step by step.

get all UTF-8 characters back correctly #1975
the Web UI might already support color coding. :)

adamralph commented 5 years ago

Thanks for the info. AFAIK the web UI still doesn't support ANSI colour codes, see https://dev.azure.com/blairconrad/SendComics/_build/results?buildId=9&view=logs&jobId=011e1ec8-6569-5e69-4f06-baf193d1351e&taskId=b8cbf579-6a08-5e02-2788-e25b5bd52b1b&lineStart=13&lineEnd=14&colStart=1&colEnd=1

@blairconrad you reported https://github.com/adamralph/bullseye/issues/150 - are you still seeing this problem in the web UI?

blairconrad commented 5 years ago

@adamralph, I see no change. ANSI codes show up in the web UI:

https://dev.azure.com/blairconrad/SendComics/_build/results?buildId=13

yaananth commented 5 years ago

The web UI supports minimal set of escape codes, but the full support is coming soon. We are prioritizing this. Since this is UI issue and not agent issue, feel free to post something here https://developercommunity.visualstudio.com/spaces/21/visual-studio-team-services.html?type=idea

But we understand the request and are working on it.

yaananth commented 5 years ago

Changes should be rolling out. We support ANSII color code now. Here's what we support:

Graphics mode (m) - 0 (reset), 1 (bold), 3 (italic), 4 (underline), 22 (not bold), 23 (not italic), 24 (not underline), 39 (default fg), 49 (default bg),

fg colors - 30 (black), 31 (red), 32 (green), 33 (yellow), 34 (blue), 35 (magenta), 36 (cyan), 37 (white), 90 (grey) bg colors - 40 (black), 41 (red), 42 (green), 43 (yellow), 44 (blue), 45 (magenta), 46 (cyan), 47 (white)

TingluoHuang commented 5 years ago

Close this issue base on this colorful screenshot. :D

blairconrad commented 5 years ago

PRETTY!

blairconrad commented 5 years ago

Not to be a Negative Nelly, but I notice that while the completed build output looks nice and colourful, the in-progress output does not deal well with the ANSI codes:

(from the in-progress build of https://dev.azure.com/blairconrad/blair%20bullseye/_build/results?buildId=18)

yaananth commented 5 years ago

That's a known limitation now unfortunately, we don't use the same view for both views :( This is getting addressed though, just won't be super soon :)

blairconrad commented 5 years ago

Oh, thanks for the speedy response. Glad it's known!

blairconrad commented 5 years ago

I hate to complain, but as @adamralph noted over at https://github.com/adamralph/bullseye/pull/199#issuecomment-447370765, in some cases the combination of the ANSI escape codes and < or > in the output confuse the renderer. I have created a minimal reproduction which you can see at https://dev.azure.com/blairconrad/azure-ansi/_build/results?buildId=22.

You can see the HTML produced for the ANSI-less and ANSI-ful output here:

<div style="top:192px;height:16px;" class="view-line">
  <span><span class="mtk1">2018-12-14T19:46:38.8739531Z&nbsp;No&nbsp;colour</span></span>
</div>
<div style="top:208px;height:16px;" class="view-line">
  <span><span class="mtk1">2018-12-14T19:46:38.8741003Z&nbsp;&lt;options&gt;</span></span>
</div>
<div style="top:224px;height:16px;" class="view-line">
  <span><span class="mtk1">2018-12-14T19:46:38.8758154Z&nbsp;</span></span>
</div>
<div style="top:240px;height:16px;" class="view-line">
  <span><span class="mtk1">2018-12-14T19:46:38.8759918Z&nbsp;Yellow&nbsp;foreground</span></span>
</div>
<div style="top:256px;height:16px;" class="view-line">
  <span><span class="mtk1"><span>2018-12-14T19:46:38.8761200Z </span>&zwnj;<span class="ansifg-y "><options></options></span>&zwnj;</span></span>
</div>

It seems that once the ASNI codes are used, <options> is translated into <options></options> instead of <options>.

Is https://developercommunity.visualstudio.com/spaces/21/visual-studio-team-services.html?type=idea still the right place to mention this or is there a better place?

yaananth commented 5 years ago

Here are the known limitations with this feature -

Visually text won't appear selected if you try to select the text (but copy-paste works)
Live feed won't honor color codes
Color codes with < > tags will make them disappear

This is something we decided to live with until we address all of them together in the next spike which is planned soon. We decided to go ahead with the feature even with those as it's much better than showing some random useless characters :)

If you are curios on the limitations, we use monaco editor, which isn't really made for "viewing" experience. So to bring color coding support, we had to have those limitations for the interm...

Thanks for reporting and we understand the issues you mentioned and we plan to work on those, let us know if you find anything else.

For this particular issue, I think having it here makes sense as everything is being discussed here, but in general yes, dev community is the place to track these.

blairconrad commented 5 years ago

Thanks for the response @yaananth. I had doubts about https://developercommunity.visualstudio.com/spaces/21/visual-studio-team-services.html?type=idea as the "idea" at the end sounds more like feature requests, not bug reports.

I eagerly await the next release and appreciate all your efforts.

And I realize that every client is different, but given the choice between "bonus funny characters in the output" and "some sections of the output just omitted altogether", I'd've picked the second. The recovery mode of the first is "just try not to look at the funny characters" while for the second it's "hopefully realize something is wrong and then pore over the source HTML".

yaananth commented 5 years ago

You can report a problem: https://developercommunity.visualstudio.com/content/problem/post.html?space=21

adamralph commented 5 years ago

@yaananth earlier you said:

We decided to go ahead with the feature even with those...

So you have knowingly introduced a bug. Why do you need someone to "report" it?

olafurpg commented 5 years ago

Visually text won't appear selected if you try to select the text (but copy-paste works)

@yaananth Is there an open issue to fix this limitation? I searched in the Visual Studio Community but couldn't find a related discussion. Copy-paste works for in-progress builds but it stops working when the colors have rendered after the build completes. To reproduce, try copy-pasting line 4000 from the CmdLine entry in https://dev.azure.com/olafurpg/Metals/_build/results?buildId=51

sbachstein commented 4 years ago

Have there been any advances regarding the character encoding not being resolved?

yaananth commented 4 years ago

Multistage pipelines feature will bring new views which will honor ansii codes, evening when live streaming!

lindluni commented 4 years ago

While coloring is better with the multistage feature enabled, codes are still dumped in the output of the multistage build

yaananth commented 4 years ago

We support https://en.wikipedia.org/wiki/ANSI_escape_code#3/4_bit It's possible the tool is using other specifications, eg: https://en.wikipedia.org/wiki/ANSI_escape_code#8-bit

Which we don't support yet. However we should be parsing them out. So, looks like there's some bug there. Could you send a repo text file or link to where it happens (if it's a public run). Thanks!

lindluni commented 4 years ago

This is actually the output of a git diff --color

Here is the link, it is the second step labeled CmdLine. We are just getting started on our migration so we are noticing small nuances like this.

AZP Build 291

octogonz commented 4 years ago

(Not sure why this was closed 9 months ago, when the problem was clearly only partially solved.) But today I noticed that if you enable the "Multi-stage pipelines" preview feature, then colors are displayed properly for the realtime log output. (In Azure DevOps, click your profile --> "..." --> "Preview features".)

Hurray! 🎉

malkinfedor commented 4 years ago

@octogonz Yes, you're completely right, I've enabled this feature and now ansible output looks pretty good! Thansk for this advice!

elisherer commented 4 years ago

Issue is back on release (not build) logs

liamfoneill commented 4 years ago

This really is not a finished feature. Take a look at what 'Terraform Plan' output looks like without using the '-no-color' flag:

Apparently, Terraform uses 'VT100 escape codes' for colorizing output.

Please don't say just use multi-stage pipelines. That is not an option. There are still lots of things you can not yet do in YAML pipeline that can be done in 'Classic'.

Please prioritize this issue.

timotheecour commented 4 years ago

@TingluoHuang could you please re-open this issue, since it's clearly not fixed?

TingluoHuang commented 4 years ago

@vtbassmatt FYI.

mcollis2 commented 4 years ago

Yes we are seeing this issue too with terraform in Azure Devops classic pipelines. Would very much appreciate a fix on this as it worked 1-2 months ago.

jtpetty commented 4 years ago

@azureDaveOps - David, would this be something that your team would own the fix for?

scottboehmer commented 4 years ago

Adding @prativen since this looks to be impacting release logs and not build logs.

octogonz commented 4 years ago

For me, the PR build logs seem to render correctly with the "Multi-stage pipelines" preview feature enabled. But if that feature is disabled, then the color codes are garbled, making the logs unreadable. (Example log)

How long will "Multi-stage pipelines" be kept in preview? Lots of NodeJS tools want to print colors to their log output.

scottboehmer commented 4 years ago

We've started the process of making the multi-stage pipelines feature on-by-default, and I expect to be on everywhere with the option to go back to the old UI removed sometime in early 2020.

rene-fonseca commented 4 years ago

@yaananth, not sure if you prefer a separate issue but I'll try this one since it is still open.

[0m for resetting all attributes is not resetting foreground color - I didn't check if other attributes are reset but this would likely be the case.

Also noticed [4m for underscore and [3m for italic are ignored. So maybe there is a general problem for the lower m values. But per your earlier note these codes should be supported.

Example line: 2020-01-02T20:53:10.4177461Z THROW in [34m[1mclass base::MyOtherObject &__cdecl base::Reference::operator *[0m[34m[3m(void)[34m[1m[0m at [37m[1mbase/mem/Reference.h:342[0m

The [0m just before "at" above - should ensure "at" gets presented in default color - but for now it uses the color that is active before.

Full public log example is available at - I do use ANSI colors a lot and it is really nice the pipeline supports it :) https://dev.azure.com/renefonseca/base/_build/results?buildId=820&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=3d94a178-cdfb-5c90-a971-5deeccbbe043&l=51

Here is an working example from Visual Studio Code terminal (run on macOS) for comparison:

jamessampford commented 4 years ago

I'm using SSH in my pipeline to run a bash script and getting the codes appear as well (rather than the colours), example below:

2020-01-08T09:08:30.6597072Z [1;34m### RECEIVED BUILD 120 ###[0m

Tried a variety of escape and colour codes but all the same...

\x1b[1;34m
\x1b[34m
\033[1;34m
\033[34m

santiago-ld commented 4 years ago

I'm having the issue using terraform too, hope to see this fixed soon.

stephenmichaelf commented 4 years ago

@davekenn Could you route to the right person on your team to take care of this?

jz3xyz commented 4 years ago

I'm having the same issue with a bash script in a release pipeline.

I'm setting colors using

# Set colors
BOLD="\e[1m"
ITALIC="\e[3m"
RESET="\e[0m"
RED="\e[31m"
GREEN="\e[92m"
CYAN="\e[96m"

Ado likes to interpret them as is

[96m[1mUsage: ...[0m

thiagofborn commented 4 years ago

Hello there.

I hope you guys are having loads of fun. However, I need to ask...

Did you find any workaround for this?

edjwhho commented 4 years ago

Hi, anyone managed to get a work around for the terraform plan to output colours in azure pipeline?

octogonz commented 4 years ago

Today we were trying to make a copy of an Azure DevOps pipeline, but could not find the "clone" menu item. (Screenshots are shown here.)

After much frustration, @iclanton figured out that the "Multistage pipelines" preview removes that feature. 😕

So it seems, I have to manually enable "Multistage pipelines" to see ANSI colors in my build logs, but remember to turn it off when I want to clone a pipeline.

@scottboehmer was this intentional? What's the deal?

scottboehmer commented 4 years ago

Is your definition using a yaml file? If so, then it is intentional that clone is unsupported because the recommended pattern is to copy the yaml file rather than creating a cloned definition pointing at the same underlying file.

up2geek commented 4 years ago

I too would like to see a fix or workaround to allow terraform plan color coding output to display properly in ADO release pipeline log output.

octogonz commented 4 years ago

Is your definition using a yaml file? If so, then it is intentional that clone is unsupported because the recommended pattern is to copy the yaml file rather than creating a cloned definition pointing at the same underlying file.

@scottboehmer No, it is a non-YAML job. It's a manually invoked job, where we usually edit the steps before running it (e.g. changing which folders it runs in, or skipping certain steps). Thus creating a Git branch and YAML file would be a clumsy approach.

ppanyukov commented 4 years ago

I'm not 100% sure why it's so difficult to take raw log files and convert them to HTML and do ASCII colors? Here is a bash script that does it, this can be done even in bash. https://github.com/pixelb/scripts/blob/master/scripts/ansi2html.sh

It works for terraform output and git diff and whatnot.

EDIT: my question is to the product team

EDIT 2: so there are actually two issues here:

colors in builds
colors in releases

Builds -- WORKS WITH CLASSIC EDITOR

Releases -- DOES NOT WORK (using classic editor)

I used this script as a test: https://github.com/pixelb/scripts/blob/master/scripts/ansi_colours.sh

octogonz commented 4 years ago

Well, "Multi-stage pipelines" has been officially released. We can no longer workaround this problem by disabling the preview feature.

@scottboehmer What's the product team's position on:

providing a way to clone a classic (non-YAML) pipeline
properly supporting ASCII colors like every other tool does

It may be that these customer requests are not a priority and will never get fixed. In that case, please just say so and close the issue.

scottboehmer commented 4 years ago

As far as I'm aware, clone is working for non-yaml pipelines. On a sample project, I see it in the menu when on a definition's page: @octogonz do you not see that menu entry? Or does it fail when you try to use it? If it is a failure, could you share any error information logged in the browser's console.

I'm not sure on the status of extending ASCII color support on the logs page, so adding @vijayma and @jtpetty to see if they know of any plans for additional work.

octogonz commented 4 years ago

As far as I'm aware, clone is working for non-yaml pipelines. On a sample project, I see it in the menu when on a definition's page

You are right! I overlooked that. Consider the "Clone" item resolved then! 👍

Marcus-James-Adams commented 4 years ago

This issue thread seemed to get hijacked by cloning but can we have an update as to where we are with colours displaying as escape characters and other text corruptions in the release pipelines as currently, it makes them unreadable, especially for actions like terraform that use colours and characters to explain what changes there will be.

eg 2020-06-25T09:50:54.6379881Z [1m # azurerm_network_security_rule.rdp-access-utils-subnet[0m will be updated in-place[0m[0m 2020-06-25T09:50:54.6384644Z [0m [33m~[0m[0m resource "azurerm_network_security_rule" "rdp-access-utils-subnet" { 2020-06-25T09:50:54.6385107Z [1m[0maccess[0m[0m = "Allow" 2020-06-25T09:50:54.6398898Z [1m[0mdestination_address_prefix[0m[0m = "*"

microsoft / azure-pipelines-agent