Feature request: run package tests in an environment that supports terminal colors

[bkamins]

As discussed in https://github.com/JuliaLang/julia/pull/41885#issuecomment-907440466 it seems that when nanosoldier is run it does not properly produce terminal colors when requesting it in sprint. The example shared there is:

Expression: sprint(show, df, context = :color => true) == "\e[1m2×2 DataFrame\e[0m\n\e[1m Row \e[0m│\e[1m Fish   \e[0m\e[1m Mass      \e[0m\n\e[1m     \e[0m│\e[90m String \e[0m\e[90m Float64?  \e[0m\n─────┼───────────────────\n   1 │ Suzy          1.5\n   2 │ Amir   \e[90m missing   \e[0m"
   Evaluated: "2×2 DataFrame\n Row │ Fish    Mass\n     │ String  Float64?\n─────┼───────────────────\n   1 │ Suzy          1.5\n   2 │ Amir    missing" == "\e[1m2×2 DataFrame\e[0m\n\e[1m Row \e[0m│\e[1m Fish   \e[0m\e[1m Mass      \e[0m\n\e[1m     \e[0m│\e[90m String \e[0m\e[90m Float64?  \e[0m\n─────┼───────────────────\n   1 │ Suzy          1.5\n   2 │ Amir   \e[90m missing   \e[0m"

and we can see that sprint(show, df, context = :color => true) did not produce output with color escape sequences. In Github Actions tests set up in DataFrames.jl GitHub CI these tests pass correctly.

[bkamins]

@DilumAluthge - pinging you, as I assume you have an understanding of what infrastructure is used here. With this problem unresolved you have an issue with DataFrames.jl that you have mentioned in https://github.com/JuliaStats/StatsBase.jl/issues/711 that you want to avoid in the packages. Thank you!

[DilumAluthge]

Hmmm. Is this specific to Nanosoldier? I am able to replicate the test failures using --color=no.

I've posted a full MWE here: https://github.com/JuliaData/DataFrames.jl/issues/2846

[bkamins]

Understood, but why then nanosoldier is run with --color=no? How can it then test if colors are produced correctly where they are required?

[DilumAluthge]

I don't believe that Nanosoldier/PkgEval is specifically run with --color=no. I think that PkgEval is essentially emulating some kind of terminal that does not support color.

[bkamins]

Ah - OK. Thank you!

[DilumAluthge]

I do think it would be a nice feature to have for NanoSoldier/PkgEval to support colors. I'm not sure how to implement that feature, though.

[bkamins]

@KristofferC - do you know who could know?

[maleadt]

Note that PkgEval now is running with --color=no to work around an issue in OutputCollectors.jl (since https://github.com/JuliaLang/julia/pull/41554#issuecomment-907611746). This wasn't the case when this issue was filed though. That said, to keep the NanosoldierLogs plain text and easy to render in the browser, I think it is a good default.

[bkamins]

The source of the issue was identified in https://github.com/KristofferC/Crayons.jl/issues/47. In DataFrames.jl we made a work-around till it is resolved.

[bkamins]

Feel free to close this issue if --color=no is a preferred option.

[maleadt]

For the sake of readable logs, I think we should stick to disabling colors for now. Feel free to re-open or file a new issue if anybody thinks otherwise.