Open acastaner opened 7 years ago
Hi! Those are console color control sequences. They're supposed to be automatically adapted into control commands for the Windows console.
Can you try running this command to check which platform beets believes it's running on?
python -c 'import sys; print(sys.platform)'
And this one to check whether Colorama, which provides colors on Windows, was successfully installed?
python -c 'import colorama; colorama.init()'
If Colorama is missing, you can install it with pip install colorama
. But it should have been automatically installed when you installed beets, so that remains a mystery.
In the mean time, you can make the output more readable by setting color: no
.
Apparently cmd.exe doesn't like these commands :/
C:\Users\Arnaud>python -c 'import sys; print(sys.platform)'
File "<string>", line 1
'import
^
SyntaxError: EOL while scanning string literal
Disabling the colors works as a workaround though!
Sorry; I'm very far from being an expert in Windows command syntax. We just need a way to escape the spaces in that argument—maybe try double quotes instead of single quotes?
No worries :)
The output of the first command is win32
. The outpout of the second command is blank but colorama is apparently already installed:
C:\Users\Arnaud>pip install colorama Requirement already satisfied: colorama in c:\users\arnaud\appdata\local\programs\python\python36-32\lib\site-packages
Note that within PowerShell there's no such error (it interprets the error correctly) ; but I tend to use cmd because PS has issues parsing the directory parameter (that's for another issue ;) )
That's strange. All this indicates that everything should be working.
Here's the relevant bit of the code: https://github.com/beetbox/beets/blob/7e8056b0e8652a6235d8e2e054c0c41eae71bff0/beets/ui/__init__.py#L47-L54
If you're interested in doing some more hardcore digging, the next step would be to sprinkle in some print()
calls around there to check exactly what's going on. Something like this:
if sys.platform == 'win32':
try:
print('importing')
import colorama
print('imported')
except ImportError:
print('import error')
pass
else:
print('initializing')
colorama.init()
print('initialized')
Seems silly, I know, but that would tell us exactly what your machine is doing.
Doesn't seem silly at all, I'll try this and let you know.
Tried the code snipped, and colorama seems to initialize fine?
E:\Music\blues\Muddy Waters>beet import "1989 - The Chess Box"
importing
imported
initializing
initialized
But then the rest of the import still shows the color control sequences.
Extremely mysterious! I took a look at the issues for Colorama and it sounds like there may be a problem with Colorama on Python 3 and writing raw bytes (which the only way we can control the output encoding). We probably have essentially the same issue as the one described here: https://github.com/tartley/colorama/issues/125#issuecomment-291140235
It looks like we'll need a different tactic. The problem, FWIW, is that this printing stanza: https://github.com/beetbox/beets/blob/6185c6ad4ba689ed438c2075699eb12b24ef5daf/beets/ui/__init__.py#L150-L152
is bypassing Colorama's translation by using sys.stdout.buffer
to write bytes directly. Apparently, Colorama assumes that you only ever use sys.stdout
directly. To control the encoding, we can't do that. (As an aside, I believe this is an important design flaw in Python 3—two steps forward, one step back.)
We'll need to look around for a way to coax Colorama into working on bytes on Python 3, in the same way as it does on Python 2. :confused: I'm not exactly sure how we'll manage this, but I'm marking this as a bug so we can look a little more closely.
No problem, let me know if I can help further.
FWIW, I gave up on using cmd.exe and moved to Powershell.
Same here, moved to PS, unusable in standard Windows 10 commandline.
I'm running into issues when using PS. It somehow can't find the specified directories:
PS C:\Users\Arnaud\Desktop> beet import '.\Overkill - 1989 - The Years of Decay (Megaforce) V0\' --set savedir="metal" error: no such file or directory: .\Overkill - 1989 - The Years of Decay (Megaforce) V0" --set savedir=metal
I'm not sure why it's doing this (and it's probably not related to beets) and that's why I tend to use cmd.exe instead.
Wow; that's pretty weird. It seems like single quotes should work as you'd expect… https://msdn.microsoft.com/en-us/powershell/reference/5.1/microsoft.powershell.core/about/about_quoting_rules
It sounds like it may be worth tracking tartley/colorama/issues/21, as that will fix this.
Good find, @jackwilsdon!
Yes, it does look like we either need to wait for that to get fixed, or we need to find a way to manually send bytes through the library.
FWIW, I can't get it to work anymore in Powershell either (maybe a Windows update broke it?). But it does work with Cmder Mini. http://cmder.net/
You can fix this issue in cmd.exe using ANSICON. Still doesn't fix the issue in powershell though.
http://softkube.com/blog/ansi-command-line-colors-under-windows
Apparently using conEmu works well according to what this posted on discourse (see comment number 9):
I can confirm that I have the same issue using the command prompt in windows 10 but not using ConEmu.
tl;dr same issue in powershell (not specific to cmd.exe), here's my setup.
See the last part
PS> beet -v import ".\Quasi -- R&B Transmogrification"
no user configuration found at C:\Users\user1\AppData\Roaming\beets\config.yaml
data directory: C:\Users\user1\AppData\Roaming\beets
plugin paths:
Sending event: pluginload
library database: C:\Users\user1\AppData\Roaming\beets\library.db
library directory: C:\Users\user1\Music
Sending event: library_opened
Sending event: import_begin
state file could not be read: [Errno 2] No such file or directory: 'C:\\Users\\user1\\AppData\\Roaming\\beets\\state.pickle'
Sending event: import_task_created
Sending event: import_task_start
Looking up: C:\Temp\Quasi -- R&B Transmogrification
Tagging -
No album ID found.
Search terms: -
Album might be VA: True
Evaluating 0 candidates.
C:\Temp\Quasi -- R&B Transmogrification (14 items)
Sending event: before_choose_candidate
No matching release found for 14 tracks.
For help, see: http://beets.readthedocs.org/en/latest/faq.html#nomatch
[36;01m[S][39;49;00mkip, [34;01mU[39;49;00mse as-is, as [34;01mT[39;49;00mracks, [34;01mG[39;49;00mroup albums, [34;01mE[39;49;00mnter search, enter [34;01mI[39;49;00md, a[34;01mB[39;49;00mort?
Possibly related to tartley/colorama/issues/48 (bettter captured by tartley/colorama/issues/79). Very likely other tartley/colorama issues, too.
I resolved this issue by enabling ANSI Terminal Control in the registry. I used Glenn Slayden solution. https://superuser.com/questions/413073/windows-console-with-ansi-colors-handling
The problem seems to be in the print_
function that is beeing used to avoid exceptions by the default print
function.
On windows we rely on colorama to convert ANSI color codes to win32 calls during encoding. Encoding the string directly seems to bypass that and break colorized output.
simply writing the text to stdout or even just passing it to the default print function would fix colors on windows however that would also throw encoding errors and break the encoding overwrite described in that function.
Yep, that’s a perfect summary. This thing about exceptions is still a sort of surprising downside of the Python 3 transition.
Windows ANSI support can also be enabled at runtime: https://github.com/mesonbuild/meson/blob/c53b6379597be5961b4e69e7f187608452874e4c/mesonbuild/mlog.py#L28
Maybe an alternative to colorama?
Is there a specific reason we use sys.stdout.buffer.write
instead of sys.stdout.write
? Because we could just decode the "out" variable and then use sys.stdout.write
after we have replaced the unwanted characters using the encode()
function. The below code works for both cmd and PS:
if hasattr(sys.stdout, 'buffer'):
out = txt.encode(_out_encoding(), 'replace')
txt = out.decode()
sys.stdout.write(txt)
(I'm new to this project so forgive me if I missed something)
Good question; yes! The problem lies in our need to write raw bytes to stdout. The most prominent place where this comes up is when printing filesystem paths. On Unix, these are byte strings, not Unicode strings in any particular encoding. The stdout stream on Unix is also made of raw bytes, so we should be able to print them in “pristine” condition.
However, sys.stdout
itself is a Unicode stream. Printing a path to it, depending on the system’s configured locale, might crash. Using the replace
policy when round-tripping through Unicode is lossy, so paths might not be preserved. This make it impossible to do, for example, stuff like beet ls -p | xargs rm
that consumes filenames on stdout.
Relatedly, the only way we currently have to let the beets configuration override the system’s locale is to write bytes directly to the underlying buffer.
(This is neither here nor there, but I think this whole I/O encoding business was and remains a design flaw in Python 3. It’s the one and only thing I still miss from the Python 2 days.)
@sampsyo
On Unix, these are byte strings, not Unicode strings in any particular encoding. The stdout stream on Unix is also made of raw bytes, so we should be able to print them in “pristine” condition.
Since we are trying to get colour output on Windows, could we just swap where to print: Use the buffer directly on Unix, and Python's print
on Windows?
That seems like a great idea to at least try! I can imagine something going wrong with Windows' special treatment of UTF-16, but it could totally work. If somebody has access to a Windows machine and is willing to give it a shot, I would be super interested.
@sampsyo : I have a Windows machine, and I'd be happy to test! (I don't think I have the time to write the code myself at the moment though...)
Windows has gotten a lot better with handling UTF with the newest versions of PowerShell and Windows Terminal.
Hey I had a similar issue and switching to ConEMu worked to solve my issue.
Problem
When running beets in a Windows 10 command-line terminal (cmd.exe) the output is wrong. It seems like there's a character encoding error. This is what it looks like (look at the last line where beets prompts for Skipping, Replace, etc) : https://imgur.com/a/b9K9x
Setup
My configuration (output of
beet config
) is: