beetbox / beets

music library manager and MusicBrainz tagger
http://beets.io/
MIT License
12.75k stars 1.82k forks source link

Crash on import with UTF-8 names under Mac OS X #1248

Closed claunia closed 9 years ago

claunia commented 9 years ago

Trying to import any file with unicode names, or inside a unicode-named folder, crashes in Mac OS X.

Tested with beets 1.3.8 and 1.3.10, under Mac OS X 10.9.5 and 10.10.0.

Traceback (most recent call last): File "/usr/local/bin/beet", line 9, in load_entry_point('beets==1.3.10', 'console_scripts', 'beet')() File "/Library/Python/2.7/site-packages/beets/ui/init.py", line 945, in main _raw_main(args) File "/Library/Python/2.7/site-packages/beets/ui/init.py", line 935, in _raw_main subcommand.func(lib, suboptions, subargs) File "/Library/Python/2.7/site-packages/beets/ui/commands.py", line 873, in import_func import_files(lib, paths, query) File "/Library/Python/2.7/site-packages/beets/ui/commands.py", line 845, in import_files session.run() File "/Library/Python/2.7/site-packages/beets/importer.py", line 305, in run pl.run_parallel(QUEUE_SIZE) File "/Library/Python/2.7/site-packages/beets/util/pipeline.py", line 299, in run out = self.coro.send(msg) File "/Library/Python/2.7/site-packages/beets/util/pipeline.py", line 181, in coro func(*(args + (task,))) File "/Library/Python/2.7/site-packages/beets/importer.py", line 1268, in plugin_stage func(session, task) File "/Library/Python/2.7/site-packages/beetsplug/keyfinder.py", line 52, in imported self.find_key(task.items) File "/Library/Python/2.7/site-packages/beetsplug/keyfinder.py", line 70, in find_key .format(key, util.displayable_path(item.path))) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 104: ordinal not in range(128)

claunia commented 9 years ago

Disabling keyfinder plugin stops crashing.

Keyfinder configuration: keyfinder: bin: /Applications/KeyFinder.app/Contents/MacOS/KeyFinder auto: yes overwrite: no

sampsyo commented 9 years ago

Hi! Thank you for reporting. I believe this should be fixed in git due to some recent refactoring of the logging machinery. If you get a chance, can you confirm that this is working now?

claunia commented 9 years ago

Unfortunately using git master just changes the error:

Traceback (most recent call last): File "/usr/local/bin/beet", line 9, in load_entry_point('beets==1.3.11', 'console_scripts', 'beet')() File "/Library/Python/2.7/site-packages/beets/ui/init.py", line 947, in main _raw_main(args) File "/Library/Python/2.7/site-packages/beets/ui/init.py", line 937, in _raw_main subcommand.func(lib, suboptions, subargs) File "/Library/Python/2.7/site-packages/beets/ui/commands.py", line 866, in import_func import_files(lib, paths, query) File "/Library/Python/2.7/site-packages/beets/ui/commands.py", line 843, in import_files session.run() File "/Library/Python/2.7/site-packages/beets/importer.py", line 314, in run pl.run_parallel(QUEUESIZE) File "/Library/Python/2.7/site-packages/beets/util/pipeline.py", line 299, in run out = self.coro.send(msg) File "/Library/Python/2.7/site-packages/beets/util/pipeline.py", line 181, in coro func((args + (task,))) File "/Library/Python/2.7/site-packages/beets/importer.py", line 1332, in plugin_stage func(session, task) File "/Library/Python/2.7/site-packages/beets/plugins.py", line 119, in wrapper result = func(_args, **kwargs) File "/Library/Python/2.7/site-packages/beetsplug/keyfinder.py", line 48, in imported self.find_key(task.items) File "/Library/Python/2.7/site-packages/beetsplug/keyfinder.py", line 68, in find_key item.store() File "/Library/Python/2.7/site-packages/beets/library.py", line 239, in store super(LibModel, self).store() File "/Library/Python/2.7/site-packages/beets/dbcore/db.py", line 354, in store tx.mutate(query, subvars) File "/Library/Python/2.7/site-packages/beets/dbcore/db.py", line 655, in mutate cursor = self.db._connection().execute(statement, subvals) sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

sampsyo commented 9 years ago

Wow, a completely different bug! Thanks for reporting. Looking into this now.

sampsyo commented 9 years ago

I took a closer look and found several glaring issues in the keyfinder plugin. :scream: They should be fixed now.

claunia commented 9 years ago

Good, updating to master of masters and testing, stay tuned :stuck_out_tongue:

claunia commented 9 years ago

I have a new bug, but at least, now, it doesn't stop importing:

Logged from file keyfinder.py, line 63
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 467, in format
    s = self._fmt % record.__dict__
  File "/Library/Python/2.7/site-packages/beets/logging.py", line 46, in __str__
    return self.msg.format(*self.args, **self.kwargs)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcc in position 139: ordinal not in range(128)
Logged from file keyfinder.py, line 63
sampsyo commented 9 years ago

Thank you again, and my apologies for the number of bugs you're uncovering in this (apparently neglected) plugin. This latest one should also be fixed now.

claunia commented 9 years ago

No need to apologize, it's been my pleasure to help a fellow developer uncover and kill nasty bugs on a software I do like :+1:

darthanubis commented 9 years ago

Traceback (most recent call last): File "/usr/bin/beet", line 9, in load_entry_point('beets==1.3.8', 'console_scripts', 'beet')() File "/usr/share/beets/beets/ui/init.py", line 964, in main _raw_main(args) File "/usr/share/beets/beets/ui/init.py", line 954, in _raw_main subcommand.func(lib, suboptions, subargs) File "/usr/share/beets/beets/ui/commands.py", line 883, in import_func import_files(lib, paths, query) File "/usr/share/beets/beets/ui/commands.py", line 855, in import_files session.run() File "/usr/share/beets/beets/importer.py", line 297, in run pl.run_parallel(QUEUE_SIZE) File "/usr/share/beets/beets/util/pipeline.py", line 299, in run out = self.coro.send(msg) File "/usr/share/beets/beets/util/pipeline.py", line 158, in coro task = func(*(args + (task,))) File "/usr/share/beets/beets/importer.py", line 1148, in user_query resolve_duplicates(session, task) File "/usr/share/beets/beets/importer.py", line 1158, in resolve_duplicates found_duplicates = task.find_duplicates(session.lib) File "/usr/share/beets/beets/importer.py", line 570, in find_duplicates album_paths = set(i.path for i in album.items()) File "/usr/share/beets/beets/library.py", line 795, in items return self._db.items(dbcore.MatchQuery('album_id', self.id)) File "/usr/share/beets/beets/library.py", line 1103, in items return self._fetch(Item, query, sort) File "/usr/share/beets/beets/library.py", line 1086, in _fetch model_cls, query, sort File "/usr/share/beets/beets/dbcore/db.py", line 758, in _fetch rows = tx.query(sql, subvals) File "/usr/share/beets/beets/dbcore/db.py", line 599, in query return cursor.fetchall() sqlite3.OperationalError: Could not decode to UTF-8 column 'lyrics' with text '[Intro:] It's the Diamonds baby [Notorious B.I.G.] "Everybody move ya body" [x8]

beets crashes on import when getting to Insomnia: The Erick Sermon Compilation Album The Wixtons The lyrics plugin then crashes attempting to read the lyrcis.

beet lyrics Insomnia: The Erick Sermon Compilation Album The Wixtons Traceback (most recent call last): File "/usr/bin/beet", line 9, in load_entry_point('beets==1.3.8', 'console_scripts', 'beet')() File "/usr/share/beets/beets/ui/init.py", line 964, in main _raw_main(args) File "/usr/share/beets/beets/ui/init.py", line 954, in _raw_main subcommand.func(lib, suboptions, subargs) File "/usr/share/beets/beetsplug/lyrics.py", line 492, in func for item in lib.items(ui.decargs(args)): File "/usr/share/beets/beets/library.py", line 1103, in items return self._fetch(Item, query, sort) File "/usr/share/beets/beets/library.py", line 1086, in _fetch model_cls, query, sort File "/usr/share/beets/beets/dbcore/db.py", line 758, in _fetch rows = tx.query(sql, subvals) File "/usr/share/beets/beets/dbcore/db.py", line 599, in query return cursor.fetchall() sqlite3.OperationalError: Could not decode to UTF-8 column 'lyrics' with text '[Intro:] It's the Diamonds baby [Notorious B.I.G.] "Everybody move ya body" [x8]

Jah Boogie: Here comes the Jah changin' faces spoiled rotten A.

Not sure if this is related to this bug, as I didn't want to create a duplicate bug report.

sampsyo commented 9 years ago

@darthanubis This is a separate bug. Can you provide a copy of your database for testing (associated with a new issue)?