«path too long for Windows» crash

prsdta commented 7 years ago

Hi, been trying beets for ordering my music library, but importing my library always leads to a crash. I've just been moving files in the same folder with $ beet import -W -A ..

Traceback (most recent call last):
  File "C:\ProgramData\Chocolatey\lib\python3\tools\Scripts\beet-script.py", line 9, in <module>
    load_entry_point('beets==1.4.2', 'console_scripts', 'beet')()
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\beets\ui\__init__.py", line 1209, in main
    _raw_main(args)
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\beets\ui\__init__.py", line 1196, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\beets\ui\commands.py", line 969, in import_func
    import_files(lib, paths, query)
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\beets\ui\commands.py", line 946, in import_files
    session.run()
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\beets\importer.py", line 320, in run
    pl.run_parallel(QUEUE_SIZE)
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\beets\util\pipeline.py", line 445, in run_parallel
    six.reraise(exc_info[0], exc_info[1], exc_info[2])
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\six.py", line 686, in reraise
    raise value
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\beets\util\pipeline.py", line 358, in run
    self.coro.send(msg)
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\beets\util\pipeline.py", line 171, in coro
    task = func(*(args + (task,)))
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\beets\importer.py", line 1422, in manipulate_files
    task.finalize(session)
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\beets\importer.py", line 540, in finalize
    move=session.config['move'])
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\beets\importer.py", line 566, in cleanup
    self.prune(old_path)
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\beets\importer.py", line 799, in prune
    if self.toppath and not os.path.exists(filename):
  File "c:\programdata\chocolatey\lib\python3\tools\lib\genericpath.py", line 19, in exists
    os.stat(path)
ValueError: stat: path too long for Windows

I tried re-running, reinstalling beets, checked the deps, it always ends up crashing. I also tried removing the album that causes the crash, but the error just happens again on the next folder.

I tried looking around to see what would cause this python error, but having no knowledge in the language didn't lead me far.

Any idea what would cause this? My library is now half-ordered half-mess. :'( I've got a lot of files with Japanese characters, maybe it's important to mention?

Setup

OS: Windows 10
Python version: 3.5.1
beets version: 1.4.2
Turning off plugins made problem go away (yes/no): no (haven't check but unrelated since I've had this issue before trying to setup plugins)

beet config:

import:
    move: yes
    copy: no
    write: no
directory: c:/Users/Lenovo/Music
library: c:/Users/Lenovo/AppData/Roaming/beets/data/musiclibrary.blb
play:
    command: mpv $args --playlist
    warning_threshold: -2
    relative_to:
    raw: no
    warning_treshold: 100
    use_folders: no

plugins: play

sampsyo commented 7 years ago

Thanks for reporting! Can you please include the verbose log leading up to this error?

The defaults should avoid too-long filenames, but in case something goes wrong, there is always the max_filename_length override. But it certainly shouldn't cause a crash like this in any case!

ghost commented 7 years ago

at least for windows 10 and python 3.6 we can offer proper long paths via http://www.howtogeek.com/266621/how-to-make-windows-10-accept-file-paths-over-260-characters/

prsdta commented 7 years ago

@sampsyo Okay I figured how to make the -vv work haha, does this help ? :

C:\Users\Lenovo\Music\PIKA - Ryu no Sumika [320]
Sending event: import_task_created
Replacing item 13731: C:\Users\Lenovo\Music\PIKA - Ryu no Sumika [320]\01.mp3
Sending event: database_change
Sending event: item_removed
Replacing item 13732: C:\Users\Lenovo\Music\PIKA - Ryu no Sumika [320]\02.mp3
Sending event: database_change
Sending event: item_removed
[...]
Replacing item 13740: C:\Users\Lenovo\Music\PIKA - Ryu no Sumika [320]\10.mp3
Sending event: database_change
Sending event: database_change
Sending event: item_removed
10 of 10 items replaced
Sending event: database_change
Sending event: database_change
[...]
Sending event: database_change
Reimported album: added 1483046946.8922455, flexible attributes dict_keys([]) from album 1327 for C:\Users\Lenovo\Music\PIKA - Ryu no Sumika [320]
Reimported item added 1483046946.9032745 from item 13731 for C:\Users\Lenovo\Music\PIKA - Ryu no Sumika [320]\01.mp3
Reimported item flexible attributes dict_keys([]) from item 13731 for C:\Users\Lenovo\Music\PIKA - Ryu no Sumika [320]\01.mp3
Sending event: database_change
Reimported item added 1483046946.9037762 from item 13732 for C:\Users\Lenovo\Music\PIKA - Ryu no Sumika [320]\02.mp3
Reimported item flexible attributes dict_keys([]) from item 13732 for C:\Users\Lenovo\Music\PIKA - Ryu no Sumika [320]\02.mp3
Sending event: database_change
[...]
Reimported item added 1483046946.9082923 from item 13740 for C:\Users\Lenovo\Music\PIKA - Ryu no Sumika [320]\10.mp3
Reimported item flexible attributes dict_keys([]) from item 13740 for C:\Users\Lenovo\Music\PIKA - Ryu no Sumika [320]\10.mp3
Sending event: database_change
Sending event: before_item_moved
Sending event: item_moved
Sending event: database_change
Sending event: database_change
Sending event: before_item_moved
Sending event: item_moved
Sending event: database_change
[...]
Sending event: database_change
Sending event: before_item_moved
Sending event: item_moved
Sending event: database_change
Sending event: database_change
[...]
Sending event: database_change
Sending event: database_change
Sending event: import_task_files
Traceback (most recent call last):
  File "C:\ProgramData\Chocolatey\lib\python3\tools\Scripts\beet-script.py", line 9, in <module>
    load_entry_point('beets==1.4.2', 'console_scripts', 'beet')()
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\beets\ui\__init__.py", line 1209, in main
    _raw_main(args)
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\beets\ui\__init__.py", line 1196, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\beets\ui\commands.py", line 969, in import_func
    import_files(lib, paths, query)
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\beets\ui\commands.py", line 946, in import_files
    session.run()
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\beets\importer.py", line 320, in run
    pl.run_parallel(QUEUE_SIZE)
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\beets\util\pipeline.py", line 445, in run_parallel
    six.reraise(exc_info[0], exc_info[1], exc_info[2])
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\six.py", line 686, in reraise
    raise value
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\beets\util\pipeline.py", line 358, in run
    self.coro.send(msg)
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\beets\util\pipeline.py", line 171, in coro

    task = func(*(args + (task,)))
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\beets\importer.py", line 1422, in manipulate_files
    task.finalize(session)
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\beets\importer.py", line 540, in finalize
    move=session.config['move'])
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\beets\importer.py", line 566, in cleanup
    self.prune(old_path)
  File "c:\programdata\chocolatey\lib\python3\tools\lib\site-packages\beets\importer.py", line 799, in prune
    if self.toppath and not os.path.exists(filename):
  File "c:\programdata\chocolatey\lib\python3\tools\lib\genericpath.py", line 19, in exists
    os.stat(path)
ValueError: stat: path too long for Windows

@jrobeson looks interesting but as you can see in my log, my path is still far from 260 characters, so it's not really an issue at the moment... Or without that the limit is much lower? I figure from the log the problem could be the next album, in which case the first file's path is 90 characters long.

sampsyo commented 7 years ago

That's incredibly weird. You're right that there are no obvious long filenames involved here.

To debug this, would you be willing to run the command again with a print statement in place so we can get a look at the filename in question? The idea would be to insert:

print('***', repr(filename))

above this line: https://github.com/beetbox/beets/blob/8bb24e31343c9707493d3c37668ebb55c69bcc0e/beets/importer.py#L797

which is where the crash happens. Then we'll at least have certainty about what the filename is that Python is complaining about.

sampsyo commented 7 years ago

Also, just to check: was that the only album imported in that run? If not, it's possible this crash came from a different album. (Because beets's importer is multithreaded, it might be working on one album while logging messages about a different one.)

prsdta commented 7 years ago

@sampsyo The multithreading got me on the wrong file indeed! Using print() I found a culprit with a 291 characters path. So the error message was actually exact. I'll set the max_filename_length so that further problems like this don't bother me again...

It could be a small interesting feature to catch this error and print the name of the file for other users who would end up in this case. If it's doable in python?

Thanks guys for helping me debug this.

sampsyo commented 7 years ago

Hmm! The mystery here, though, is why the problem wasn't caught earlier. The crash seems to come at the very end of the import process, when we have accessed the file many times earlier. So yes, we should catch this error, but I'm still somewhat mystified as to why it's not caught by our existing error-handling code.

Maybe it would be helpful to see a verbose log from importing just the album (or even just a single file) that causes the crash?

beetbox / beets

«path too long for Windows» crash #2356

Setup