search

beetbox / beets

music library manager and MusicBrainz tagger
http://beets.io/
MIT License
12.72k stars 1.81k forks source link

program abortion due to exception in fnmatch_all() #1996

Closed Manganus closed 5 years ago

Manganus commented 8 years ago

Sorry about not knowing much how to compose this kind of messages!

In the duplicates init.py, under _def prunedirs(), I've added a try-except-clause in order to avoid error and aborted run

It now looks like this:

for directory in ancestors:
        directory = syspath(directory)
        if not os.path.exists(directory):
            # Directory gone already.
            continue
        try:
            if fnmatch_all(os.listdir(directory), clutter):
                # Directory contains only clutter (or nothing).
                try:
                    shutil.rmtree(directory)
                except OSError:
                    break
                except BaseException as foo:
                    print(foo)
                    break
            else:
                break
        except BaseException as foo:
            print(foo)
            break

Problem

I haven't really understood exactly what's going wrong. (Maybe some day when I've more energy?)

The program aborts with a non-intuitive error message about too many levels of symlinks.

Setup

My configuration is:

user configuration: /home/johan/.config/beets/config.yaml
data directory: /home/johan/.config/beets
plugin paths: 
Sending event: pluginload
library database: /tmp/tracks.db
library directory: /mnt/qnap-212
Sending event: library_opened
verbose: 1
library: /tmp/tracks.db
per_disc_numbering: yes

statefile: prints.pickle
chroma:
    auto: yes
original_date: yes
ftintitle:
    format: (feat. {0})
lastgenre:
    auto: no
    count: 5
    force: no
    source: track
    separator: '/ '
    whitelist: ~/.config/beets/whitelist.txt
    canonical: ~/.config/beets/canonical.txt
duplicates:
    keys: [acoustid_fingerprint]
    merge: yes
    tiebreak:
        items: [bitrate]
    count: no
    full: no
    format: ''
    move: ''
    tag: ''
    path: no
    copy: ''
    album: no
    strict: no
    checksum: ''
    delete: no

plugins: chroma duplicates
directory: /mnt/qnap-212/

import:
    write: yes
    copy: no
    move: no
    link: no
    delete: no
    resume: ask
    incremental: yes
    timid: no
    log: prints_importlog.txt
    autotag: no
    singletons: yes
    default_action: apply
    detail: no
    flat: no
    group_albums: no
    pretend: no
    languages:
    - en
    - de
    - es
    - sv
    - fr
    - pt
    - fi
    - it
log: printslog.txt

paths:
    singleton: '%if{$mb_trackid,mb_}Singles/%if{$artist,%lower{%asciify{$artist}},_}/%if{$album,%lower{%asciify{$album}},_}%if{$year, [$year]}/%if{$disc,$disc-}%if{$track,$track. }%if{$title,$title,_}%if{$album, [$album]}'
    comp: '%if{$mb_albumid,mb_}Collections/%lower{%asciify{$album}}_%aunique{}/%if{$disc,$disc-}%if{$track,$track. }%if{$title,$title,_}%if{$artist, [$artist]}'
    default: '%if{$mb_albumid,mb_}Albums/%if{$albumartist,%lower{%asciify{$albumartist}}_}/%lower{%asciify{$album}}_%aunique{}/%if{$disc,$disc-}%if{$track,$track. }%if{$title,$title,_}'
replace:
    '[\\/\xa0]': _
    '[`\x27]': "\u2019"
    '[\"]': "\u201D"
    \.\.\.: "\u2026"
    ^\-: _
    ^\.: _
    '[\x00-\x1f]': _
    '[<>:"\?\*\|]': _
    \.$: _
    \s+$: ''
    ^\s+: ''

match:
    distance_weights:
        artist: 2.0
        album: 2.5
        year: 1.0
        label: 0.5
        catalognum: 0.5
        albumdisambig: 0.5
        album_id: 2.0
        tracks: 2.0
        missing_tracks: 0.1
        unmatched_tracks: 5.0
        track_title: 2.0
        track_artist: 2.0
        track_index: 1.0
        track_length: 9.0
        track_id: 2.0
    preferred:
        countries: []
        media: []
        original_year: yes
    ignored: [track_length unmatched_tracks]
    track_length_grace: 3
    track_length_max: 15
jackwilsdon commented 8 years ago

Do you think you could provide the exact message you're getting? Thanks :smile:!

Manganus commented 8 years ago

That may be possible - but it's far from sure. The condition that caused the abortions may, or may not, be removed now.

I ought to be able to produce a clean beets-system, restore the source, fill up the library with dupes, and then try to provoke the error...

However, I think I would start with looking at the code. That may demand less time and effort... ;)

The error message seemed confused/confusing. I didn't find any soft links at all.

The traceback referred to line 266 in the init-script.

Line 266 then had this content: if fnmatch_all(os.listdir(directory), clutter):

sampsyo commented 8 years ago

Well, we really do need to know which exception was raised so we know what to handle. Thank you!

Manganus commented 8 years ago

Is there some logging function that can be called to catch errors one is looking for, and log them to another file than stderr?

Manganus commented 8 years ago

If I provoke the program(s) in order to try to establish the same error messages, would it be enough with catching the exceptions like below, or is the exceptions you are asking for something else? 

try:
    debug1 = os.listdir(directory)
except BaseException as foo:
    debug_log(foo)
    debug_log(directory)
    continue
try:
    debug2 = fnmatch_all(debug1, clutter)
except BaseException as foo:
    debug_log(foo)
    debug_log(debug1)
    continue
if debug2:
    # Directory contains only clutter (or nothing).
    try:
        shutil.rmtree(directory)
    except OSError:
        break
    except BaseException as foo:
        debug_log(foo)
        debug_log(directory)
        continue
else:
    break

the debug_log() printing to some dedicated file

jackwilsdon commented 8 years ago

Honestly it would be better if the exception was thrown, as we can then see the stack trace.

You can re-throw an exception by just doing throw foo.

Also what is the issue with printing to stderr? You can capture it to a file like so if you want:

my_command 2> stderr_file
Manganus commented 8 years ago 
Traceback (most recent call last):
  File "/usr/local/bin/beet", line 9, in <module>
    load_entry_point('beets==1.3.17', 'console_scripts', 'beet')()
  File "/usr/local/lib/python2.7/dist-packages/beets/ui/__init__.py", line 1236, in main
    _raw_main(args)
  File "/usr/local/lib/python2.7/dist-packages/beets/ui/__init__.py", line 1226, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/usr/local/lib/python2.7/dist-packages/beetsplug/duplicates.py", line 162, in _dup
    fmt=fmt.format(obj_count))
  File "/usr/local/lib/python2.7/dist-packages/beetsplug/duplicates.py", line 179, in _process_item
    item.remove(delete=True)
  File "/usr/local/lib/python2.7/dist-packages/beets/library.py", line 720, in remove
    util.prune_dirs(os.path.dirname(self.path), self._db.directory)
  File "/usr/local/lib/python2.7/dist-packages/beets/util/__init__.py", line 266, in prune_dirs
    debug1 = os.listdir(directory)
OSError: [Errno 40] Too many levels of symbolic links: '/mnt/qnap-119/vbr/Singles/buena vista cuba'

Program was invoked with: $ beet -vvc /mnt/qnap-119/vbr_import.yaml dup -d

The code there currently looks like this:

        debug1 = os.listdir(directory)
        debug2 = fnmatch_all(debug1, clutter)
        if debug2:
Manganus commented 8 years ago

The configuration for this run looked like this:

$ beet -vvc /mnt/qnap-119/vbr_import.yaml config
user configuration: /home/johan/.config/beets/config.yaml
data directory: /home/johan/.config/beets
plugin paths: 
Sending event: pluginload
library database: /mnt/qnap-119/vbr/tracks.db
library directory: /mnt/qnap-119/vbr
Sending event: library_opened
verbose: 2
scrub:
    auto: no

statefile: /mnt/qnap-119/vbr/tracks.pickle
library: /mnt/qnap-119/vbr/tracks.db
replace:
    '[\\/\xa0]': _
    '[`\x27]': "\u2019"
    '[\"]': "\u201D"
    \.\.\.: "\u2026"
    ^\-: _
    ^\.: _
    '[\x00-\x1f]': _
    '[<>:"\?\*\|]': _
    \.$: _
    \s+$: ''
    ^\s+: ''
original_date: yes
ftintitle:
    format: (feat. {0})
    auto: yes
    drop: no

plugins: ftintitle duplicates mbsync lastgenre chroma acousticbrainz echonest
acousticbrainz:
    auto: no
per_disc_numbering: yes

paths:
    singleton: '%if{$mb_trackid,mb_}Singles/%if{$album,%lower{%asciify{$album}},%if{$artist,%lower{%asciify{$artist}},_}}/%if{$disc,$disc-}%if{$track,$track. }%if{$title,$title,_}%if{$artist, [$artist],%if{$albumartist, [$albumartist]}}%if{$year, ($year)}'
    comp: '%if{$mb_albumid,mb_}Collections/%lower{%asciify{$album}}_%aunique{}/%if{$disc,$disc-}%if{$track,$track. }%if{$title,$title,_}%if{$artist, [$artist]}'
    default: '%if{$mb_albumid,mb_}Albums/%if{$albumartist,%lower{%asciify{$albumartist}}_}/%lower{%asciify{$album}}_%aunique{}/%if{$disc,$disc-}%if{$track,$track. }%if{$title,$title,_}'
log: /mnt/qnap-119/vbr/trackslog.txt
chroma:
    auto: yes
duplicates:
    merge: yes
    keys: [acoustid_fingerprint]
    tiebreak:
        items: [bitrate]
    count: no
    full: no
    format: ''
    move: ''
    tag: ''
    path: no
    copy: ''
    album: no
    strict: no
    checksum: ''
    delete: no
directory: /mnt/qnap-119/vbr

import:
    write: yes
    copy: yes
    move: no
    link: no
    delete: no
    resume: ask
    incremental: yes
    quiet_fallback: skip
    none_rec_action: ask
    timid: no
    log: /mnt/qnap-119/vbr/importlog.txt
    autotag: no
    quiet: yes
    singletons: yes
    default_action: apply
    languages:
    - en
    - de
    - es
    - sv
    - fr
    - pt
    - fi
    - it
    detail: no
    flat: no
    group_albums: no
    pretend: no
    search_ids: []
echonest:
    auto: no
    upload: no
    convert: no
    apikey: REDACTED
    truncate: yes
    energy: energy
    liveness: liveness
    tempo: bpm
    speechiness: speechiness
    acousticness: acousticness
    danceability: danceability
    valence: valence
lastgenre:
    count: 5
    source: track
    separator: '/ '
    whitelist: ~/.config/beets/whitelist.txt
    force: yes
    min_weight: 10
    auto: yes
    fallback:
    canonical: no

match:
    distance_weights:
        artist: 2.0
        album: 2.5
        year: 1.0
        label: 0.5
        catalognum: 0.5
        albumdisambig: 0.5
        album_id: 2.0
        tracks: 2.0
        missing_tracks: 0.1
        unmatched_tracks: 5.0
        track_title: 2.0
        track_artist: 2.0
        track_index: 1.0
        track_length: 9.0
        track_id: 2.0
    preferred:
        countries: []
        media: []
        original_year: yes
    ignored: [track_length unmatched_tracks]
    track_length_grace: 3
    track_length_max: 15

Sending event: cli_exit
Manganus commented 8 years ago

The real culprit, it seems, must be the os.listdir() function. However, it's easier to catch an exception in python code than to wait for os.listdir() to become updated.

The directory, that couldn't get listed now, is nothing particular: It's one of the many directories created by beet import. There are absoulutely no symlinks at all in beet's directory tree:

johan@T60-SSD ~ $ find /mnt/qnap-119/vbr/Singles -type l
johan@T60-SSD ~ $ ls -ls '/mnt/qnap-119/vbr/Singles' | wc
   2082   26742  147268
johan@T60-SSD ~ $

And for the curious, the directory listing looks like this:

$ ls -lstr '/mnt/qnap-119/vbr/Singles/svensk jazzhistoria, vol 6 - 1947-1951'
total 621240
 5368 -rw-rw-r-- 1 johan johan  5496684 May 10 02:52 01. Undecided [Alice Babs].mp3
 1896 -rw-rw-r-- 1 johan johan  1939005 May 10 02:52 02. Julpotpurri [Gunnar Nilson;Flickery Flies].mp3
 4568 -rw-rw-r-- 1 johan johan  4675621 May 10 02:53 03. Down the hatch [Jularbo J-rs trio].mp3
 4916 -rw-rw-r-- 1 johan johan  5030443 May 10 02:53 04. Talahasse [Harry Arnolds orkester].mp3
 4672 -rw-rw-r-- 1 johan johan  4782491 May 10 02:53 05. The way you look tonight [Simon Brehms sextett].mp3
 3812 -rw-rw-r-- 1 johan johan  3900477 May 10 02:53 06. Emanon [Simon Brehms sextett].mp3
 5288 -rw-rw-r-- 1 johan johan  5414895 May 10 02:53 07. I got rhythm [Anders Burmans kvintett].mp3
 4200 -rw-rw-r-- 1 johan johan  4300484 May 10 02:53 08. All the things you are [Thore Jederbys sextett].mp3
 4652 -rw-rw-r-- 1 johan johan  4761549 May 10 02:53 09. Rhythm in riff [Thore Jederbys kvartett].mp3
 5292 -rw-rw-r-- 1 johan johan  5415333 May 10 02:53 10. Reactions [Gsta Theselius orkester].mp3
 6072 -rw-rw-r-- 1 johan johan  6217509 May 10 02:53 11. Yesterdays [Putte Wickman].mp3
 3068 -rw-rw-r-- 1 johan johan  3140982 May 10 02:53 12. Mean to me [Hasse Kahns kvintett].mp3
 4676 -rw-rw-r-- 1 johan johan  4781903 May 10 02:53 13. Sweet Sue [Gsta Thrners orkester].mp3
 4784 -rw-rw-r-- 1 johan johan  4896365 May 10 02:53 14. Jazz me blues [Gsta Thrners orkester 2].mp3
 5156 -rw-rw-r-- 1 johan johan  5277415 May 10 02:53 15. Weary blues [Hep Cats].mp3
 4132 -rw-rw-r-- 1 johan johan  4230983 May 10 02:53 16. Call of the freaks [Grav-Olle Storyville Four;Brita Lindahl].mp3
 3696 -rw-rw-r-- 1 johan johan  3781556 May 10 02:53 17. Jackass blues [Buntas Jazz Band].mp3
 5508 -rw-rw-r-- 1 johan johan  5637287 May 10 02:53 18. Panama [Olle Sundhs Dixielandband].mp3
 5132 -rw-rw-r-- 1 johan johan  5253535 May 10 02:53 19. All the things you are [Expressens Elitorkester 1949].mp3
 4420 -rw-rw-r-- 1 johan johan  4522896 May 10 02:53 20. Idaho [Parisorkestern 1949].mp3
 3396 -rw-rw-r-- 1 johan johan  3476015 May 10 02:53 21. Sugar [Alice Babs].mp3
 4644 -rw-rw-r-- 1 johan johan  4753763 May 10 02:53 22. Indiana [Reinhold Svenssons trio].mp3
 4500 -rw-rw-r-- 1 johan johan  4606445 May 10 02:53 23. Diggin’ in minor [Carl-Henrik Norins orkester].mp3
 4136 -rw-rw-r-- 1 johan johan  4234965 May 10 02:53 24. Body and soul [Arne Domnérus’ Favourite Four].mp3
 4944 -rw-rw-r-- 1 johan johan  5058593 May 10 02:53 25. Lonely moments [Seymour sterwalls orkester].mp3
 5292 -rw-rw-r-- 1 johan johan  5416333 May 10 02:53 26. Blue Skies [Putte Wickmans Sextett].mp3
 4440 -rw-rw-r-- 1 johan johan  4544387 May 10 02:53 27. Cherokee [Putte Wickmans Sextett].mp3
 4444 -rw-rw-r-- 1 johan johan  4548260 May 10 02:53 28. All The Things You Are [Putte Wickmans Sextett].mp3
 4300 -rw-rw-r-- 1 johan johan  4399740 May 10 02:53 29. All the things you are 2 [Putte Wickmans Sextett].mp3
 5272 -rw-rw-r-- 1 johan johan  5396217 May 10 02:53 30. Boogie Woogie On St. Louis Blues [Charles Norman Quintet].mp3
 4736 -rw-rw-r-- 1 johan johan  4846185 May 10 02:53 31. Be-bop Läres Här [Wille Wallén;Totty Wallén och hans Vilda Vikingar].mp3
 5504 -rw-rw-r-- 1 johan johan  5633047 May 10 02:53 32. How High The Moon [Bibbi Johnson].mp3
 3604 -rw-rw-r-- 1 johan johan  3690119 May 10 02:53 33. Almost Bald [Thore Swaneruds Trio].mp3
 4840 -rw-rw-r-- 1 johan johan  4955602 May 10 02:53 34. Dixie March [Olle Jacobssons Dixielandband].mp3
 3968 -rw-rw-r-- 1 johan johan  4059173 May 10 02:53 35. Out Of Nowhere [James Moody And His Swedish Crowns].mp3
 4540 -rw-rw-r-- 1 johan johan  4648262 May 10 02:53 36. Be-bop Accordeon [Lill-Arne Söderbergs Kvintett].mp3
 4792 -rw-rw-r-- 1 johan johan  4905678 May 10 02:53 37. Be-bop-session, part 2 (Fine And Dandy) [Body Buser Presents_ Jazz At The Java].mp3
 4976 -rw-rw-r-- 1 johan johan  5091841 May 10 02:53 38. Ice Feeling [Malte Johnsons Orkester].mp3
 5152 -rw-rw-r-- 1 johan johan  5273923 May 10 02:53 39. Lover Man [Bengt Hallberg Trio].mp3
 3936 -rw-rw-r-- 1 johan johan  4028380 May 10 02:53 40. Love [Dubbelpianisterna Reinhold och Gunnar Svensson].mp3
 4036 -rw-rw-r-- 1 johan johan  4129735 May 10 02:53 41. Tiger Boogie [Olson Brothers].mp3
 3776 -rw-rw-r-- 1 johan johan  3863294 May 10 02:53 42. How High The Moon [Sören Christensens Kvartett].mp3
 5028 -rw-rw-r-- 1 johan johan  5147612 May 10 02:53 43. Wilhelmina [Brita Borg;Flickery Flies].mp3
 7064 -rw-rw-r-- 1 johan johan  7232655 May 10 02:53 44. Body And Soul [Jamsession På Karlaplan].mp3
 7168 -rw-rw-r-- 1 johan johan  7337210 May 10 02:53 45. Perdido [Jamsession På Karlaplan].mp3
 4928 -rw-rw-r-- 1 johan johan  5043944 May 10 02:53 46. Domaredansen [Delta Rhythm Boys With Metronome All-Stars].mp3
 6060 -rw-rw-r-- 1 johan johan  6205218 May 10 02:53 47. Out Of Nowhere [Arne Domnérus].mp3
 8148 -rw-rw-r-- 1 johan johan  8342700 May 10 02:53 48. I Surrender Dear [Rolf Blomquist].mp3
 5548 -rw-rw-r-- 1 johan johan  5678568 May 10 02:53 49. Godchild [Lars Gullin].mp3
 5124 -rw-rw-r-- 1 johan johan  5243451 May 10 02:53 50. Barit [Arne Domnérus orkester].mp3
 4976 -rw-rw-r-- 1 johan johan  5092111 May 10 02:53 51. The Heat’s on [Roy Eldridge Sextet].mp3
 3104 -rw-rw-r-- 1 johan johan  3178360 May 10 02:53 52. How High the Moon [Sven Geys Gitarrtrio].mp3
 4240 -rw-rw-r-- 1 johan johan  4341203 May 10 02:53 53. Laura [Sven Geys Gitarrtrio].mp3
 5028 -rw-rw-r-- 1 johan johan  5147518 May 10 02:54 54. Tiger Rag [Erik Franks Sextett].mp3
11176 -rw-rw-r-- 1 johan johan 11440221 May 10 02:54 55. Of All the Wrong You’ve Done to Me [Black Bottom Stompers].mp3
 4868 -rw-rw-r-- 1 johan johan  4983548 May 10 02:54 56. Cupol Boogie [Gunnar Hoffstens Orkester].mp3
 4492 -rw-rw-r-- 1 johan johan  4599264 May 10 02:54 57. Ack Värmeland Du Sköna [Stan Getz Quartet].mp3
 5228 -rw-rw-r-- 1 johan johan  5352494 May 10 02:54 58. The Man I Love [Arne Domnérus med stråkorkester].mp3
 4728 -rw-rw-r-- 1 johan johan  4839619 May 10 02:54 59. Remember When [Carl-Henrik Norins Kvartett].mp3
 5040 -rw-rw-r-- 1 johan johan  5158242 May 10 02:54 60. Leonard Feather Intervjuas Av Olle Helander [Leonard Feather].mp3
 4168 -rw-rw-r-- 1 johan johan  4264324 May 10 02:54 61. Swedish Butterfly [Leonard Feather’s Swinging Swedes].mp3
 4776 -rw-rw-r-- 1 johan johan  4888387 May 10 02:54 62. The Swedish Music This Side of Heaven [Leonard Feather’s Swinging Swedes].mp3
 4436 -rw-rw-r-- 1 johan johan  4538474 May 10 02:54 63. Tesselation [Paramountorkestern].mp3
 4600 -rw-rw-r-- 1 johan johan  4708080 May 10 02:54 64. Bei Mir Bist Du Schön [Simon Brehms Orkester].mp3
 3616 -rw-rw-r-- 1 johan johan  3701517 May 10 02:54 65. What Is This Thing Called Love [Sonya Hedenbratt, Kenneth Fagerlunds Trio].mp3
 5592 -rw-rw-r-- 1 johan johan  5723062 May 10 02:54 66. First Walk [Putte Wickmans Specialorkester].mp3
 4964 -rw-rw-r-- 1 johan johan  5082368 May 10 02:54 67. Smiles [Johan Adolfsson Sextett].mp3
 3972 -rw-rw-r-- 1 johan johan  4063714 May 10 02:54 68. Beat the Clock [Reinhold Svensson Quintet].mp3
 4636 -rw-rw-r-- 1 johan johan  4744691 May 10 02:54 69. Attribution [Kenneth Fagerlunds Kvintett].mp3
12028 -rw-rw-r-- 1 johan johan 12315273 May 10 02:54 70. Three Without a Key [Ulf Lindes Kvartett].mp3
 8492 -rw-rw-r-- 1 johan johan  8691944 May 10 02:54 71. Cream of the Crop [Jazzkritikerorkestern 1951].mp3
 2376 -rw-rw-r-- 1 johan johan  2429771 May 10 06:35 01. Undecided [Alice Babs].1.mp3
  896 -rw-rw-r-- 1 johan johan   915446 May 10 06:35 02. Julpotpurri [Gunnar Nilson].mp3
 2076 -rw-rw-r-- 1 johan johan  2122853 May 10 06:35 03. Down the hatch [Jularbo J-rs trio].1.mp3
 2220 -rw-rw-r-- 1 johan johan  2269396 May 10 06:35 04. Talahasse [Harry Arnolds orkester].1.mp3
 2496 -rw-rw-r-- 1 johan johan  2551901 May 10 06:35 05. The way you look tonight [Simon Brehms sextett].1.mp3
 2016 -rw-rw-r-- 1 johan johan  2064179 May 10 06:35 06. Emanon [Simon Brehms sextett].1.mp3
 2556 -rw-rw-r-- 1 johan johan  2614295 May 10 06:35 07. I got rhythm [Anders Burmans kvintett].1.mp3
 1888 -rw-rw-r-- 1 johan johan  1931742 May 10 06:35 08. All the things you are [Thore Jederbys sextett].1.mp3
 2104 -rw-rw-r-- 1 johan johan  2150583 May 10 06:35 09. Rhythm in riff [Thore Jederbys kvartett].1.mp3
 2752 -rw-rw-r-- 1 johan johan  2814609 May 10 06:35 10. Reactions [Gsta Theselius orkester].1.mp3
 3148 -rw-rw-r-- 1 johan johan  3220602 May 10 06:35 11. Yesterdays [Putte Wickman].1.mp3
 1568 -rw-rw-r-- 1 johan johan  1604547 May 10 06:35 12. Mean to me [Hasse Kahns kvintett].1.mp3
 2136 -rw-rw-r-- 1 johan johan  2186102 May 10 06:35 13. Sweet Sue [Gsta Thrners orkester].1.mp3
 2200 -rw-rw-r-- 1 johan johan  2250148 May 10 06:35 14. Jazz me blues [Gsta Thrners orkester 2].1.mp3
 2504 -rw-rw-r-- 1 johan johan  2562220 May 10 06:35 15. Weary blues [Hep Cats].1.mp3
 2048 -rw-rw-r-- 1 johan johan  2093901 May 10 06:35 16. Call of the freaks [Grav-Olle Storyville Four].mp3
 1660 -rw-rw-r-- 1 johan johan  1698993 May 10 06:35 17. Jackass blues [Buntas Jazz Band].1.mp3
 2588 -rw-rw-r-- 1 johan johan  2649783 May 10 06:35 18. Panama [Olle Sundhs Dixielandband].1.mp3
 2340 -rw-rw-r-- 1 johan johan  2392208 May 10 06:35 19. All the things you are [Expressens Elitorkester 1949].1.mp3
 2008 -rw-rw-r-- 1 johan johan  2052948 May 10 06:35 20. Idaho [Parisorkestern 1949].1.mp3
 1872 -rw-rw-r-- 1 johan johan  1913049 May 10 06:35 21. Sugar [Alice Babs].1.mp3
 2532 -rw-rw-r-- 1 johan johan  2588909 May 10 06:36 22. Indiana [Reinhold Svenssons trio].1.mp3
 1960 -rw-rw-r-- 1 johan johan  2006332 May 10 06:36 23. Diggin’ in minor [Carl-Henrik Norins orkester].1.mp3
 2116 -rw-rw-r-- 1 johan johan  2164928 May 10 06:36 24. Body and soul [Arne Domnérus’ Favourite Four].1.mp3
 2432 -rw-rw-r-- 1 johan johan  2489978 May 10 06:36 25. Lonely moments [Seymour sterwalls orkester].1.mp3
 2516 -rw-rw-r-- 1 johan johan  2573068 May 10 06:36 26. Blue Skies [Putte Wickmans Sextett] (1949).mp3
 2028 -rw-rw-r-- 1 johan johan  2074519 May 10 06:36 27. Cherokee [Putte Wickmans Sextett] (1949).mp3
 1984 -rw-rw-r-- 1 johan johan  2029050 May 10 06:36 28. All The Things You Are [Putte Wickmans Sextett] (1950).mp3
 2236 -rw-rw-r-- 1 johan johan  2287624 May 10 06:36 29. All the things you are 2 [Putte Wickmans Sextett] (1951).mp3
 2432 -rw-rw-r-- 1 johan johan  2489853 May 10 06:36 30. Boogie Woogie On St. Louis Blues [Charles Norman Quintet] (1949).mp3
 2208 -rw-rw-r-- 1 johan johan  2257058 May 10 06:36 31. Be-bop Läres Här [Wille Wallén] (1949).mp3
 2872 -rw-rw-r-- 1 johan johan  2937773 May 10 06:36 32. How High The Moon [Bibbi Johnson] (1949).mp3
 1600 -rw-rw-r-- 1 johan johan  1636777 May 10 06:36 33. Almost Bald [Thore Swaneruds Trio] (1949).mp3
 2168 -rw-rw-r-- 1 johan johan  2217206 May 10 06:36 34. Dixie March [Olle Jacobssons Dixielandband] (1949).mp3
 2068 -rw-rw-r-- 1 johan johan  2114765 May 10 06:36 35. Out Of Nowhere [James Moody And His Swedish Crowns] (1949).mp3
 2076 -rw-rw-r-- 1 johan johan  2124896 May 10 06:36 36. Be-bop Accordeon [Lill-Arne Söderbergs Kvintett] (1949).mp3
 2164 -rw-rw-r-- 1 johan johan  2215706 May 10 06:36 37. Be-bop-session, part 2 (Fine And Dandy) [Body Buser Presents_ Jazz At The Java] (1950).mp3
 2288 -rw-rw-r-- 1 johan johan  2340842 May 10 06:36 38. Ice Feeling [Malte Johnsons Orkester] (1950).mp3
 2676 -rw-rw-r-- 1 johan johan  2736378 May 10 06:36 39. Lover Man [Bengt Hallberg Trio] (1950).mp3
 1760 -rw-rw-r-- 1 johan johan  1799986 May 10 06:36 40. Love [Dubbelpianisterna Reinhold och Gunnar Svensson] (1950).mp3
 1788 -rw-rw-r-- 1 johan johan  1828291 May 10 06:36 41. Tiger Boogie [Olson Brothers] (1950).mp3
 1684 -rw-rw-r-- 1 johan johan  1723542 May 10 06:36 42. How High The Moon [Sören Christensens Kvartett] (1950).mp3
 2240 -rw-rw-r-- 1 johan johan  2290229 May 10 06:36 43. Wilhelmina [Brita Borg] (1950).mp3
 3524 -rw-rw-r-- 1 johan johan  3604646 May 10 06:36 44. Body And Soul [Jamsession På Karlaplan] (1950).mp3
 3576 -rw-rw-r-- 1 johan johan  3659338 May 10 06:36 45. Perdido [Jamsession På Karlaplan] (1950).mp3
 2252 -rw-rw-r-- 1 johan johan  2302823 May 10 06:36 46. Domaredansen [Delta Rhythm Boys With Metronome All-Stars] (1950).mp3
 2952 -rw-rw-r-- 1 johan johan  3021449 May 10 06:36 47. Out Of Nowhere [Arne Domnérus] (1950).mp3
 3980 -rw-rw-r-- 1 johan johan  4071887 May 10 06:36 48. I Surrender Dear [Rolf Blomquist] (1950).mp3
 2816 -rw-rw-r-- 1 johan johan  2879908 May 10 06:36 49. Godchild [Lars Gullin] (1950).mp3
 2196 -rw-rw-r-- 1 johan johan  2247903 May 10 06:36 50. Barit [Arne Domnérus orkester] (1950).mp3
 2220 -rw-rw-r-- 1 johan johan  2272149 May 10 06:36 51. The Heat’s on [Roy Eldridge Sextet] (1951).mp3
 1724 -rw-rw-r-- 1 johan johan  1762344 May 10 06:37 52. How High the Moon [Sven Geys Gitarrtrio] (1951).mp3
 2292 -rw-rw-r-- 1 johan johan  2345565 May 10 06:37 53. Laura [Sven Geys Gitarrtrio] (1951).mp3
 2272 -rw-rw-r-- 1 johan johan  2323235 May 10 06:37 54. Tiger Rag [Erik Franks Sextett] (1951).mp3
 5636 -rw-rw-r-- 1 johan johan  5768965 May 10 06:37 55. Of All the Wrong You’ve Done to Me [Black Bottom Stompers] (1951).mp3
 2200 -rw-rw-r-- 1 johan johan  2252629 May 10 06:37 56. Cupol Boogie [Gunnar Hoffstens Orkester] (1951).mp3
 2072 -rw-rw-r-- 1 johan johan  2118895 May 10 06:37 57. Ack Värmeland Du Sköna [Stan Getz Quartet] (1951).mp3
 2264 -rw-rw-r-- 1 johan johan  2316547 May 10 06:37 58. The Man I Love [Arne Domnérus med stråkorkester] (1951).mp3
 2180 -rw-rw-r-- 1 johan johan  2229569 May 10 06:37 59. Remember When [Carl-Henrik Norins Kvartett] (1951).mp3
 2504 -rw-rw-r-- 1 johan johan  2563541 May 10 06:37 60. Leonard Feather Intervjuas Av Olle Helander [Leonard Feather] (1951).mp3
 1888 -rw-rw-r-- 1 johan johan  1931667 May 10 06:37 61. Swedish Butterfly [Leonard Feather’s Swinging Swedes] (1951).mp3
 2208 -rw-rw-r-- 1 johan johan  2256953 May 10 06:37 62. The Swedish Music This Side of Heaven [Leonard Feather’s Swinging Swedes] (1951).mp3
 2032 -rw-rw-r-- 1 johan johan  2078298 May 10 06:37 63. Tesselation [Paramountorkestern] (1951).mp3
 2172 -rw-rw-r-- 1 johan johan  2220669 May 10 06:37 64. Bei Mir Bist Du Schön [Simon Brehms Orkester] (1951).mp3
 1652 -rw-rw-r-- 1 johan johan  1687628 May 10 06:37 65. What Is This Thing Called Love [Sonya Hedenbratt, Kenneth Fagerlunds Trio] (1951).mp3
 2648 -rw-rw-r-- 1 johan johan  2708802 May 10 06:37 66. First Walk [Putte Wickmans Specialorkester] (1951).mp3
 2248 -rw-rw-r-- 1 johan johan  2301575 May 10 06:37 67. Smiles [Johan Adolfsson Sextett] (1951).mp3
 1836 -rw-rw-r-- 1 johan johan  1878407 May 10 06:37 68. Beat the Clock [Reinhold Svensson Quintet] (1951).mp3
 2312 -rw-rw-r-- 1 johan johan  2366942 May 10 06:37 69. Attribution [Kenneth Fagerlunds Kvintett] (1951).mp3
 5980 -rw-rw-r-- 1 johan johan  6123432 May 10 06:37 70. Three Without a Key [Ulf Lindes Kvartett] (1951).mp3
 4284 -rw-rw-r-- 1 johan johan  4384193 May 10 06:37 71. Cream of the Crop [Jazzkritikerorkestern 1951] (1951).mp3
  896 -rw-rw-r-- 1 johan johan   915410 May 10 15:49 02. Julpotpurri [Gunnar Nilson].1.mp3
 2076 -rw-rw-r-- 1 johan johan  2122787 May 10 15:49 03. Down the hatch [Jularbo J-rs trio].2.mp3
 2496 -rw-rw-r-- 1 johan johan  2551833 May 10 15:49 05. The way you look tonight [Simon Brehms sextett].2.mp3
 2016 -rw-rw-r-- 1 johan johan  2064111 May 10 15:49 06. Emanon [Simon Brehms sextett].2.mp3
 2556 -rw-rw-r-- 1 johan johan  2614224 May 10 15:49 07. I got rhythm [Anders Burmans kvintett].2.mp3
 1888 -rw-rw-r-- 1 johan johan  1932639 May 10 15:49 08. All the Things You Are [Thore Jederbys sextett].mp3
 2104 -rw-rw-r-- 1 johan johan  2150512 May 10 15:49 09. Rhythm in riff [Thore Jederbys kvartett].2.mp3
 2752 -rw-rw-r-- 1 johan johan  2814538 May 10 15:49 10. Reactions [Gsta Theselius orkester].2.mp3
 1568 -rw-rw-r-- 1 johan johan  1604479 May 10 15:49 12. Mean to me [Hasse Kahns kvintett].2.mp3
 2136 -rw-rw-r-- 1 johan johan  2186033 May 10 15:49 13. Sweet Sue [Gsta Thrners orkester].2.mp3
 2200 -rw-rw-r-- 1 johan johan  2250077 May 10 15:49 14. Jazz me blues [Gsta Thrners orkester 2].2.mp3
 2504 -rw-rw-r-- 1 johan johan  2562164 May 10 15:49 15. Weary blues [Hep Cats].2.mp3
 2048 -rw-rw-r-- 1 johan johan  2093828 May 10 15:50 16. Call of the freaks [Grav-Olle Storyville Four].1.mp3
 1660 -rw-rw-r-- 1 johan johan  1698929 May 10 15:50 17. Jackass blues [Buntas Jazz Band].2.mp3
 2588 -rw-rw-r-- 1 johan johan  2649710 May 10 15:50 18. Panama [Olle Sundhs Dixielandband].2.mp3
 2340 -rw-rw-r-- 1 johan johan  2393094 May 10 15:50 19. All the Things You Are [Expressens Elitorkester] (1949).mp3
 2008 -rw-rw-r-- 1 johan johan  2052881 May 10 15:50 20. Idaho [Parisorkestern 1949].2.mp3
 1872 -rw-rw-r-- 1 johan johan  1912996 May 10 15:50 21. Sugar [Alice Babs].2.mp3
 2116 -rw-rw-r-- 1 johan johan  2164715 May 10 15:50 24. Body and soul [Arne Domnérus’ Favourite Four].2.mp3
 2432 -rw-rw-r-- 1 johan johan  2489904 May 10 15:50 25. Lonely moments [Seymour sterwalls orkester].2.mp3
 2028 -rw-rw-r-- 1 johan johan  2074449 May 10 15:50 27. Cherokee [Putte Wickmans Sextett] (1949).1.mp3
 1984 -rw-rw-r-- 1 johan johan  2029949 May 10 15:50 28. All the Things You Are [Putte Wickmans Sextett] (1950).mp3
 2236 -rw-rw-r-- 1 johan johan  2288521 May 10 15:50 29. All the Things You Are 2 [Putte Wickmans Sextett] (1951).mp3
 2432 -rw-rw-r-- 1 johan johan  2489783 May 10 15:50 30. Boogie Woogie On St. Louis Blues [Charles Norman Quintet] (1949).1.mp3
 2208 -rw-rw-r-- 1 johan johan  2257022 May 10 15:50 31. Be-bop Läres Här [Wille Wallén] (1949).1.mp3
 2168 -rw-rw-r-- 1 johan johan  2217129 May 10 15:51 34. Dixie March [Olle Jacobssons Dixielandband] (1949).1.mp3
 2288 -rw-rw-r-- 1 johan johan  2340772 May 10 15:51 38. Ice Feeling [Malte Johnsons Orkester] (1950).1.mp3
 1788 -rw-rw-r-- 1 johan johan  1828229 May 10 15:51 41. Tiger Boogie [Olson Brothers] (1950).1.mp3
 3524 -rw-rw-r-- 1 johan johan  3604573 May 10 15:51 44. Body And Soul [Jamsession På Karlaplan] (1950).1.mp3
 2952 -rw-rw-r-- 1 johan johan  3021387 May 10 15:51 47. Out Of Nowhere [Arne Domnérus] (1950).1.mp3
 2272 -rw-rw-r-- 1 johan johan  2323167 May 10 15:52 54. Tiger Rag [Erik Franks Sextett] (1951).1.mp3
 5636 -rw-rw-r-- 1 johan johan  5768896 May 10 15:52 55. Of All the Wrong You’ve Done to Me [Black Bottom Stompers] (1951).1.mp3
 2180 -rw-rw-r-- 1 johan johan  2229565 May 10 15:52 59. Remember When [Carl-Henrik Norins Kvartett] (1951).1.mp3
 1888 -rw-rw-r-- 1 johan johan  1931586 May 10 15:52 61. Swedish Butterfly [Leonard Feather’s Swinging Swedes] (1951).1.mp3
 2204 -rw-rw-r-- 1 johan johan  2256872 May 10 15:52 62. The Swedish Music This Side of Heaven [Leonard Feather’s Swinging Swedes] (1951).1.mp3
 2032 -rw-rw-r-- 1 johan johan  2078232 May 10 15:52 63. Tesselation [Paramountorkestern] (1951).1.mp3
 2172 -rw-rw-r-- 1 johan johan  2220610 May 10 15:52 64. Bei Mir Bistu Shein [Simon Brehms Orkester] (1951).mp3
 1652 -rw-rw-r-- 1 johan johan  1687597 May 10 15:52 65. What Is This Thing Called Love [Sonya Hedenbratt] (1951).mp3
 2248 -rw-rw-r-- 1 johan johan  2301504 May 10 15:52 67. Smiles [Johan Adolfsson Sextett] (1951).1.mp3
 1836 -rw-rw-r-- 1 johan johan  1878332 May 10 15:52 68. Beat the Clock [Reinhold Svensson Quintet] (1951).1.mp3
 2312 -rw-rw-r-- 1 johan johan  2366867 May 10 15:53 69. Attribution [Kenneth Fagerlunds Kvintett] (1951).1.mp3
 5980 -rw-rw-r-- 1 johan johan  6123365 May 10 15:53 70. Three Without a Key [Ulf Lindes Kvartett] (1951).1.mp3
Manganus commented 8 years ago

A similar exception from the os.listdir() function gets easily provoked:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import os
os.listdir('/mnt/qnap-119/vbr/Singles/svensk jazzhistoria, vol 6 - 1947-1951/')

Results in:


johan@T400s-Jfs ~ $ time /mnt/qnap-212/py/error_hunt.py 
Traceback (most recent call last):
  File "/mnt/qnap-212/py/error_hunt.py", line 4, in <module>
    os.listdir('/mnt/qnap-119/vbr/Singles/svensk jazzhistoria, vol 6 - 1947-1951/')
OSError: [Errno 40] Too many levels of symbolic links: '/mnt/qnap-119/vbr/Singles/svensk jazzhistoria, vol 6 - 1947-1951/'

real    0m0.039s
user    0m0.020s
sys 0m0.008s
johan@T400s-Jfs ~ $
Manganus commented 8 years ago

Finally, I avoided Jack's question:

Also what is the issue with printing to stderr? You can capture it to a file like so if you want:

my_command 2> stderr_file

Maybe it's because I'm inexperienced, daft, or something, but I can not see that it matters in this case.

jackwilsdon commented 8 years ago

Do you think you could run this and provide the output?

find /mnt/qnap-119 -follow -printf ""
sampsyo commented 8 years ago

OK, sounds like we need to catch OSError, either inside prune_dirs or outside of its calls. (It will take some thinking to decide which.)

jackwilsdon commented 8 years ago

The best way to do it IMO would just be to change this:

    # Traverse upward from path.
    ancestors.append(path)
    ancestors.reverse()
    for directory in ancestors:
        directory = syspath(directory)
        if not os.path.exists(directory):
            # Directory gone already.
            continue
        if fnmatch_all(os.listdir(directory), clutter):
            # Directory contains only clutter (or nothing).
            try:
                shutil.rmtree(directory)
            except OSError:
                break
        else:
            break

to this:

    # Traverse upward from path.
    ancestors.append(path)
    ancestors.reverse()
    for directory in ancestors:
        directory = syspath(directory)
        if not os.path.exists(directory):
            # Directory gone already.
            continue
        try:
            if fnmatch_all(os.listdir(directory), clutter):
                # Directory contains only clutter (or nothing).
                shutil.rmtree(directory)
            else:
                break
        except OSError:
            break

i.e. we move the except outside the if statement.

sampsyo commented 8 years ago

Nice; I hadn't looked to see that we already have an exception handler there. If the current behavior is just to silently give up when the filesystem prevents further pruning, then this solution seems perfect.