Please don't abort the entire task when a file is missing

trapd00r commented 3 years ago

Problem

I just enabled the bucket plugin and moving one directory at a time like this:

cd 2/
for x in * ;do beet move artist:"$x"; done

When beets traverses the individual artist directories and stumble upon an album with a missing track, the entire task is aborted. I would rather have beets complain and move on to the next album in the directory instead of simply bailing out:

Moving 13 items (3 already in place).
Error: No such file or directory while moving /mnt/music8/+TAGGED/2/2113 Pritt/2113 Pritt │2020│ Wrist Game 101 [MP3]/04 Ball 2.0.mp3 to /mnt/music8/+TAGGED/0-9/2113 Pritt/2113 Pritt │2020│ Wrist Game 101 [MP3]/04 Ball 2.0.mp3
Moving 40 items (19 already in place).
Error: No such file or directory while moving /mnt/music8/+TAGGED/2/213/213 │2004│ The Hard Way [CD, MP3]/01 Intro.1.mp3 to /mnt/music8/+TAGGED/0-9/213/213 │2004│ The Hard Way [CD, MP3]/01 Intro.mp3
Moving 24 items (5 already in place).
Error: No such file or directory while moving /mnt/music8/+TAGGED/2/21 Savage/21 Savage │2018│ I Am  I Was [CD, MP3]/06 1.5.mp3 to /mnt/music8/+TAGGED/0-9/21 Savage/21 Savage │2018│ I Am  I Was [CD, MP3]/06 1.5.mp3

The missing tracks are (mostly) files I have deleted like this:

find . -type f -regextype awk -iregex '.+[.][0-9][.](mp3|flac|jpg|m3u|ogg|m4a)' -delete

because they are duplicates and leftovers from previous imports.

Now, I know that I'm supposed to run a

beet update to reflect deleted files, however, I've had bad experiences with that in the past so I'm a bit nervous to do so again: https://github.com/beetbox/beets/discussions/3973

Setup

OS: Linux 5.19.6
Python version: 3.9.4
beets version: 1.5.0
Turning off plugins made problem go away (yes/no): well, no

My configuration (output of beet config) is:

# vim:synmaxcol=500:expandtab:fdm=marker:fdl=0:
#      ${HOME}/etc/beets/config.yaml
#   ‗‗‗‗‗‗‗‗‗‗‗‗ ‗‗‗‗‗‗ ‗‗‗‗‗‗‗‗ ‗‗‗‗‗‗‗‗‗‗‗
#         owner  Magnus Woldrich <m@japh.se>
#         btime  2021-05-13 10:31:39
#         mtime  2021-05-31 16:38:07
#   permissions  You are free to use things you may find useful here.
#                Please improve and share.
#           git  http://github.com/trapd00r/configs/  (up-to-date)
#           url  http://japh.se
#           irc  japh@irc.libera.chat #vim #perl #beets
#   ‗‗‗‗‗‗‗‗‗‗‗‗ ‗‗‗‗‗‗‗‗‗‗‗‗‗ ‗‗‗‗ ‗‗‗‗ ‗‗‗‗

#< what does it look like
## Alanis Morissette │2020│ Reckoning [Single, WEB, MP3]
## Alanis Morissette │2020│ Such Pretty Forks in the Road [WEB, FLAC]
## Anna Ternheim │2003│ My Secret [EP, CD, MP3]
## Anna Ternheim │2004│ Somebody Outside [CD, MP3]
## Anna Ternheim │2005│ Shoreline EP [EP, CD, MP3]
##  ├── 01 Shoreline (radio Version).mp3
##  ├── 02 Little Lies.mp3
##  ├── 03 China Girl.mp3
##  ├── 04 When Tomorrow Comes.mp3
##  ├── 05 Anywhere I Lay My Head.mp3
##  └── cover.jpg

## A/Anna Ternheim/Anna Ternheim │2003│ My Secret [EP, CD, MP3]
#   '- 01 My Secret.mp3
#   '- 02 All For Me.mp3
#   '- 03 A Voice To Calm You Down.mp3
#   '- 04 I Say No (gotland Version).mp3
#   '- 05 Wedding Song (demo Version).mp3
#   '- cover.jpg
#>

#< core options
directory:  /mnt/music8/+TAGGED
library:    ~/var/beets/beets202105.db
pluginpath: /usr/lib/python3.9/site-packages/beetsplug/

# files matching these patterns are deleted from source after import
clutter: ["Thumbs.DB", ".DS_Store", ".m3u", ".pls",
          ".jpg", ".nfo", ".txt", ".log", ".gif",
         ]

replace:
    '[\\]': ''
    '[_]': '-'
    '[/]': '-'
    '^\.': ''
    '[\x00-\x1f]': ''
    '[<>:"\?\*\|]': ''
    '\.$': ''
    '\s+$': ''
    '^\s+': ''
    '^-': ''

ignore: [".*", "*~", "System Volume Information"]

art_filename:          cover # cover.jpg
#asciify_paths:         yes
format_item:           $path
per_disc_numbering:    false
sort_album:            path+
sort_case_insensitive: yes
sort_item:             path+
threaded:              yes
timeout:               5.0
verbose:               no

# <importer
import:
  languages:        en
  write:            yes
  default_action:   apply
#  remove causes a crash: https://github.com/beetbox/beets/issues/716
  duplicate_action: keep
  non_rec_action:   ask
  autotag:          yes
  move:             yes
#  quiet_fallback:   asis # when using the -q flag
  quiet_fallback:   skip # when using the -q flag

  # Either yes or no, controlling whether imported directories are recorded
  # and whether these recorded directories are skipped. This corresponds to
  # the -i flag to beet import.
  incremental:      yes
#>
#< plugins
#plugins: [ # mosaic
#  'fetchart',   'discogs',  'fromfilename', 'inline',     'smartplaylist',
#  'ftintitle',  'info',     'lastgenre',    'lastimport', 'thumbnails',
#  'mpdupdate',  'mpdstats', 'rewrite',      'duplicates', 'missing',
#  'extrafiles', 'edit',     'lyrics',       'mpdqueue',
#]

plugins: [
  'bucket',
  'discogs',
  'duplicates',
#  'edit',
#  'extrafiles',
  'fetchart',
  'fromfilename',
  'ftintitle',
  'info',
  'inline',
  'lastgenre',
  'lastimport',
#  'lyrics', # takes forever
#  'missing',
#  'mpdqueue',
#  'mpdstats',
  'mpdupdate',
  'rewrite',
#  'smartplaylist', # only run it once in a while
]

bucket:
  bucket_alpha:
  - '#-!'
  - '0-9'
  - 'A'
  - 'B'
  - 'C'
  - 'D'
  - 'E'
  - 'F'
  - 'G'
  - 'H'
  - 'I'
  - 'J'
  - 'K'
  - 'L'
  - 'M'
  - 'N'
  - 'O'
  - 'P'
  - 'Q'
  - 'R'
  - 'S'
  - 'T'
  - 'U'
  - 'V'
  - 'W'
  - 'X'
  - 'Y'
  - 'Z'
  bucket_alpha_regex:
    '#-!': ^[^0-9a-zA-ZåÅäÄöÖ]
  bucket_year: []
  extrapolate: no

bpd:
  host: 127.0.0.1
  port: 6601

edit:
  itemfields: track title artist album year
  albumfields: track title artist albumartist album year

extrafiles:
  patterns:
    single_tracks:
      - '+tracks/'
      - '_tracks/'
    single_live:
      - '_live'
      - '+live'
  paths:
    single_tracks: $artist/+tracks
    single_live: $artist/+live

rewrite:
  album .*Sommar i P1:               P1 Sommar
  album .*Sommar och Vinter i P1.*:  P1 Sommar
  album Söndagsintervjun:            P1 Söndagsintervjun
  album .*Musikguiden i p3:          P3 Musikguiden
  album Jukeboxen i p4:              P4 Jukeboxen
  album Musikspecial i p4:           P4 Musikspecial
#  albumartist Various Artists:       VA
  artist Pst.q:                     Pst-q
  artist 10,000 Maniacs:            10000 Maniacs
  artist Fronda.*:                   Fronda
  artist Magnus Rytterstam.*:        Magnus Rytterstam
  artist 江海迦:                     Aga
  artist Whoo Kid:                   DJ Whoo Kid
  artist Looptroop.*:                Looptroop
  artist T.R:                        Öris
  artist Organismen:                 Organism 12
  artist Gms.*:                      GMS
  artist (tupac|2[pP]ac).*:          2pac
  artist .*weird Al.*:               Weird Al Yankovic
  artist .*Green Lantern.*:          DJ Green Lantern
  artist .ingenting.:                Ingenting
  artist Sin[eé]ad O.Connor.*:       Sinéad O'Connor
  artist .*Suzanne.*Vega:            Suzanne Vega
  artist .*1[23]00 mic.*:            1200 Micrograms
  artist elin (ruth)? sigvardsson:   Elin Sigvardsson
  artist elin ruth:                  Elin Sigvardsson
  artist ^Game$:                     The Game
  artist ^Ken$:                      Ken Ring
  artist Special D:                  Special D.
  artist Danne W.*:                  Sjätte Sinnet
  artist Sjatte Sinnet:              Sjätte Sinnet
  artist Ante Barazza:               Sjätte Sinnet

mpd:
  host: localhost
  port: 6600
#  music_directory: /mnt/music8

thumbnails:
  auto: yes # default

lastfm:
  user: betbot

lastgenre:
  auto: yes # default
  canonical: yes
  force: no
  source: artist

ftintitle:
  auto: yes # default

smartplaylist:
# it's better to set save_absolute_paths_in_playlist option in mpd.conf
#  relative_to: ~/mp3/
  playlist_dir: ~/mp3/_playlists
  playlists:
    - name: '+all.m3u'
      query: ''

    - name: 'eminem.m3u'
      query: 'artist:Eminem'

#    - name: 'sm %lower{$genre}.m3u'
#      query: ''

    - name: '$year.m3u'
      query: 'year::(199[0-9]|200[0-9]|201[0-9])'

    - name: '+decent.m3u'
      query: 'play_count:1..'

    - name: '+wow.m3u'
      query: 'play_count:5..'

    - name: 'psychedelic'
      query: 'genre:psychedelic'
    - name: 'loved.m3u'
      query: 'loved:1'

musicbrainz:
  searchlimit: 10

missing:
  format: "$path"

lyrics:
  auto: yes
#  fallback: ''

fetchart:
  auto: yes
  sources: coverart itunes amazon albumart wikipedia google

embedart:
  auto: yes
#>
#< path setup
aunique:
  disambuguators: media mastering label catalognum albumdisambig releasegroupdisambig

match:
  preferred:
    media: ['CD', 'Digital Media|File', 'Vinyl']

paths:
#    mb_trackid::^$:              +unmatched/
# https://www.japh.se/2021/05/23/custom-beet-path-rules-for-record-labels.html
#    label:8bitpeoples:           8/8bitpeoples/%if{$hasyear,${year}} %title{$albumartist} - %title{$album}/${padded_tracknr} %title{$title}
    tag:8bitpeoples:             0-9/8bitpeoples/%if{$hasyear,${year}} $first_artist - %title{$album}/${padded_tracknr} %title{$title}
    label:whoa.nu:               W/Whoa.nu/$base_name
    label:rap_swe:               +swe_hiphop/$base_name
    label:randombastards:        R/Randombastards/$base_name
    label:Norrköping:            S/Sjätte Sinnet/+tracks/$base_name
    tag:rosamannen:              +live/$artist - $album/$base_name
#    tag:Rosamannen:              +live/+rosamannen/$base_name
    label:frizon.info:           F/Frizon.info/$base_name
    label:"Masters of Hardcore": M/Masters of Hardcore/$moh_catalog %if{$hasyear,${year}} %title{$album}/${padded_tracknr} %title{$artist} - %title{$title}
    label:gamesoundtrack:        +game/+ost/$album%if{$hasyear, (${year})}/${padded_tracknr} $artist - %title{$title}
    label:ocremix:               +game/+ocremix/$base_name
    tag:overlooked:              +game/+overlookedremix/$base_name
    label:amigaremix:            +game/+amigaremix/$base_name
#    tag:gamealbums:            +game/$first_artist - %title{$album}%if{$hasyear, (${year})}/${padded_tracknr} $artist - %title{$title}
    tag:gamealbums:            +game/%title{$album} [$albumartist]%if{$hasyear, (${year})}/${padded_tracknr} $artist - %title{$title}
#    label:gamealbums:            +game/$first_artist - %title{$album}%if{$hasyear, (${year})}/${padded_tracknr} $artist - %title{$title}
    label:game:                  +game/+tracks/$base_name
    tag:sr:                      +radio/Sveriges Radio/%title{$album}/%title{$title}
#    label:radio:                 +radio/%title{$first_artist}/%title{$album}/%title{$title}
    label:demo:                  %bucket{$first_artist, alpha}/%title{$first_artist}/+demo/$base_name
    label:live:                  %bucket{$first_artist, alpha}/%title{$first_artist}/+live/%title{$albumartist}%if{$hasyear, │${year}│} %title{$album} [$alb_type$media_type$format]/${padded_tracknr} %title{$title}
#    label:test:                  +test/%title{$artist}/%title{$album}/%title{$title}
#    albumtype:mixtape:           +mixtape/$mixtape_album [$first_artist]%if{$hasyear, (${year})}/${padded_tracknr} $artist - %title{$title}
    tag:mixtape:                 +mixtape/$mixtape_album [$first_artist]%if{$hasyear, (${year})}/${padded_tracknr} $artist - %title{$title}
#    albumtype:mixtape:           +mixtape/$mixtape_album%if{$hasyear, (${year})}/${padded_tracknr} $artist - %title{$title}
#    default:                     %upper{%left{$albumartist,1}}/%title{$first_artist}/%title{$albumartist}%if{$hasyear, │${year}│} %title{$album} [$alb_type$media_type$format]/${padded_tracknr} %title{$title}
    albumtype:soundtrack:        +OST/%if{$hasyear,│${year}│}$album/${padded_tracknr} $artist - %title{$title}
    default:                     %bucket{$albumartist, alpha}/%title{$first_artist}/%title{$albumartist}%if{$hasyear, │${year}│} %title{$album} [$alb_type$media_type$format]/${padded_tracknr} %title{$title}
    comp:                        +VA/%title{$album}%if{$hasyear, (${year})}/${padded_tracknr} $artist - %title{$title}

# https://www.japh.se/2021/05/26/how-to-add-singletons-to-artist-dir-correctly-in-beets.html
    singleton:                   %upper{%left{$artist,1}}/%title{$first_artist_singleton}/+tracks/$base_name

# I want bootlegs etc in the same setup as default
    albumtype:other:             %upper{%left{$albumartist,1}}/%title{$first_artist}/%title{$albumartist}%if{$hasyear, │${year}│} %title{$album} [$alb_type$media_type$format]/${padded_tracknr} %title{$title}
#>
#< inline fun!
item_fields:
# pad track number with zero if < 10
  padded_tracknr: "'{:02n}'.format(track)"
  moh_catalog: catalognum.replace(" ", "")

# capture first artist as primary artist to avoid directories like this:
# · B/Britney Spears/
# · B/Britney Spears feat Madonna/
# · B/Britney Spears vs Metallica/
# > https://github.com/beetbox/beets/issues/3176
# > https://www.japh.se/2021/06/01/capture-primary-artist-as-a-separate-field-in-beets.html
#
# handles:
# · Artist,
# · Artist &
# · Artist feat
# · Artist feat.
# · Artist featuring
# · Artist ft.
# · Artist vs
# · Artist vs.
# · Artist &
#
# The idea is to use $first_artist in the beginning of the path format
# like so:
#
# %title{$first_artist}/%title{$albumartist}
#
# which will put 'Jennifer Lopez feat. Pitbull' inside the main Jennifer
# Lopez directory, but still keep the feat. part in the directory name
# inside it.
#
# J/Jennifer Lopez Feat. Pitbull/Jennifer Lopez Feat. Pitbull │2012│ Dance Again [Single, WEB, MP3]/01 Dance Again.mp3
# -> J/Jennifer Lopez/Jennifer Lopez Feat. Pitbull │2012│ Dance Again [Single, WEB, MP3]/01 Dance Again.mp3

  first_artist: |
    import re
    return re.split(',|\s+(feat(.?|uring)|&|(Vs|Ft).)', albumartist, 1, flags=re.IGNORECASE)[0]

  first_artist_singleton: |
    import re
    return re.split(',|\s+(feat(.?|uring)|&|(Vs|Ft).)', artist, 1, flags=re.IGNORECASE)[0]

# file basename for singletons import - import as is, minus the extension.
  base_name: |
    import os.path
    base = os.path.basename(path)
    return os.path.splitext(base)[0]

album_fields:
  mixtape_album: |
    import re
    album_fixed = album
    return re.sub(r"G.unit Radio,?\s+(Pt|Part)[.]?\s*(.*)", r"G-Unit Radio \2", album_fixed, flags=re.IGNORECASE)

  alb_status: |
    # MB returns 4 values describing how "offical" a release is, they are:
    # Official, Promotional, Bootleg, and Pseudo-Release
    # We only note the middle two.
    # https://musicbrainz.org/doc/Release#Status
    if 'Promo' in albumstatus:
      return 'Promo'
    elif 'Bootleg' in albumstatus:
      return 'Bootleg'
    else:
      return None
  # Check if https://github.com/beetbox/beets/issues/2200 affects below
  alb_type: |
    alb_types = ""
    albumtypes_list = {
      'ep': 'EP, ',
      'single': 'Single, ',
      'live': 'Live, ',
      'remix': 'Remix, ',
      'dj-mix': 'DJ-mix, ',
      'mixtape/street': 'Mixtape, ',
      'interview': 'Interview, ',
    }
    for key, value in albumtypes_list.items():
      if albumtype == key:
        alb_types += str(value)
      if alb_types is not None:
        return alb_types
      else:
        return None

  media_type: |
      # https://musicbrainz.org/doc/Release/Format
      # Lets Merge the variations of the same medium into the main medium name (Opinonated)
      media_list = {
       '12" Vinyl': 'VINYL, ',
       '10" Vinyl': 'VINYL, ',
       '7" Vinyl': 'VINYL, ',
       'Cassette': 'CASSETTE, ',
       'Digital Media': 'WEB, ',
       'CD': 'CD, ',
       'File': 'WEB, ',
      }
      # Lets omit these instead of converging them under a similar label like above (Opinonated)
      media_types_to_omit = ['Enhanced CD', 'CDDA', 'Blu-spec CD', 'SHM-CD', 'HQCD', '']
      if items[0].media in media_list:
        return str(media_list[items[0].media])
      elif items[0].media in media_types_to_omit:
        return None
      else:
        return str(items[0].media)

  hasyear: 1 if year > 0 else 0
#>
#< autotagger
# To control how tolerant the autotagger is of differences, use the
# strong_rec_thresh option, which reflects the distance threshold below
# which beets will make a “strong recommendation” that the metadata
# be used.
#
# default is 0.04
match:
  strong_rec_thresh: 0.10
  medium_rec_thresh: 0.25
  ignored:           missing_tracks unmatched_tracks
  ignored_media:     ['Data CD', 'DVD', 'DVD-Video', 'Blu-ray', 'HD-DVD',
                      'VCD', 'SVCD', 'UMD', 'VHS']
#>
#< ui
ui:
  color: yes
  colors:
    text_success:         green
    text_warning:         yellow
    text_error:           red
    text_highlight:       blue
    text_highlight_minor: lightgray
    action_default:       turquoise
    action:               blue
#>

trapd00r commented 3 years ago

I can do

beet update -M path:./

in the meantime, but I still think it's broken behaviour to abort the entire task. :)

wisp3rwind commented 3 years ago

Sounds like a reasonable improvement --- I guess beets is generally not very good at gracefully handling errors. In many cases, that's probably for the better, since an appropriate action is not easy to infer. But simply skipping ahead to the next album in this specific situation should be fine.

sampsyo commented 3 years ago

Indeed; this would probably be a good idea. The default behavior in most beets actions is "fail-stop" to avoid cases where forging ahead blindly would just mess things up even further, such as leaving albums partially split across directories. But beet move actually seems like a perfect match for a "keep on truckin'" policy instead, since having a few immovable files in the target set is unlikely to be indicative of some larger problem.

wisp3rwind commented 3 years ago

Actually, the best thing to do would probably to check for all relevant files to exist before starting to move an object (album or item, depending on the command), and skip the object if something is missing. There's of course a race if any other tool is simultaneously modifying these files. Then, if nevertheless an error occurred when doing the actual move, the "fail-stop" would be sensible (since it might indicate such a concurrent program, or out-of-disk-space, or a flaky connection to the disk, etc). I might have a look at implementing this.

sampsyo commented 3 years ago

Yeah. At the risk of overcomplicating things, what you're describing is sort of a "transactional" file system interface… ideally, we could batch up several copy/move/etc. operations and ask that either they all complete or none complete. Of course, true transactions are pretty much impossible without support from the OS, but maybe we would want a little utility to do sorta-atomic move/copy operations on sets of files using this "check permissions first, then really do the operation" strategy.

beetbox / beets

Please don't abort the entire task when a file is missing #3986

Problem

Setup