Data loss during Beets import

[chiamarc]

Problem

Ran a previous non-tagging import with

beet import -A D:\Audio\Music

completed successfully. Some time later I ran a new import with

beet import D:\Audio\Music

Many albums were matched 100% and of course there were many albums already in the library.

At some point, I chose to "Remove old" of an Ace of Base album with unmatched tracks (there were duplicates in that directory somehow):

D:\Audio\Music\Ace of Base\Greatest Hits (33 items)
Tagging:
    Ace of Base - Greatest Hits
URL:
    https://musicbrainz.org/release/455f00f9-ce80-4893-8e62-bbbb0c645873
(Similarity: 85.8%) (unmatched tracks) (Digital Media, 2008, XE)
 * Lucky Love 2009
 * The Sign
 * All That She Wants
 * Wheel of Fortune
 * Lucky Love
 * Beautiful Life
 * Happy Nation (Faded edit)
 * Life Is a Flower
 * Don't Turn Around
 * Hallo Hallo
 * Always Have, Always Will
 * Cruel Summer (Big bonus mix)
 * Unspeakable
 * C'est la Vie (Always 21)       -> C'est la vie (Always 21)
 * Living in Danger (single edit)
 * Beautiful Morning
 * Da Capo
 * Whenever You're Near Me
 * Everytime It Rains
 * Love In December               -> Love in December
Unmatched tracks (13):
 ! All That She Wants (#1) (3:31)
 ! Everytime It Rains (#3) (4:17)
 ! Beautiful Life (#4) (3:39)
 ! Cruel Summer (#5) (3:33)
 ! Don't Turn Around (#6) (3:48)
 ! Lucky Love (Acoustic Version) (#7) (2:53)
 ! Always Have, Always Will (#8) (3:48)
 ! Life Is A Flower (#9) (3:45)
 ! C'est La Vie (#10) (3:26)
 ! C'est La Vie (Always 21) (#10) (3:26)
 ! Lucky Love (Frankie Knuckles Mix) (#11) (3:42)
 ! Beautiful Life (Junior Vasquez Mix) (#12) (8:26)
 ! Beautiful Life (Junior Vazquez Mix) (#12) (8:26)
[A]pply, More candidates, Skip, Use as-is, as Tracks, Group albums,
Enter search, enter Id, aBort?
This album is already in the library!
Old: 31 items, MP3, 320kbps, 114:03, 262.8 MiB
New: 20 items, MP3, 320kbps, 69:10, 159.6 MiB
[S]kip new, Keep both, Remove old? r

A few seconds later I received the following errors (I had seen this kind before but just ignored them):

error reading D:\Audio\Music\Ace of Base\Greatest Hits\11 Always Have, Always Will.mp3: [Errno 2] No such file or directory: u'\\\\?\\D:\\Audio\\Music\\Ace of Base\\Greatest Hits\\11 Always Have, Always Will.mp3'
error reading D:\Audio\Music\Ace of Base\Greatest Hits\19 Everytime It Rains.mp3: [Errno 2] No such file or directory: u'\\\\?\\D:\\Audio\\Music\\Ace of Base\\Greatest Hits\\19 Everytime It Rains.mp3'
error reading D:\Audio\Music\Ace of Base\Greatest Hits\03 All That She Wants.mp3: [Errno 2] No such file or directory: u'\\\\?\\D:\\Audio\\Music\\Ace of Base\\Greatest Hits\\03 All That She Wants.mp3'
error reading D:\Audio\Music\Ace of Base\Greatest Hits\14 C'est la vie (Always 21).mp3: [Errno 2] No such file or directory: u"\\\\?\\D:\\Audio\\Music\\Ace of Base\\Greatest Hits\\14 C'est la vie (Always 21).mp3"
error reading D:\Audio\Music\Ace of Base\Greatest Hits\13 Unspeakable.mp3: [Errno 2] No such file or directory: u'\\\\?\\D:\\Audio\\Music\\Ace of Base\\Greatest Hits\\13 Unspeakable.mp3'
error reading D:\Audio\Music\Ace of Base\Greatest Hits\08 Life Is a Flower.mp3: [Errno 2] No such file or directory: u'\\\\?\\D:\\Audio\\Music\\Ace of Base\\Greatest Hits\\08 Life Is a Flower.mp3'
error reading D:\Audio\Music\Ace of Base\Greatest Hits\17 Da Capo.mp3: [Errno 2] No such file or directory: u'\\\\?\\D:\\Audio\\Music\\Ace of Base\\Greatest Hits\\17 Da Capo.mp3'
error reading D:\Audio\Music\Ace of Base\Greatest Hits\15 Living in Danger (single edit).mp3: [Errno 2] No such file or directory: u'\\\\?\\D:\\Audio\\Music\\Ace of Base\\Greatest Hits\\15 Living in Danger (single edit).mp3'
error reading D:\Audio\Music\Ace of Base\Greatest Hits\06 Beautiful Life.mp3: [Errno 2] No such file or directory: u'\\\\?\\D:\\Audio\\Music\\Ace of Base\\Greatest Hits\\06 Beautiful Life.mp3'
error reading D:\Audio\Music\Ace of Base\Greatest Hits\12 Cruel Summer (Big bonus mix).mp3: [Errno 2] No such file or directory: u'\\\\?\\D:\\Audio\\Music\\Ace of Base\\Greatest Hits\\12 Cruel Summer (Big bonus mix).mp3'
error reading D:\Audio\Music\Ace of Base\Greatest Hits\18 Whenever You're Near Me.mp3: [Errno 2] No such file or directory: u"\\\\?\\D:\\Audio\\Music\\Ace of Base\\Greatest Hits\\18 Whenever You're Near Me.mp3"
error reading D:\Audio\Music\Ace of Base\Greatest Hits\05 Lucky Love.mp3: [Errno 2] No such file or directory: u'\\\\?\\D:\\Audio\\Music\\Ace of Base\\Greatest Hits\\05 Lucky Love.mp3'
error reading D:\Audio\Music\Ace of Base\Greatest Hits\04 Wheel of Fortune.mp3: [Errno 2] No such file or directory: u'\\\\?\\D:\\Audio\\Music\\Ace of Base\\Greatest Hits\\04 Wheel of Fortune.mp3'
error reading D:\Audio\Music\Ace of Base\Greatest Hits\16 Beautiful Morning.mp3: [Errno 2] No such file or directory: u'\\\\?\\D:\\Audio\\Music\\Ace of Base\\Greatest Hits\\16 Beautiful Morning.mp3'
error reading D:\Audio\Music\Ace of Base\Greatest Hits\01 Lucky Love 2009.mp3: [Errno 2] No such file or directory: u'\\\\?\\D:\\Audio\\Music\\Ace of Base\\Greatest Hits\\01 Lucky Love 2009.mp3'
error reading D:\Audio\Music\Ace of Base\Greatest Hits\20 Love in December.mp3: [Errno 2] No such file or directory: u'\\\\?\\D:\\Audio\\Music\\Ace of Base\\Greatest Hits\\20 Love in December.mp3'
error reading D:\Audio\Music\Ace of Base\Greatest Hits\10 Hallo Hallo.mp3: [Errno 2] No such file or directory: u'\\\\?\\D:\\Audio\\Music\\Ace of Base\\Greatest Hits\\10 Hallo Hallo.mp3'
error reading D:\Audio\Music\Ace of Base\Greatest Hits\07 Happy Nation (Faded edit).mp3: [Errno 2] No such file or directory: u'\\\\?\\D:\\Audio\\Music\\Ace of Base\\Greatest Hits\\07 Happy Nation (Faded edit).mp3'
error reading D:\Audio\Music\Ace of Base\Greatest Hits\02 The Sign.mp3: [Errno 2] No such file or directory: u'\\\\?\\D:\\Audio\\Music\\Ace of Base\\Greatest Hits\\02 The Sign.mp3'

The result was that ALL but two of the tracks in that folder have been deleted (before my very eyes actually). I have no idea how that unicode string got there for those files. Nothing was changed from under beets (e.g., files deleted) after the import had already begun. I will say that the D: drive is a mounted Stablebit DrivePool (a poor person's file-level RAID solution) that might be returning unexpected path values to Python. But if that is the case, why does the error only occur on some albums?

This may have been a fluke but it would be nice to have beets "delete" by moving to Recycle Bin or an equivalent folder of choice on Linux.

Setup

• OS: Windows 10
• Python version: 2.7.13
• beets version: 1.4.5
• Turning off plugins made problem go away (yes/no): Didn't try

My configuration (output of beet config) is:

lyrics:
    bing_lang_from: []
    bing_client_secret: REDACTED
    google_API_key: REDACTED
    force: no
    google_engine_ID: REDACTED
    sources: [google, lyricwiki, musixmatch]
    auto: yes
    fallback:
    genius_api_key: REDACTED
    bing_lang_to:

paths:
    default: $albumartist/$album%aunique{}/$track $title
    comp: $albumartist/$album%aunique{}/$track $title ($artist)
threaded: yes
item_fields:
    multidisc: 1 if disctotal > 1 else 0
per_disc_numbering: no
library: D:\Audio\beets_library.db
lastgenre:
    count: 3
    source: album
    prefer_specific: no
    force: yes
    min_weight: 10
    auto: yes
    whitelist: yes
    separator: ', '
    fallback:
    canonical: no

plugins: discogs duplicates fromfilename ftintitle fuzzy inline lastgenre lyrics missing
directory: D:\Audio\Music

import:
    move: yes
    copy: no
    write: yes
    resume: yes
    detail: yes
    log: D:\Audio\beets.log
pathfields: {}
album_fields: {}
missing:
    count: no
    album: no
    total: no
fuzzy:
    threshold: 0.7
    prefix: '~'
ftintitle:
    auto: yes
    drop: no
    format: feat. {0}
duplicates:
    count: no
    full: no
    format: ''
    keys: []
    move: ''
    tag: ''
    path: no
    copy: ''
    tiebreak: {}
    album: no
    strict: no
    checksum: ''
    merge: no
    delete: no
discogs:
    tokenfile: discogs_token.json
    user_token: REDACTED
    apikey: REDACTED
    apisecret: REDACTED
    source_weight: 0.5

[sampsyo]

Hi! This is very worrisome, and I think it's an instance of #702—an extremely nasty and hard-to-solve bug. Can you take a look and check whether it looks like the same issue?

[chiamarc]

Yes, this was obviously happening on an album that had already been imported (and had probably produced duplicates somehow due to various choices I made earlier). The read error is weird though because the files were there and not changed while beets was importing, unless maybe the multithreading had something to do with it. Is Beets designed to permanently delete files? If so, this bug may be mitigated somewhat by being able to recover from this situation; a kind of undo last operation would be helpful, where you write the changes temporarily somewhere and swap out on a "commit". Either that or sending to Recycle Bin, which I imagine would be much easier.

It seems from the description of #702 that you know the situations in which this occurs and so it is reproducible. The question is how do you solve it? I can try to run it on a couple scenarios and report back but I don't have a lot of bandwidth right now.

[chiamarc]

OK, so this is quite vexing. Right now I'm importing into a brand new database. Here was the sequence of operations (in a single session, no previous imports):

D:\Audio\Music\Daft Punk\Around The World (CD Maxi-Single) (4 items)
Correcting tags from:
    Daft Punk - Around The World (CD Maxi-Single)
To:
    Daft Punk - Around the World
URL:
    https://musicbrainz.org/release/b8243e29-2672-4b60-8ae0-7f3e793a8a41
(Similarity: 94.0%) (album) (CD, 1997, GB)
 * Around The World (Radio Edit) ->
   Around the World (radio edit)
 * Around The World (LP Version) ->
   Around the World (LP version)
 * Teachers (Extended Mix) ->
   Teachers (extended mix)
 * Around The World (Motorbass Vice Mix) ->
   Around the World (Motorbass Vice mix)
[A]pply, More candidates, Skip, Use as-is, as Tracks, Group albums,
Enter search, enter Id, aBort? a

D:\Audio\Music\Daft Punk\Around The World (CD Maxi-Single, Promo) (5 items)
Correcting tags from:
    Daft Punk - Around The World (CD Maxi-Single, Promo)
To:
    Daft Punk - Around the World
URL:
    https://musicbrainz.org/release/1c0eaaf9-0852-4232-8dc1-e53597a9b441
(Similarity: 95.2%) (tracks, album) (CD, 1996, US, Virgin Records America)
 * Around The World (Radio Edit) ->
   Around the World (radio edit)
 * Around The World (Tee's Frozen Sun Mix) ->
   Around the World (Tee's Frozen Sun mix)
 * Around The World (Motorbass Vice Mix) ->
   Around the World (Motorbass Vice mix)
 * Around The World (Daft Punk Edit) ->
   Around the World (Daft Punk edit)
 * Around The World (Call Out Research Hook) ->
   Call Out Research Hook (title)
[A]pply, More candidates, Skip, Use as-is, as Tracks, Group albums,
Enter search, enter Id, aBort?
This album is already in the library!
Old: 4 items, MP3, 320kbps, 23:42, 54.6 MiB
New: 5 items, MP3, 320kbps, 22:05, 51.0 MiB
[S]kip new, Keep both, Remove old? r

Both paths are now gone! This bug has to be fixed somehow or I will need to stop using Beets.

I really like Beets so I'm willing to help debug in any way I can. I'm familiar with Python but by no means a developer. I cut my teeth on gdb way back in the day so I'm no stranger to debugging but I'll probably need some help.

Please let me know how I can assist.

[sampsyo]

Whoa; that sounds like a different and even more mysterious problem. It sounds like just importing two "copies" of the same album back-to-back and choosing R removed both copies. That won't do at all.

Is this reliably reproducible? Can you check whether the same thing happens when you use the default import settings (i.e., copy is true and move is false) vs. your current settings?

[chiamarc]

OK, I have to find a similar situation...it might take me a bit.

[chiamarc]

I can confirm that the default settings of copy:true and move:false produces a new directory containing the correct filenames and tags but doesn't remove the original directory or files. This is acceptable, if inconvenient.

[sampsyo]

That's also strange—so in that case, the "remove" option just behaves like a no-op? So both modes seem to do something wrong.

How about the other question: with move mode enabled, is the data loss (both albums are deleted) reproducible?

[sourcefilter]

Okay, I can confirm that songs were completely deleted doing a two-stage import (first a full non-interactive import, then an interactive one to fix metadata). Using version 1.4.5 installed via pip on a mac.

[sampsyo]

@andrewbenedictwallace Can you please provide instructions to reproduce the problem?

[sourcefilter]

I'll work on reproducing when I get a chance but I think it happened when I tried to fix an issue by moving files around outside of beets.

[stale[bot]]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.