Track Ordering Within Albums is Incorrect

spt5007 commented 8 years ago

I have the following Album on my USB drive:

title: Horn Intro       artist: Modest Mouse
track: 1        genre: Rock (id 17)
title: The World At Large       artist: Modest Mouse
track: 2        genre: Rock (id 17)
title: Float On     artist: Modest Mouse
track: 3        genre: Rock (id 17)
title: Ocean Breathes Salty     artist: Modest Mouse
track: 4        genre: Rock (id 17)
title: Dig Your Grave       artist: Modest Mouse
track: 5        genre: Rock (id 17)
title: Bury Me With It      artist: Modest Mouse
track: 6        genre: Rock (id 17)
title: Dance Hall       artist: Modest Mouse
track: 7        genre: Rock (id 17)
title: Bukowski     artist: Modest Mouse
track: 8        genre: Rock (id 17)
title: This Devil's Workday     artist: Modest Mouse
track: 9        genre: Rock (id 17)
title: The View     artist: Modest Mouse
track: 10       genre: Rock (id 17)
title: Satin In A Coffin        artist: Modest Mouse
track: 11       genre: Rock (id 17)
title: Interlude (Milo)     artist: Modest Mouse
track: 12       genre: Rock (id 17)
title: Blame It On The Tetons       artist: Modest Mouse
track: 13       genre: Rock (id 17)
title: Black Cadillacs      artist: Modest Mouse
track: 14       genre: Rock (id 17)
title: One Chance       artist: Modest Mouse
track: 15       genre: Rock (id 17)
title: The Good Times Are Killing Me        artist: Modest Mouse
track: 16       genre: Rock (id 17)

I run DapGen on my drive thusly:

./DapGen.py /media/user/Music/

I notice when I play the album on my deck (KDC-BT31BU) that the tracks are in reverse order. I look at the database file via:

./KenwoodDBReader.py -i /media/user/Music/kenwood.dap/kenwood.dap

    Album- 0x0017: 'Good News For People Who Love Bad News'
            0x0017: Title- 'The Good Times Are Killing Me'; genre 0015; performer 0030; album 0017
            0x0017: Title- 'One Chance'; genre 0015; performer 0030; album 0017
            0x0017: Title- 'Black Cadillacs'; genre 0015; performer 0030; album 0017
            0x0017: Title- 'Blame It On The Tetons'; genre 0015; performer 0030; album 0017
            0x0017: Title- 'Interlude (Milo)'; genre 0015; performer 0030; album 0017
            0x0017: Title- 'Satin In A Coffin'; genre 0015; performer 0030; album 0017
            0x0017: Title- 'The View'; genre 0015; performer 0030; album 0017
            0x0017: Title- 'This Devil's Workday'; genre 0015; performer 0030; album 0017
            0x0017: Title- 'Bukowski'; genre 0015; performer 0030; album 0017
            0x0017: Title- 'Dance Hall'; genre 0015; performer 0030; album 0017
            0x0017: Title- 'Bury Me With It'; genre 0015; performer 0030; album 0017
            0x0017: Title- 'Dig Your Grave'; genre 0015; performer 0030; album 0017
            0x0017: Title- 'Ocean Breathes Salty'; genre 0015; performer 0030; album 0017
            0x0017: Title- 'Float On'; genre 0015; performer 0030; album 0017
            0x0017: Title- 'The World At Large'; genre 0015; performer 0030; album 0017
            0x0017: Title- 'Horn Intro'; genre 0015; performer 0030; album 0017

This leads me to believe that DapGen isn't respecting the track number in the media file.

chrrrisw commented 8 years ago

Hi @spt5007, thanks for your feedback. I must admit that I'd missed this, though I've thought occasionally that the tracks seem to be in a different order than I expected.

My track files have the track number at the start of the filename, and have usually been copied in bulk, so the order of the files on the USB drive are probably in track order in the directory table, and hence mostly play correctly. It's curious that your track order, in this case, is the exact reverse.

I'll take a look, though I can't promise an instant fix (I have to work this weekend).

Regards, Chris.

spt5007 commented 8 years ago

Thanks for your work on this project @chrrrisw. It enables me to update the music on my drive from linux (or any os really) and that's very helpful!

I'm not in a hurry for a fix. I've looked at the code and bit and I'm considering fixing it myself if I can find the time. If you need any more information from me don't hesitate to ask.

Thanks, Sean

spt5007 commented 8 years ago

Also, I forgot to mention that I'm running e3c27133e7ec7770583006eaccc602533430dc85

chrrrisw commented 8 years ago

Thanks Sean, the project solved the same issue for me also (and was one of the projects I set myself to learn Python, also). Regards, Chris.

chrrrisw commented 8 years ago

Hi @spt5007, I've discovered it's not so simple. I have an album that has two discs, which means that track numbers are duplicated within the album. Although the best way to resolve this is to look at the TPOS (Part of a set) ID3v2 tag to order by disc number also, this is not currently available through the hsaudiotag library that I'm using (at first glance, anyway). The 794a824 commit just made will print out duplicated track numbers, but won't otherwise affect the output database.

spt5007 commented 8 years ago

I see the same problem when I run 794a824 against my music. Unfortunate about hsaudiotag3k. I wonder why there is no TPOS attribute. I suppose it's not standard in the other file types it supports.

How about deriving a new album title from the original title + TPOS? Now each disc of multipart albums is treated separately. Alternatively this process could be done manually by the user at the tag level. I don't have so many multidisc albums that I'd consider this all that cumbersome.

Another idea is to consider a different tagging library. I used mutagen some years ago and it appears under active development. That's probably a big rewrite but I figured I'd mention it. Thanks again for your help @chrrrisw. With any luck things will slow down for me and I'll have more time to help.

chrrrisw commented 8 years ago

Hi Sean, I've been looking into adding TPOS to hsaudiotag3k, and it doesn't look too difficult - so I might try that avenue first. Thanks for your offer to help and for your input so far. Cheers, Chris.

chrrrisw commented 8 years ago

Hi @spt5007 , there's been some progress on getting disc number into hsaudiotag3k, so I should be able to progress this soon. If not, I'm working on a mutagen-based branch also. Regards, Chris.

chrrrisw commented 8 years ago

Hi all, I have a pull request at:

https://github.com/hsoft/hsaudiotag/pull/3

to add disc numbers to hsaudiotag3k, which (with the latest push to this repository) should fix this issue.

If you're impatient, you could try installing my branch into a virtualenv and testing that out before the new version of hsaudiotag3k is published. My branch is at:

https://github.com/chrrrisw/hsaudiotag/tree/TPOS

Regards, Chris.

spt5007 commented 8 years ago

Awesome work @chrrrisw. Thanks again!

spt5007 commented 8 years ago

I tried your hsaudiotag branch in a virtual env. I notice when I run DapGen.py that my multidisc album tracks are printed:

Duplicate track number 1 Birthday
Duplicate track number 2 Yer Blues
Duplicate track number 3 Mother Nature's Son
Duplicate track number 4 Everybody's Got Something To Hide Except Me And My Monkey
Duplicate track number 5 Sexy Sadie
Duplicate track number 6 Helter Skelter
Duplicate track number 7 Long, Long, Long
Duplicate track number 8 Revolution 1
Duplicate track number 9 Honey Pie
Duplicate track number 10 Savoy Truffle
Duplicate track number 11 Cry Baby Cry
Duplicate track number 12 Revolution 9
Duplicate track number 13 Good Night

It seemed to work okay otherwise, though I haven't tried it in my deck just yet. When I run KenwoodDBReader.py I see this just after the last album on my drive (alphabetically):

DEBUG:  Album Ordered Title- 'Complexity'; genre 0015; performer 0011; album 0044
DEBUG:  Album Ordered Title- 'Silverlake (K.S.O.F.M.)'; genre 0015; performer 0011; album 0044
DEBUG:  Album Ordered Title- 'Got A Woman'; genre 0015; performer 0011; album 0044
DEBUG:  Album Ordered Title- 'I Love You All The Time'; genre 0015; performer 0011; album 0044
DEBUG:  Album Ordered Title- 'Oh Girl'; genre 0015; performer 0011; album 0044
DEBUG:  Album Ordered Title- 'Got The Power'; genre 0015; performer 0011; album 0044
DEBUG:  Album Ordered Title- 'Skin-Tight Boogie'; genre 0015; performer 0011; album 0044
DEBUG:  Album Ordered Title- 'Got A Woman (Slight Return)'; genre 0015; performer 0011; album 0044
DEBUG:  Album Ordered Title- 'The Deuce'; genre 0015; performer 0011; album 0044
DEBUG:  Album Ordered Title- 'Save A Prayer'; genre 0015; performer 0011; album 0044
DEBUG:  Album Ordered Title- 'The Reverend'; genre 0015; performer 0011; album 0044
Traceback (most recent call last):
  File "./KenwoodDBReader.py", line 1548, in <module>
    db = DBfile(args.inputfile)
  File "./KenwoodDBReader.py", line 479, in __init__
    self.parse_album_ordered_titles()
  File "./KenwoodDBReader.py", line 771, in parse_album_ordered_titles
    title = self.entries[title_index]
IndexError: list index out of range

The Reverend is the last track on the last album. I can provide the dap file if it's useful to you. I suspect the tracks will be correct on my deck and that's what matters to me. Great work!

spt5007 commented 8 years ago

I tried the new dap db with my deck yesterday. The Modest Mouse album now plays in the correct order, but a few others are still backwards: Pigeons Playing Ping Pong and Spin Doctors. I dug into the issue a bit this morning and gathered up some debug output. Here are my insights:

The two backwards albums have tracks with the track number in the middle of the filename instead of the beginning
Modest Mouse is named similarly but the order is correct when played on my deck.
All three albums have correct track numbers as far as I can tell
The 'DEBUG: Album Ordered Title' output lines are ordered correctly for all albums
I see a warning in the KenwoodDBReader output: WARNING: Unexpected album_title_offset end WARNING: 0x2cceb != 0x2d41b
There may be other backwards albums that I haven't noticed yet

I'm beginning to wonder if the issue is with the way the deck reads albums. Attached is the debug output which I hope is useful to you.

I'm running 1ac93ded930988da8da41f06f01e48a353ac60a6 of this project and a7a62013ffe53e77f44d8f6e875dd14366733cca of hsaudiotag in a virtualenv. TrackOrderDebugOutput.zip

chrrrisw commented 8 years ago

Hi, thanks again for the feedback. Sorry the issue is not yet fixed - I noticed some of my albums now play in the correct order but one, at least, did not. I'll have to go back to the original KMEL program to get a file to which to compare. I may not have fully interpreted the format of these tables. Cheers, Chris.

chrrrisw commented 8 years ago

Hi again, the duplicate track message will be printed if a track number is repeated within the same disc number. Do the files have a 'part of set' tag (disc number tag)?

spt5007 commented 8 years ago

I believe so. Looking at the files with mid3v2 (utility that shipts with mutagen) I see:

Raw IDv2 tag info for White Album Disc 1/01 Back In The USSR.mp3 
TALB(encoding=3, text=['The Beatles (White Album)']) 
TDRL(encoding=3, text=['1968']) 
TXXX(encoding=3, desc='MCN', text=['0094638345329']) 
TCON(encoding=3, text=['Rock']) 
TIT2(encoding=3, text=['Back In The USSR']) 
TXXX(encoding=3, desc='DISCTOTAL', text=['2']) 
TPE1(encoding=3, text=['The Beatles']) 
TXXX(encoding=3, desc='REPLAYGAIN_TRACK_GAIN', text=['-8.62 dB']) 
TPOS(encoding=3, text=['1']) 
TXXX(encoding=3, desc='ITUNES_CDDB_1', text=['E50AE011+209006+17+150+12409+30092+40430+54602+58575+73137+94512+106853+117996+127258+137637+146957+162940+180224+187811+195756']) 
TXXX(encoding=3, desc='TOTALDISCS', text=['2']) 
TXXX(encoding=3, desc='REPLAYGAIN_TRACK_PEAK', text=['0.95858765']) 
TXXX(encoding=3, desc='TOTALTRACKS', text=['17']) 
TXXX(encoding=3, desc='TRACKTOTAL', text=['17']) 
TCOM(encoding=3, text=['Lennon-McCartney']) 
TXXX(encoding=3, desc='ISRC', text=['GBAYE0601644']) 
TSSE(encoding=3, text=['Lavf53.21.1']) 
TRCK(encoding=3, text=['1'])
...
Raw IDv2 tag info for White Album Disc 2/01 Birthday.mp3 
TALB(encoding=3, text=['The Beatles (White Album)']) 
TDRL(encoding=3, text=['1968']) 
TXXX(encoding=3, desc='MCN', text=['0094638345428']) 
TCON(encoding=3, text=['Rock']) 
TIT2(encoding=3, text=['Birthday']) 
TXXX(encoding=3, desc='DISCTOTAL', text=['2']) 
TPE1(encoding=3, text=['The Beatles']) 
TXXX(encoding=3, desc='REPLAYGAIN_TRACK_GAIN', text=['-6.89 dB']) 
TPOS(encoding=3, text=['2']) 
TXXX(encoding=3, desc='ITUNES_CDDB_1', text=['C711CA0E+341759+14+150+12381+30415+43017+53875+68520+88754+102727+121905+133992+147077+160733+198384+224316']) 
TXXX(encoding=3, desc='TOTALDISCS', text=['2']) 
TXXX(encoding=3, desc='REPLAYGAIN_TRACK_PEAK', text=['0.96218872']) 
TXXX(encoding=3, desc='TOTALTRACKS', text=['14']) 
TXXX(encoding=3, desc='TRACKTOTAL', text=['14']) 
TCOM(encoding=3, text=['John Lennon & Paul McCartney/Paul McCartney']) 
TXXX(encoding=3, desc='ISRC', text=['GBAYE0601661']) 
TSSE(encoding=3, text=['Lavf53.21.1']) 
TRCK(encoding=3, text=['1'])

chrrrisw commented 8 years ago

Hi @spt5007, the latest commits (ending with ac3a697), produce a file much closer to what KMEL does, and so hopefully gets us a lot closer to resolving this issue.

I had a album which played in the correct order when selected by Artist and Album on the deck, but played incorrectly when 'All Artists' -> 'All Albums' was selected.

I've had to reorder all the tracks after reading them all in and processing all albums - that seems to have fixed that particular album on my deck, at least.

spt5007 commented 8 years ago

I saw this output on my multidisc albums after running DapGen:

Duplicate track numbers:
    First 1 Back In The USSR
    Second 1 Birthday
    Setting disc number to: 1 - you may want to edit the file and set disc number yourself.
Duplicate track numbers:
    First 2 Dear Prudence
    Second 2 Yer Blues
    Setting disc number to: 1 - you may want to edit the file and set disc number yourself.
Duplicate track numbers:
    First 3 Glass Onion
    Second 3 Mother Nature's Son
    Setting disc number to: 1 - you may want to edit the file and set disc number yourself.
Duplicate track numbers:
    First 4 Ob-La-Di, Ob-La-Da
    Second 4 Everybody's Got Something To Hide Except Me And My Monkey
    Setting disc number to: 1 - you may want to edit the file and set disc number yourself.
Duplicate track numbers:
    First 5 Wild Honey Pie
    Second 5 Sexy Sadie
    Setting disc number to: 1 - you may want to edit the file and set disc number yourself.
Duplicate track numbers:
    First 6 The Continuing Story Of Bungalow Bill
    Second 6 Helter Skelter
    Setting disc number to: 1 - you may want to edit the file and set disc number yourself.
Duplicate track numbers:
    First 7 While My Guitar Gently Weeps
    Second 7 Long, Long, Long
    Setting disc number to: 1 - you may want to edit the file and set disc number yourself.
Duplicate track numbers:
    First 8 Happiness Is A Warm Gun
    Second 8 Revolution 1
    Setting disc number to: 1 - you may want to edit the file and set disc number yourself.
Duplicate track numbers:
    First 9 Martha My Dear
    Second 9 Honey Pie
    Setting disc number to: 1 - you may want to edit the file and set disc number yourself.
Duplicate track numbers:
    First 10 I'm So Tired
    Second 10 Savoy Truffle
    Setting disc number to: 1 - you may want to edit the file and set disc number yourself.
Duplicate track numbers:
    First 11 Blackbird
    Second 11 Cry Baby Cry
    Setting disc number to: 1 - you may want to edit the file and set disc number yourself.
Duplicate track numbers:
    First 12 Piggies
    Second 12 Revolution 9
    Setting disc number to: 1 - you may want to edit the file and set disc number yourself.
Duplicate track numbers:
    First 13 Rocky Racoon
    Second 13 Good Night
    Setting disc number to: 1 - you may want to edit the file and set disc number yourself.
Oops, reordering failed - please report as a bug
    Number of titles 926
    Number re-ordered 925
1#############################################
2#############################################

I will try the drive in my deck in a few minutes. Attaching KenwoodDBReader output for reference. Is it possible that I didn't setup my virtualenv correctly? Thanks again for all your help.

KenwoodDBReader_output.txt

chrrrisw commented 8 years ago

Hi Sean, The output does seem indicative of not having access to the disc parameter in hsaudiotag3k.

I'm also concerned about the last two output lines, I'll check what they mean and get back to you.

In terms of the disc number issue, the fallback should be correct in your case.

chrrrisw commented 8 years ago

Hi Sean, I think the key is that the album based re-ordering fails (one fewer titles), and that leads to further problems down the chain. I've just pushed a change that introduces more robust handling of duplicate track numbering - which may help. I've also added more meaningful debug output for the last two lines. I can successfully run without a hsaudiotag library that supports disc numbers, but I'll be interested to see what the output is on your data. Are you running DapGen.py within your virtualenv (i.e. after you activate it?).

To everybody else, apologies for not working on this in a branch off master.

Regards, Chris.

spt5007 commented 8 years ago

The tracks of the single disc albums appear to be correct. Nice work there.

I re-ran DapGen and see this output

Duplicate track numbers:
    First: Disc 0, Track 1, Title 'Back In The USSR'
    Second: Disc 0, Track 1, Title 'Birthday'
    Trying disk 1
Duplicate track numbers:
    First: Disc 0, Track 2, Title 'Dear Prudence'
    Second: Disc 0, Track 2, Title 'Yer Blues'
    Trying disk 1
Duplicate track numbers:
    First: Disc 0, Track 3, Title 'Glass Onion'
    Second: Disc 0, Track 3, Title 'Mother Nature's Son'
    Trying disk 1
Duplicate track numbers:
    First: Disc 0, Track 4, Title 'Ob-La-Di, Ob-La-Da'
    Second: Disc 0, Track 4, Title 'Everybody's Got Something To Hide Except Me And My Monkey'
    Trying disk 1
Duplicate track numbers:
    First: Disc 0, Track 5, Title 'Wild Honey Pie'
    Second: Disc 0, Track 5, Title 'Sexy Sadie'
    Trying disk 1
Duplicate track numbers:
    First: Disc 0, Track 6, Title 'The Continuing Story Of Bungalow Bill'
    Second: Disc 0, Track 6, Title 'Helter Skelter'
    Trying disk 1
Duplicate track numbers:
    First: Disc 0, Track 7, Title 'While My Guitar Gently Weeps'
    Second: Disc 0, Track 7, Title 'Long, Long, Long'
    Trying disk 1
Duplicate track numbers:
    First: Disc 0, Track 8, Title 'Happiness Is A Warm Gun'
    Second: Disc 0, Track 8, Title 'Revolution 1'
    Trying disk 1
Duplicate track numbers:
    First: Disc 0, Track 9, Title 'Martha My Dear'
    Second: Disc 0, Track 9, Title 'Honey Pie'
    Trying disk 1
Duplicate track numbers:
    First: Disc 0, Track 10, Title 'I'm So Tired'
    Second: Disc 0, Track 10, Title 'Savoy Truffle'
    Trying disk 1
Duplicate track numbers:
    First: Disc 0, Track 11, Title 'Blackbird'
    Second: Disc 0, Track 11, Title 'Cry Baby Cry'
    Trying disk 1
Duplicate track numbers:
    First: Disc 0, Track 12, Title 'Piggies'
    Second: Disc 0, Track 12, Title 'Revolution 9'
    Trying disk 1
Duplicate track numbers:
    First: Disc 0, Track 13, Title 'Rocky Racoon'
    Second: Disc 0, Track 13, Title 'Good Night'
    Trying disk 1

I'm not super savy with Python virtualev but here's my flow:

virtualenv hsaudiotag_env
source hsaudiotag_env/bin/activity # prompt changes
cd path/to/chriss/hsaudiotag
python setup.py install
cd path/to/kmel_db
python DapGen.py /path/to/usb

chrrrisw commented 8 years ago

That flow looks correct to me, although I'm no expert either. I'm still puzzled by the output you're getting on the White Album, since your earlier output from mid3v2 indicates that TPOS is set. In any case, the output is telling me that the tracks for disc 2 are being adjusted appropriately - is it playing in disc and track order? Is there no other output from DapGen.py?

Just a thought, after you've activated your virtualenv with my hsaudiotag installed, try starting python and trying the following:

>>> from hsaudiotag import auto
>>> print (auto.TAG_ATTRS)

It should print out something like:

{'title', 'disc', 'track', 'album', 'artist', 'genre', 'comment', 'year'}

If 'disc' doesn't appear, then my changes are not present.

Regards, Chris.

spt5007 commented 8 years ago

I found the issue. It was actually two things. First I needed to checkout your TPOS branch in my hsaudio clone. I was also relying on the executable bit in the DapGen.py script which was using my non-virtualenv python which lacked your hsaudiotag changes. Executing with my virtualenv python did the trick. The tracks look great. Thanks again for going the extra mile and adding features to hsaudiotag to make this all work. I really appreciate it!

chrrrisw commented 8 years ago

Hi Sean, I'm glad you found the issue - I would not have thought of the shebang at the start of the file, so well done - I should change the first line so that it takes into account any virtualenv that may be used rather than hard code the Python path. Thanks for your help with this issue, too. Cheers, Chris.

chrrrisw / kmel_db

Track Ordering Within Albums is Incorrect #10