search

khenriks / mp3fs

FUSE-based transcoding filesystem from FLAC to MP3
http://khenriks.github.io/mp3fs/
GNU General Public License v3.0
382 stars 46 forks source link

Problem with mp3fs and rsync #30

Closed rspberrypii closed 9 years ago

rspberrypii commented 9 years ago

Using rsync to copy files from the mp3fs file system fails most of the time but works occasionally.

rsync -av /media/raspii/mp3/ /media/raspii/Lexar/MP3/ sending incremental file list ./ AC-DC/Back_in_Black/ AC-DC/Back_inBlack/Disc 1 - 1-_Hells_Bells.mp3 AC-DC/Back_inBlack/Disc 1 - 2-_Shoot_to_Thrill.mp3 rsync: read errors mapping "/media/raspii/mp3/AC-DC/Back_inBlack/Disc 1 - 2-_Shoot_to_Thrill.mp3": No data available (61)

I can play the flac file, cp the mp3 and play it, but when I use rsync it throws errors. The flac file appears to be fine:

mutagen-inspect AC-DC/Back_inBlack/Disc\ 1\ -\ 2-_Shoot_to_Thrill.flac -- AC-DC/Back_inBlack/Disc 1 - 2-_Shoot_to_Thrill.flac

Here's the mount command:

mp3fs -d -quality=2 -b 320 /home/raspii/FLAC/ /media/raspii/mp3/ -o allow_other,ro

Version (downloaded via github, installed via configure/make/make install):

$ mp3fs -V mp3fs version: 0.91 LAME library version: 3.99.5 FLAC library version: 1.3.0 FUSE library version: 2.9.2 fusermount version: 2.9.2 using FUSE kernel interface version 7.19

xubuntu version:

lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 14.04.1 LTS Release: 14.04 Codename: trusty

First part of the debug output:

mp3fs -d -b 320 /home/raspii/FLAC /media/raspii/mp3 -o allow_other,ro mp3fs: MP3FS options: basepath: /home/raspii/FLAC bitrate: 320 quality: 5 gainmode: 1 gainref: 89.000000 desttype: mp3

FUSE library version: 2.9.2 nullpath_ok: 0 nopath: 0 utime_omit_ok: 0 unique: 1, opcode: INIT (26), nodeid: 0, insize: 56, pid: 0 INIT: 7.22 flags=0x0000f7fb max_readahead=0x00020000 INIT: 7.19 flags=0x00000010 max_readahead=0x00020000 max_write=0x00020000 max_background=0 congestion_threshold=0 unique: 1, success, outsize: 40

I've got the rest of the debug output if you want me to post it here.

khenriks commented 9 years ago

Thanks for the detailed bug report!

The best way to test the flac file is with flac -t filename.flac. If it is fine, this could be an issue with reported vs. actual size of the file, which makes rsync behave badly, but which I thought I had fixed. To test this, cp the file in question, and compare the size of the file you get with the size reported in the mp3fs mount. This should actually show up in the debug log too, buried somewhere, if you could post that in a gist.

rspberrypii commented 9 years ago

I'm not sure if the WARNING is relevant to this problem but the last line says 'ok':

$flac -t Disc\ 1\ -\ 2_-_Shoot_to_Thrill.flac

flac 1.3.0, Copyright (C) 2000-2009, 2011-2013 Josh Coalson & Xiph.Org Foundation flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. Type `flac' for details.

Disc 1 - 2_-_Shoot_to_Thrill.flac: WARNING, cannot check MD5 signature since it was unset in the STREAMINFO ok

Size in the mp3fs:

-rw------- 1 raspii raspii 12717778 Sep 18 20:34 Disc 1 - 2_-_Shoot_to_Thrill.mp3

Size after copy:

/media/raspii/mp3/AC-DC/Back_inBlack$ cp Disc\ 1\ -\ 2-_Shoot_to_Thrill.mp3 /home/raspii/

$ls -l /home/raspii/Disc\ 1\ -\ 2_-_Shoot_toThrill.mp3 -rw------- 1 raspii raspii 12715688 Sep 19 07:59 /home/raspii/Disc 1 - 2-_Shoot_to_Thrill.mp3

This might not be the part of the debug log you're looking for but the sizes here match up with the filesystem sizes:

mp3fs: read /AC-DC/Back_inBlack/Disc 1 - 2-_Shoot_to_Thrill.mp3: 131072 bytes from 12582912 mp3fs: Reading 131072 bytes from offset 12582912. mp3fs: Buffer reallocate: 0xb40de008 -> 0xb34bc008 ; 12717778 -> 12721786 mp3fs: Finishing file. Predicted size: 12717778, final size: 12715688 read[18446744072456613136] 131072 bytes from 12582912 unique: 342, success, outsize: 131088 unique: 343, opcode: READ (15), nodeid: 13, insize: 80, pid: 6934 read[18446744072456613136] 4096 bytes from 12713984 flags: 0x8000

I'll work on creating the gist with whole debug log.

rspberrypii commented 9 years ago

Here's the gist:

https://gist.github.com/rspberrypii/1e180936a042c28070b5

Thanks for taking a look at this. mp3fs is great and if I can get it working with rsync it's a pretty slick solution for me.

khenriks commented 9 years ago

If you look in line 1157, you can see a similar issue happens with track 1 as well. For track 1, the file is too long (actual longer than predicted), so rsync doesn't care. It will only complain if the file is too short, as in track 2. The warning from flac -t combined with the strange file sizes make me nervous that the STREAMINFO block in your FLAC files is corrupted.

One way that should let you test this is to re-encode track two using the flac command line, with

flac -o temp.flac original.flac

and see if the new file behaves better.

rspberrypii commented 9 years ago

Unfortunately flac -o is pegging my CPU and never finishing after it gets this far:

Disc 1 - 2_-_Shoot_to_Thrill.flac: 99% complete, ratio=1.002

If I ctrl-c the process the new output file is simply corrupted.

It looks like SoundJuicer, which I've unfortunately used to rip the files, has bugs around this issue:

https://bugzilla.gnome.org/show_bug.cgi?id=727802 https://bugs.launchpad.net/ubuntu/+source/sound-juicer/+bug/1288872 https://bugzilla.redhat.com/show_bug.cgi?id=961881

Weirdly though I haven't noticed any problems listening to any of the files. If I can just find a way to update the MD5 checksum hopefully it will work.

khenriks commented 9 years ago

The hang on the flac command line is almost certainly a bug in flac, because even with invalid input at worst it should terminate with an error.

That's also unfortunate about Sound Juicer. I actually switched a while ago to abcde combined with MusicBrainz Picard for all ripping and tagging, so I haven't encountered the bug.

I think the best option for the problem you're encountering is to try and fix the files somehow or rerip them. Based on the output in the debug log, I think there's more than just the missing md5sum which is wrong; I think the number of samples in the STREAMINFO header doesn't match the number actually in the file. There's basically no way mp3fs can handle this correctly; it doesn't crash and produces mostly correct output, so that's about the best.

For fixing it, I think one option would be to see if you can decode the file with the flac command line. Try flac -d -o file.wav file.flac if that works, then it's a matter of the easiest way to copy the metadata from the old flac file to the new re-encoded one. If it doesn't work, you can try sending me the problematic file so I can take a look.

rspberrypii commented 9 years ago

I was able to use flac to decode them, then encode them back again with a select amount of meta-data. mp3fs, and rsync too, is then able to work with them successfully.

This isn't my area of expertise so I'm not sure my methodology or tools are sound but I took the original file, re-ripped it with flac, then decoded it again and validated that the md5sum is the same as the original file. When I decode the SoundJuicer encoded file back to wav I get a different checksum.

Further, if run this:

avconv -i DecodedSoundJuicerThrill.wav -f framemd5 - avconv -i DecodedFlacThrill.wav -f framemd5 -

And diff the output they're the same until the last two frames with the flac version having two extra frames at the very end:

SoundJuicer encoded/decoded framemd5 output:

.. 0, 14015488, 14015488, 1024, 4096, 492db581c792854cfa78821cd240eac2 0, 14016512, 14016512, 1024, 4096, f5a63e82f818ace887232b70588165dd size= 1069kB time=317.86 bitrate= 27.6kbits/s
video:0kB audio:54756kB global headers:0kB muxing overhead -98.046848%

flac encoded/decoded framemd5 output:

0, 14015488, 14015488, 1024, 4096, 492db581c792854cfa78821cd240eac2 0, 14016512, 14016512, 1024, 4096, f5a63e82f818ace887232b70588165dd 0, 14017536, 14017536, 1024, 4096, e9905fd3c4e0a513ec529e889a0c39d9 0, 14018560, 14018560, 536, 2144, 90e32d21fb20dd3574151b4f14a5d417 size= 1070kB time=317.89 bitrate= 27.6kbits/s
video:0kB audio:54762kB global headers:0kB muxing overhead -98.046780%

If I'm reading that correctly the last two frames aren't being written out correctly by SoundJuicer/gstreamer and that's resulting in the last .03 seconds being lopped off. That's probably why I've never noticed in playing the files back and I'll have to pick a few to see if it warrants re-ripping my entire collection.

Thanks for the recommendation of abcde combined with MusicBrainz Picard, I'd been wanting to move away from SoundJuicer for a while and this is all the argument I need. Thanks again for all your help on what wasn't a mp3fs problem in the first place.

khenriks commented 9 years ago

I think your assessment of the situation is right. SoundJuicer seems to write most of the file correctly but doesn't seem to finish it off for whatever reason.

abcde is actually a command-line program, and it's pretty flexible, but also can be involved to set up. I've put my abcde.conf file into a gist, although I should note that it's specific to my setup and preferences. The CDROM variable is what you want for a Mac, but for Linux it should work if you comment that out completely. The last three options let you customize where the files get placed and how they get named, but I pretty much just use Picard to rename everything anyway. Overall my routine is more complicated than the option of just using SoundJuicer, but I feel a lot better about the result I get. You could also look into morituri, which could also be good.

I'm glad I could help you out. I want people to use mp3fs and to like it, and if it was an mp3fs problem I would want to fix it. Even though it wasn't, I'm glad I could help.