Tone corrupts some m4b files, rendering them unable to play in ABS

[gone]

I have a collection of audiobooks living in an AudioBookShelf server, and ABS has an option to embed metadata using tone. Recently I found that running that on m4b files renders them unplayable, with ABS attempting to spin up an ffmpeg transcoder and failing, rather than natively playing them like they did before. The underlying audio file itself hasn't changed, and removing it from ABS it's still able to play in desktop tools.

I was able to get a dump of the tone session and recreate locally with an audiobook, paring down keys added until I arrived at a command that should change nothing:

❯ tone tag --meta-remove-property ItunesPlayGap --meta-remove-property ItunesMediaType --meta-remove-property Narrator feedback-test.m4b

ItuesPlayGap and MediaType are automatically added when missing.

Doing:

❯ tone dump feedback-test.m4b --format=json > test.json                                                                                         

~/abs   
❯ tone dump Feedback-good.m4b --format=json > good.json  

gets no meaningful differences:

❯ diff good.json test.json                                                                                                                      
747,750c747,750
<     "size": 31555439,
<     "created": "2024-01-01T22:27:45.2833975-05:00",
<     "modified": "2024-01-01T22:27:45.2833975-05:00",
<     "accessed": "2024-01-06T11:31:25.8446715-05:00",
---
>     "size": 31555523,
>     "created": "2024-01-06T15:59:58.4538813-05:00",
>     "modified": "2024-01-06T15:59:58.4538813-05:00",
>     "accessed": "2024-01-06T16:00:31.1699222-05:00",
752c752
<     "name": "Feedback-good.m4b"
---
>     "name": "feedback-test.m4b"

However if I scp the file up to the server and attempt to play, it still fails.

Running cmp on the files I get some meaningful differences in the first 115 bites of the file and can compare them with a hexdump of the two files:

bad (post tone):

00000000: 0000 0020 6674 7970 6973 6f6d 0000 0200  ... ftypisom....
00000010: 6973 6f32 6d70 3431 4d34 4120 4d34 4220  iso2mp41M4A M4B 
00000020: 01db 6284 6d64 6174 000f 4f70 656e 696e  ..b.mdat..Openin
00000030: 6720 4372 6564 6974 7300 0000 0c65 6e63  g Credits....enc
00000040: 6400 0001 0000 0846 6565 6462 6163 6b00  d......Feedback.
00000050: 0000 0c65 6e63 6400 0001 0000 0b45 6e64  ...encd......End
00000060: 2043 7265 6469 7473 0000 000c 656e 6364   Credits....encd

good (pre tone):

00000000: 0000 0020 6674 7970 6973 6f6d 0000 0200  ... ftypisom....
00000010: 6973 6f32 6d70 3431 4d34 4120 4d34 4220  iso2mp41M4A M4B 
00000020: 01db 6284 6d64 6174 2110 0500 a01b ffc0  ..b.mdat!.......

Writing a python script to write the first 200 bytes from the good file to the tester fixes the issue and the file plays successfully, so I'm pretty sure the issue is somewhere within those bytes.

The ABS discord pushed me to https://rentry.co/n4ost which is a similar situation. Potentially the same issue, but the magic bytes they're searching for to test:

magic=$(sed -re 's/^.*0580808002(....).*$/\1/' $t)
if [ "$magic" != "1212" ]; then

aren't in my broken files.

I am not knowledgeable enough in the mpeg spec to translate the difference in binary between the two files and either hone in on the specific difference or update the fix.sh script, so I would be happy for any help or direction.

[sandreas]

Great analysis, thank you very much for your effort. It's been a while since the last tone release and it might already be fixed (probably not) bumping the atldotnet lib... However, this needs investigation. I'll try to free some time and keep you up to date.

[gone]

Let me know if I can provide sample files or other debugging info!

[sandreas]

Let me know if I can provide sample files or other debugging info!

Samples would be great, if you could provide some that do not infringe copyright. I would need the working file and the file tone corrupted.

[gone]

The samples I've been using are unfortunately copyrighted - I'll try to get a clean version of the issue

[sandreas]

The samples I've been using are unfortunately copyrighted - I'll try to get a clean version of the issue

Don't put too much effort in this. I'll invite you to a private repository where you can upload / post a link. Make sure you accept the invitation within the next 24 hours.

[sandreas]

Invitation is out, please make sure to read the rules in the README.md :-) Have fun.

[DanielBrierton]

Having what I think is the same issue. I'm using m4b-tool to merge some m4a files and end up with a corrupt m4b. I ran with -vvv to grab the listing.txt file contents and run the FFmpeg concat command, and the resulting file works fine. It's only after tone runs on them that they break. ffprobe returns the following error:

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55a2eeb5e700] Format mov,mp4,m4a,3gp,3g2,mj2 detected only with low score of 1, misdetection possible!
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55a2eeb5e700] moov atom not found

Any luck with figuring out the issue, or at least a workaround? Thanks!

Edit: If I grab the tone.json file, remove the chapters and run tone tag with it, the file is successfully tagged without corrupting the file.

[sandreas]

I'm still on it, turns out to be complex. @DanielBrierton Possible Workaround: Set environment variable M4B_TOOL_DISABLE_TONE=1 to disable tone. Instead mp4v2 will be used which leads to some tags not being set (e.g. movement).

[sandreas]

@gone So I tried to reproduce the issue (very carefully), but my files all play fine. I can reproduce the flipping bytes though. Could you elaborate:

• Which player are you using?
• Which Operating System are you using? (including the architecture)

I already bumped the library versions so maybe the issue is fixed in my personal code. After you provided the infos above, I would build you a custom 0.1.6-pre version to check, if the issue might be fixed with this.

At least with the 0.1.6-pre the bad file is different from the resulting file... so maybe it'll work.

Some further notes for debugging:

# to check the file bytes in terminal
hexdump input.m4b  -n 50 -C

# compare the files
cmp Feedback-good.m4b input.m4b

[sandreas]

@DanielBrierton

tone 0.1.6 is out. Since it is the first release after a long time, this issue may have been fixed with the new atldotnet release. I would love to get feedback, please backup your files.

[sandreas]

tone 0.1.7 is out, no feedback - I'm going to assume this is fixed. Feel free to reopen the issue, if not.

[gone]

apologies - I think this is safe to close for now? I'll double check and reopen if I can recreate.

Unfortunaly ABS ended up removing tone for FFmpeg directly and now has a host of new issues for the same problem with different tools 😓 https://github.com/advplyr/audiobookshelf/issues/3042

[sandreas]

apologies

Never mind. I'm really glad you found this and reproduced it in a so detailed way...

Unfortunaly ABS ended up removing tone for FFmpeg directly and now has a host of new issues for the same problem with different tools 😓

Well, as suprised as I am tone was removed, I do understand the decision. The project tone is, has been and probably will be a one man show of my own (which I really like, because I can do what I want ...). I created tone, because I did not find an audio tagger, that is really cross platform and supports the vast majority of audio book specific tags. It's mainly meant for my personal use :-)

I also think that ABS is a well maintained and pretty impressive project, although I think it has some technical debts due to moving fast.