search

rmcrackan / Libation

Libation: Liberate your Library
GNU General Public License v3.0
2.82k stars 153 forks source link

Incomplete audiobooks? #306

Closed StrawberryBop closed 2 years ago

StrawberryBop commented 2 years ago

I'm sorry, I just registered on here today and don't know how github works, I apologize if I'm posting this incorrectly. I'm also really new to Audible and don't know anything about computers, but if I'm being too confusing just tell me and I'll try to be more clear :)

This might not be a bug, might be a problem with Audible, but I don't know what's going on so I though I would ask.

In some of my books there are big differences between the backup through Libation and what's being played in Audible's web player. I have only checked some of my Discworld books so far and some are the same, while some have issues.

For example, in at least two of them Libation found the "Isis Audiobook presents..." that these books always used to start with, but in the Audible web player that part is missing, but instead it has the end credits, while the Libation backup is missing that. And also the chapters don't start at the same spot, like if the intro has pushed the whole book out of sync and the end fell off. No idea why Audible has the intro but isn't playing it in it's web player :o

And at least three books are missing the actual ending of the books not "just" the end credits, and the chapters are even more out of sync. Maybe because the Libation backup version starts with about 44 seconds of Audible saying that Terry Pratchett has asked them to remind their customers of the Audible terms, and then lists the terms and some threats before the intro starts? The Audible web player doesn't have this bit and starts directly with the intro, and all chapters start when they should start.

I haven't checked all my books, only 9 Discworld books, and 4 books were ok and 5 had issues. I'm using Libation 8.1.7.1 (but some of these books were downloaded with an earlier version, but re-downloading with current version made no difference), and I have "strip Audible branding" checked due to it being so loud and sudden compared to the rest of the book, and I let Libation split the books into chapters, and downloading as m4b.

Any idea what's happening?

Mbucari commented 2 years ago

Can you please provide audible.com links to the problem books?

StrawberryBop commented 2 years ago

Sorry for the delay! I went through all my Discworld books and found 5 problem books:

Book 32: A Hat Full of Sky (narrated by Stephen Briggs) https://www.audible.co.uk/pd/A-Hat-Full-of-Sky-Audiobook/B004EXKHES Book 35: Wintersmith (narrated by Stephen Briggs) https://www.audible.co.uk/pd/Wintersmith-Audiobook/B004EXGYNQ

Some of them (the ones with Rincewind) were removed from the store and replaced just two or three days ago with newly released versions so I can't link them sadly, but they are still in my library and here are the names and problems they have:

Discworld Book 1: The Colour of Magic (narrated by Nigel Planer) Discworld Book 2: The Light Fantastic (narrated by Nigel Planer)

Discworld Book 5: Sourcery (narrated by Nigel Planer)

All other Discworld books read by Nigel Planer and Stephen Briggs that I own seems to be OK :)

That bit where Audible says that Terry Pratchett wants them to remind everyone about the Audible terms is also included in Soul Music (narrated by Nigel Planer): https://www.audible.co.uk/pd/Soul-Music-Audiobook/B004FU7LE4 but here there is no problem. Both the Audible web player and the Libation backup has the Audible warning but is still exactly the same and nothing seems to be missing.

rmcrackan commented 2 years ago

When you listen on your phone app, does it more closely match Libation's download? Or the web player's?

StrawberryBop commented 2 years ago

I don't have a cell phone, only a landline, so I can't try, sorry.

Mbucari commented 2 years ago

This is most curious. I created an audible.co.uk trial account so that I could buy one of the problem books and explore what's going on. For the first 2 links (Books 32 and 35), I can view the books' product pages but cannot buy them. Audible gives me the following message:

We're Sorry Audible does not currently have the rights to sell this title in your country/region. Please consider another book

At first I thought Audible was being very clever and knew that I am actually in the United States; however, I am allowed to purchase other titles that are not available in the US.

I found The Colour of Magic and The Light Fantastic, but they are not narrated by Nigel Planer. It seems like Audible is replacing some titles with new recordings. You appear to own the older versions of these books and can still listen to them and download them, but they are being superseded in the store.

My hunch is that Audible is has updated all the chapter titles and timestamps for these books to match their new versions, and they are improperly serving you the chapter info for the new recordings despite you owning the older recordings. The new chapter info must not match the audio in the older timestamps, which is why the chapter markers and even overall length are incorrect.

I would like to test this theory, but I will need the book ID for your copies of The Colour of Magic and The Light Fantastic. The easiest way to get those is to upload your log file here. Can you please do that?

Mbucari commented 2 years ago

@rmcrackan there should be an easier way for users to find the asin. Also, you should definitely get an Audible uk account. The subscription is £8 per month, which is just shy of $10. Just use a credit card that doesn't charge you foreign transaction fees if you have one.

StrawberryBop commented 2 years ago

Maybe there's different rules for different titles, so that one may be sold to people in the US when using Audible UK while others may not, depending on what rights they have gotten from the publishers? Book 32 and 35 will probably be removed in April next year since that is when the new versions is set to be released. Though they might delete them early, it's hard to say. The old versions of the Rincewind-books were removed the day before they released the new ones.

I downloaded The Colour of Magic, The Light Fantastic, and Sourcery almost two weeks before they were replaced with the new versions, but maybe Audible changed the info in advance?

Where can I find the book IDs? Is it the numbers that are in the urls to the webpages the books were on? I saved the urls and page info in a text file before they removed the pages, so that I would still have the info like when they were released and such. If those are the book IDs, then here are the old urls :) https://www.audible.co.uk/pd/The-Colour-of-Magic-Audiobook/B004EXGW06 https://www.audible.co.uk/pd/The-Light-Fantastic-Audiobook/B004EXKHKW https://www.audible.co.uk/pd/Sourcery-Audiobook/B004EXKHCU

rmcrackan commented 2 years ago

there should be an easier way for users to find the asin

Sorry, you can find these in the export (column C). I'll add to the 'book details' dialog too.

you should definitely get an Audible uk account

This would only help if I got a uk account and also spent my credits on books for trouble shooting.

rmcrackan commented 2 years ago

Maybe there's different rules for different titles, so that one may be sold to people in the US when using Audible UK while others may not, depending on what rights they have gotten from the publishers?

For sure there are tricks that audible plays with regions, but their IDs are sacrosanct. That's how they keep things straight in their own system. If it's the same ID, it's the same everything, irrespective of region -- if something is different between regions, then the IDs won't match.

Mbucari commented 2 years ago

@StrawberryBop Yes, those are indeed the IDs I was talking about.

So My theory is wrong. Taking The Color of Magic as an example, @StrawberryBop's version is B004EXGW06 and has the following metadata:

B004EXGW06 ```Json { "content_metadata": { "chapter_info": { "brandIntroDurationMs": 2043, "brandOutroDurationMs": 5062, "chapters": [ { "length_ms": 196948, "start_offset_ms": 0, "start_offset_sec": 0, "title": "Opening Credits" }, { "length_ms": 7798253, "start_offset_ms": 196948, "start_offset_sec": 197, "title": "The Colour of Magic" }, { "length_ms": 302790, "start_offset_ms": 7995201, "start_offset_sec": 7995, "title": "The Sending of Eight" }, { "length_ms": 4159853, "start_offset_ms": 8297991, "start_offset_sec": 8298, "title": "The Sending of Eight" }, { "length_ms": 5829560, "start_offset_ms": 12457844, "start_offset_sec": 12458, "title": "The Lure of the Wyrm" }, { "length_ms": 6631643, "start_offset_ms": 18287404, "start_offset_sec": 18287, "title": "Close to the Edge" }, { "length_ms": 14029, "start_offset_ms": 24919047, "start_offset_sec": 24919, "title": "End Credits" } ], "is_accurate": true, "runtime_length_ms": 24933076, "runtime_length_sec": 24933 }, "content_reference": { "acr": "CR!XF2JM322HN6WN8D3NT0R6ZGVWQQP", "asin": "B004EXGW06", "content_format": "M4A_AAX", "content_size_in_bytes": 99732837, "file_version": "1", "marketplace": "A2I9A3Q2GNFNGQ", "sku": "BK_RHUK_000259", "tempo": "1.0", "version": "433947" } }, "response_groups": [ "always-returned", "chapter_info", "content_reference" ] } ```

The updated recording is B09LZ1X1RK and has the following chapters:

B09LZ1X1RK ```Json { "content_metadata": { "chapter_info": { "brandIntroDurationMs": 1625, "brandOutroDurationMs": 3297, "chapters": [ { "length_ms": 90371, "start_offset_ms": 0, "start_offset_sec": 0, "title": "Opening Credits" }, { "length_ms": 246688, "start_offset_ms": 90371, "start_offset_sec": 90, "title": "The Color of Magic: Prologue" }, { "length_ms": 8659602, "start_offset_ms": 337059, "start_offset_sec": 337, "title": "The Color of Magic" }, { "length_ms": 355451, "start_offset_ms": 8996661, "start_offset_sec": 8997, "title": "The Sending of Eight: Prologue" }, { "length_ms": 4749222, "start_offset_ms": 9352112, "start_offset_sec": 9352, "title": "The Sending of Eight" }, { "length_ms": 6642298, "start_offset_ms": 14101334, "start_offset_sec": 14101, "title": "The Lure of the Wyrm" }, { "length_ms": 7900866, "start_offset_ms": 20743632, "start_offset_sec": 20744, "title": "Close to the Edge" }, { "length_ms": 59953, "start_offset_ms": 28644498, "start_offset_sec": 28644, "title": "End Credits" } ], "is_accurate": true, "runtime_length_ms": 28704451, "runtime_length_sec": 28704 }, "content_reference": { "acr": "CR!BXJYACJHCN3V91MA263BA94A3VDS", "asin": "B09LZ1X1RK", "content_format": "M4A_AAX", "content_size_in_bytes": 114818465, "file_version": "1", "marketplace": "A2I9A3Q2GNFNGQ", "sku": "BK_PAUK_003392", "tempo": "1.0", "version": "41593721" } }, "response_groups": [ "always-returned", "chapter_info", "content_reference" ] } ```

As you can see, they are different for the 2 different versions of the audiobook. My new guess is that Audible's data is simply wrong. It's been known to happen with other audiobooks. They sometimes deliver chapter info with incorrect chapter time offsets.

@rmcrackan I hate not being able to do anything for this user, but I'm afraid that's the case. Unless you have any ideas?

Mbucari commented 2 years ago

This would only help if I got a uk account and also spent my credits on books for trouble shooting.

I was just saying you should get one just for the hell of it for cheaper books. But yeah, troubleshooting opportunities are a plus.

CharlieRussel commented 2 years ago

Watch for Prime Day sales. I got a year's worth of Audible for an average of just over $5 USD / credit. Had to buy a year's worth, but it was way worth it compared to the typical US Audible price of $12.95-$14.99 USD per credit.

Charlie. (he, him, his)

From: Mbucari @.> Sent: July 8, 2022 8:57 PM To: rmcrackan/Libation @.> Cc: Subscribed @.***> Subject: Re: [rmcrackan/Libation] Incomplete audiobooks? (Issue #306)

there should be an easier way for users to find the asin

Sorry, you can find these in the export (column C). I'll add to the 'book details' dialog too.

you should definitely get an Audible uk account

This would only help if I got a uk account and also spent my credits on books for trouble shooting.

I was just saying you should get one just for the hell of it for cheaper books. But yeah, troubleshooting opportunities are a plus.

— Reply to this email directly, view it on GitHubhttps://github.com/rmcrackan/Libation/issues/306#issuecomment-1179472989, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFAHCRGPDF3JAVCKHSLKA23VTD2AZANCNFSM53B64B5Q. You are receiving this because you are subscribed to this thread.Message ID: @.***>

rmcrackan commented 2 years ago

The issues here are legion.

Regarding this issue: I don't know what more we can do without at least being able to troubleshoot vs a real app. How books act in the app is the real "source of truth."

rmcrackan commented 2 years ago

RE cheaper uk account or prime day credits: thanks! I'll definitely look into those. I'm pretty good with credits right now but more never hurts -- I blow through books and now my whole family is addicted to audiobooks also.

rmcrackan commented 2 years ago

@Mbucari you find the coolest github features. Did you know that the json would collapse like that??

Mbucari commented 2 years ago

Did you know that the json would collapse like that??

Yup! I just googled "GitHub markdown spoilers" and found this:

https://gist.github.com/jbsulli/03df3cdce94ee97937ebda0ffef28287

StrawberryBop commented 2 years ago

Regarding this issue: I don't know what more we can do without at least being able to troubleshoot vs a real app. How books act in the app is the real "source of truth."

Aww, I hope it's not a lost cause :( I don't dare contacting Audible's support about the books, seems like their solution to problems with old books they don't sell anymore is to delete the book from the library and give a credit instead, and then it's gone forever since you can't rebuy it, and 3 of the 5 problem books aren't available anymore :o

I tried doing a little bit of testing just now on one of the books by downloading it again with and without chapters and audible branding, though I don't understand what the results means so I'm not sure if it's helpful:

Discworld Book 1: The Colour of Magic (narrated by Nigel Planer)

Split into chapters, WITH audible branding:

NOT split into chapters, WITHOUT audible branding: duration 6:55:25

NOT split into chapters, WITH audible branding: duration 6:55:40

rmcrackan commented 2 years ago

I don't dare contacting Audible's support about the books, seems like their solution to problems with old books they don't sell anymore is to delete the book from the library and give a credit instead, and then it's gone forever since you can't rebuy it, and 3 of the 5 problem books aren't available anymore :o

Good call

Do I understand correctly that if you download the book without chapters, that it plays the same as the web player? If yes, then either Libation is flubbing the chapter stuff, or else audible has messed up the metadata for chapter breaks.

StrawberryBop commented 2 years ago

Do I understand correctly that if you download the book without chapters, that it plays the same as the web player? If yes, then either Libation is flubbing the chapter stuff, or else audible has messed up the metadata for chapter breaks.

No, the web player isn't playing the intro at all, so no matter how I download, Libation finds the missing intro and adds it, which is good. But if I download without chapters it cuts off at the end of the book, just like when downloading with chapters. It only keeps the end if I both uncheck the "split into chapters" and the "audible branding", as far as I can tell.

Mbucari commented 2 years ago

@StrawberryBop The only way Libation does not try to set chapters is if "Allow Libation to fix up audiobook metadata" is unchecked. Have you tried downloading the bad books with that setting unchecked? If not, please do.

What's the result?

StrawberryBop commented 2 years ago

@StrawberryBop The only way Libation does not try to set chapters is if "Allow Libation to fix up audiobook metadata" is unchecked. Have you tried downloading the bad books with that setting unchecked? If not, please do.

What's the result?

Tried it just now, and the result seem to be the same as when I only unchecked "split into chapters" and the "strip audible branding". It ends up as one file with duration 6:55:40.

StrawberryBop commented 2 years ago

Maybe I could buy and gift one of the problem books that is still being sold and send to one of you? As far as I can tell I should be able to do it in a way where I just get a code I can give. I think all that's needed is that the receiving person has an Audible UK account :)

rmcrackan commented 2 years ago

That’s generous but feel excessive. Let’s think through it for another day before resorting to something like that

StrawberryBop commented 2 years ago

Ok, but the offer stands :) I have a couple of credits left over after my audiobook shopping spree anyway :)

rmcrackan commented 2 years ago

So the problem here is with chapter splitting, right? All is well if downloaded as a single file.

StrawberryBop commented 2 years ago

All is well if downloaded as a single file and if leaving the "This is Audible" part in. If chapters and removal of audible branding is used, it cuts off the end.

rmcrackan commented 2 years ago

I'm inclined to agree that, lacking other information, the highest likelihood is that audible effed up the chapter metadata -- since it sounds like splitting by chapter at all always goes bad.

Mbucari commented 2 years ago

@StrawberryBop I'm the audio specialist here (such as there is one). I would love to see what exactly audible is sending you, metadata and all. If you're comfortable doing so, can you email me your AccountsSettings.json file? That will allow me to sync libation with all books in your account(s) and try downloading for myself, but I will not be able to make any changes to your account or buy anything, so the risk to you is minimal (also, I'm not a crook for whatever my assurances are worth to you).

My email is mbucari1@gmail.com

StrawberryBop commented 2 years ago

@Mbucari Maybe we could start with me sending you one of the still available problem books by giving you a gift code for it, and you can try that one first? :) Audible offered me some cheap credits recently and I still have some left over so it's not a problem :)

Mbucari commented 2 years ago

@StrawberryBop I can't download them because I'm not in the UK. I tried to do just that yesterday when I signed up for a trial UK account, and I could not buy any of the books you were having problems with.

StrawberryBop commented 2 years ago

@Mbucari Aah, that's a pity :/ I went to the US site to look and it seems like Audible US is selling the book but from a different publisher, so I guess they have the rights to sell to US people and the other publisher sells in Europe or something. I've read on reddit that it's possible to buy books in other Audible stores by changing the address or whatnot, but it all seemed confusing to me. Not sure amazon would be happy about stuff like that either, though I don't know what the rules are for things like this.

About sending the AccountsSettings.json file, I'll sleep on it, and will get back to you first thing tomorrow :) Thanks!

StrawberryBop commented 2 years ago

@Mbucari I just sent the file so it should be in your inbox now :)

I don't know anything about stuff like this so I'm wondering - can Audible see that you are trying to access the books from the US and not let you since it's the wrong region? Or doesn't it matter when using Libation to access them?

rmcrackan commented 2 years ago

I’ve heard conflicting reports but I believe it’s based on your billing address, not current location

StrawberryBop commented 2 years ago

I’ve heard conflicting reports but I believe it’s based on your billing address, not current location

Ah, Ok :)

Mbucari commented 2 years ago

I’ve heard conflicting reports but I believe it’s based on your billing address, not current location

That tracks with my experience setting up the trial. I even tried to use a privacy one-time credit card app, but that didn't work and they locked down my trial account.

@StrawberryBop Got it. It works, and I'm looking into it now.

StrawberryBop commented 2 years ago

@Mbucari Great! Thank you :)

Mbucari commented 2 years ago

OK, so there are two different things going on here that combined to cause the problems you're seeing:

Problem 1

Audible inserted a 44 second long legal threat between "This is Audible" and the publisher's opening credits, but they did not update the metdata to reflect the increased length or chapter marker offsets.

The metadata says the chapters should be like this:

But the actual audio file is like this:

As you can see, all subsequent chapters' actual start positions are pushed back ~44 seconds, but the metadata doesn't reflect that. There's nothing Libation can do about that. Your only real hope is to edit the audio file itself to either remove the legal threat or correct the chapter markers.

Problem 2

When splitting the output file by chapters, the final chapter is truncated. In fact, it's truncated by exactly the length of the legal threat (44 seconds). The reason for this is because the splitter assumes that there is no audio after the end of the last chapter marker, so it ends the final audio file there. The reason for this was to avoid problems that were being caused by some books whose chapter markers were very slightly shorter than the audio output (#251). I never expected the end chapter marker to be a whopping 44 seconds shorter than the actual audio file. I'll fix this soon, so at least when you're splitting the audiobooks they won't be cut short. But like I said before, there's nothing I can do about the markers being in the wrong place.

StrawberryBop commented 2 years ago

@Mbucari Ah so that's what's happening! Thank you so much for looking into it! Nice to know what's going on with the books :)

I'm thinking the best thing might be to keep one version not split into chapters as a kind of backup, and use the one split into chapters to listen to after you have fixed so it won't cut off the end :) The chapters starting at the wrong place is probably better than no chapters at all for easy listening :)

I would really like to at least try to correct the chapter markers though, but I don't know how. This whole thing is very new to me. I see there is a cue file that seems to list the chapters when downloading the book as one file, is that how? I tried google but I think I might have too little knowledge in the subject to understand the results I got.

rmcrackan commented 2 years ago

@Mbucari Another great sleuthing job. You're really good at this.

@StrawberryBop I thought earlier of just asking you to manually edit a cue file because it wouldn't be that hard for a few books. I avoided that suggestion because 1 I don't want to avoid addressing a real issue if there was one and 2 that begs the obvious question: then how do I use it? There are tools out there (you can just google cue split) but there's not a specific one I'd recommend.

Maybe one day we'll build a split-by-cue tool directly into Libation. We already have convenience features like m4b => mp3. Such a tool might be a good companion. I really don't want Libation to become a swiss army knife of little things that other people do better, but I do want to serve our users. So we'll see.

StrawberryBop commented 2 years ago

@rmcrackan Thank you, I will google "cue split" and see what I find! :)

A split-by-cue tool in Libation would be a great addition if you do decide to add it some day :) It would give even more control over how we store and listen to the books we own which is nice :)

I just want to add that Libation is so easy to use that even someone like me, with my lack of knowledge in anything computer-related, can use it easily to backup my books, and that is something I very much appreciate!

And thank you @Mbucari and rmcrackan for all the help you're giving me with my issue! :)

rmcrackan commented 2 years ago

I spent the morning researching what it would take to create a cue splitter. There are 2 common file types we support. With one type, we can write a tiny amount of code and re-use stuff we already have -- easy peasy. With the other type (mp3), it's not too hard to do with windows but I'm not sure how to it for non-windows. Since we're starting to talk about moving to cross platform, I don't want to start introducing more windows-only code.

@Mbucari the rub is: NAudio is incredibly not forthcoming about what is or is not xplat. They say NAudio.Core is, but they're very non-committal about the rest including NAudio.Wave which is where the mp3 split stuff is. This brings up another conversation we need to have soon anyway -- my understanding is that all of our mp3 stuff is currently windows-only. With the transition to cross platform, what were your ideas for segregating off those pieces?

Mbucari commented 2 years ago

In my opinion, if you're going to split an mp3 file you should do it without transcoding or you shouldn't do it at all. If NAudio.Wave houses the mp3 splitter, that suggests to me that it works by decoding the audio and then re-encoding to split files. That's a no-no for audio fidelity. I'll look into how to do this without transcoding. I know it's possible, but I don't know how much of a pain it is.

As for my windows-only code, I plan to make 3 different AAXClean.Codecs packages. One for windows, one for mac, and one for linux. nupack files support multiple platforms by using internal directories with platform-specific copies of each lib.

rmcrackan commented 2 years ago

NAudio is obviously a complex library; here's what I've found so far this morning. NAudio.Wave houses the mp3 reader which itself calls a library which is extremely heavy on windows-only stuff. When you're using the reader to do a split, you just progress through the frames and write them to a new stream. ie: you aren't encoding. However, you're still likely deep into windows-only territory.

Mbucari commented 2 years ago

I briefly looked at the mp3 file structure to see what it would take to split, and I decided I don't want to build one. It's a mix of unofficial standards and a fair amount of guess work as to where the key frames actually are.

Here's one thought: Design an interface that has the mp3 tools you need. Then implement the interface using the lightest weight package you can find that will get the job done, even if it's windows-only. When it comes time to build xplat releases, we can implement that interface using ffmpeg and require the user to install ffmpeg using their package manager.

Alternatively, use ffmpeg for windows too. It would absolutely suck to have to add ffmpeg as a dependency again, though. Maybe do a custom build of ffmpeg that only includes the mp3 splitting ability? I have experience building custom ffmpeg binaries (my aac decoder dll is just such a custom build, and it's < 1MB).

rmcrackan commented 2 years ago

Oops, we've more than a little high-jacked this user's issue with our conversation about cross platform splitting. RE the original issue: you said one issue is audible screwed up their metadata -- nothing we can do about that. The other can be fixed. Is that a fix which is going into AAXClean?

Mbucari commented 2 years ago

I thought about "Problem 2" a little bit more, and now I'm not so sure it's a bug with AAXClean (or a bug at all). You're calling ConvertToMultipartX() and telling it to split the input file using the chapter markers. Expected behavior should be that the output files match the input chapter definitions. If you provide chapter markers that don't span the full length of the input file, what should it do? I think it's logical that it should ignore all audio not in input chapters ranges, including ignoring any gaps between chapters.

rmcrackan commented 2 years ago

If there are gaps (other than maybe something tiny like <2 seconds), should we assume the metadata is in error?

Mbucari commented 2 years ago

Sure, but that needs to be done at the Libation level. At the audio splitting level, ChapterInfo should be law.

rmcrackan commented 2 years ago

Crap, this fell off my radar. Is there a to-do item here?

Mbucari commented 2 years ago

@rmcrackan I don't think so. If I change aaxclean so that it always trims the output chapter markers, I'll update libation so it will fix ChapterInfo's that are shorter tan the audio file length.