search

chbrown / overdrive

Bash script to download mp3s from the OverDrive audiobook service
451 stars 66 forks source link

Fails with recent curl (e.g., macOS 10.14.6+, 10.15+, a.o.) "certificate has expired" #36

Closed cox890 closed 2 years ago

cox890 commented 3 years ago

The download & return suddenly stopped working I have tried with a fresh install same problem a .odm.license is created by the script it's empty & nothing happens

Glocktober commented 3 years ago

same here - same error. Tried with both the current and an older version.

9mido commented 3 years ago

What OS are you all using? It still works for me on Debian.

Glocktober commented 3 years ago

OSX - Mojave 10.14.6 - been using this for a couple of years but the problem only appeared a week or so ago.

I tried this on Amazon Linux 2 and have the same problem. but I was able to run it on CentOS 8.4 (with a fresh ODM in each case)

choc96208 commented 3 years ago

I also have this problem and am also using OSX Mojave 10.14.6. The output does this:

ClientID=
Using AcquisitionUrl=
Using MediaID=
Using RawHash=
Using Hash=

but nothing downloads.

gtTracy commented 3 years ago

Have you tried using the --verbose flag to see what might be going on? Also, which version of xmllint do you have installed?

Glocktober commented 3 years ago

This on my Mac (Mojave):

I edited overdrive.sh and added '-v' to the curl options and it reports a certificate expiration error

I added -k (ignore ssl errors) to the curl opts (getting a fresh ODM) and sure enough it works - I'm donloading files

so I directly check the certificate: $ openssl s_client -connect 207.54.136.40:443 | openssl x509 -noout -text depth=1 O = Digital Signature Trust Co., CN = DST Root CA X3 verify error:num=10:certificate has expired notAfter=Sep 30 14:01:15 2021 GMT verify return:0 depth=1 O = Digital Signature Trust Co., CN = DST Root CA X3 verify error:num=10:certificate has expired notAfter=Sep 30 14:01:15 2021 GMT verify return:0 depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3 verify error:num=10:certificate has expired notAfter=Sep 30 14:01:15 2021 GMT verify return:0 Certificate: Data: Version: 3 (0x2) Serial Number: 03:81:e3:4c:c8:6a:7b:e6:ca:93:d2:a0:1b:e9:d6:01:dc:f6 Signature Algorithm: sha256WithRSAEncryption Issuer: C=US, O=Let's Encrypt, CN=R3 Validity Not Before: Aug 9 16:20:39 2021 GMT Not After : Nov 7 16:20:37 2021 GMT Subject: CN=*.cdn.overdrive.com Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: ...etc....

The certs expired Sept 30 - which right when I started having problems.

It looks like they've switched to Lets Encrypt but never removed the expired certs. Now I'm not sure why some are choking and others are fine with this - maybe different versions of curl? my Mac is running 7.54.0 while my cent box is 7.61

As a workaround I think I'm not seeing too much risk in using -k on curl...

29c29 < CURLOPTS=(-s -L -A "$UserAgent" --compressed)

CURLOPTS=(-s -k -L -A "$UserAgent" --compressed)

choc96208 commented 3 years ago

Not OP but here's my output. overdrive download *.odm --verbose

Entering debug (verbose) mode
+ CURLOPTS=("${CURLOPTS[@]:1}")
+ shift
+ [[ 0 -gt 0 ]]
+ [[ 1 -eq 0 ]]
+ [[ 1 -eq 0 ]]
+ HEADER_PRINTED=
+ for ODM in '"${MEDIA[@]}"'
+ for COMMAND in '"${COMMANDS[@]}"'
+ case $COMMAND in
+ download BellJar-7138.odm
+ license_path=BellJar-7138.odm.license
+ acquire_license BellJar-7138.odm BellJar-7138.odm.license
+ [[ -e BellJar-7138.odm.license ]]
++ uuidgen
++ tr '[:lower:]' '[:upper:]'
+ ClientID=ID
+ printf 'Generating random ClientID=%s\n' DD2537FF-5F56-4375-B515-02BA63949318
Generating random ClientID=DD2537FF-5F56-4375-B515-02BA63949318
++ xmllint --xpath '/OverDriveMedia/License/AcquisitionUrl/text()' BellJar-7138.odm
+ AcquisitionUrl=https://ofs.contentreserve.com/MP3LicenseAcquisitionService.svc/ab3d7e4e-90c5-48cc-93f3-adb60078a400
+ printf 'Using AcquisitionUrl=%s\n' https://ofs.contentreserve.com/MP3LicenseAcquisitionService.svc/ab3d7e4e-90c5-48cc-93f3-adb60078a400
Using AcquisitionUrl=https://ofs.contentreserve.com/MP3LicenseAcquisitionService.svc/ab3d7e4e-90c5-48cc-93f3-adb60078a400
++ xmllint --xpath 'string(/OverDriveMedia/@id)' BellJar-7138.odm
+ MediaID=348c3c61-b9d4-402e-873f-b2d10f610395-425
+ printf 'Using MediaID=%s\n' 348c3c61-b9d4-402e-873f-b2d10f610395-425
Using MediaID=348c3c61-b9d4-402e-873f-b2d10f610395-425
+ RawHash='DD2537FF-5F56-4375-B515-02BA63949318|1.2.0|10.11.6|ELOSNOC*AIDEM*EVIRDREVO'
+ printf 'Using RawHash=%s\n' 'DD2537FF-5F56-4375-B515-02BA63949318|1.2.0|10.11.6|ELOSNOC*AIDEM*EVIRDREVO'
Using RawHash=DD2537FF-5F56-4375-B515-02BA63949318|1.2.0|10.11.6|ELOSNOC*AIDEM*EVIRDREVO
++ echo -n 'DD2537FF-5F56-4375-B515-02BA63949318|1.2.0|10.11.6|ELOSNOC*AIDEM*EVIRDREVO'
++ iconv -f ASCII -t UTF-16LE
++ openssl dgst -binary -sha1
++ base64
+ Hash=K9n1Of/bjq/1tNMC9cxjDvuCt5A=
+ printf 'Using Hash=%s\n' K9n1Of/bjq/1tNMC9cxjDvuCt5A=
Using Hash=K9n1Of/bjq/1tNMC9cxjDvuCt5A=
+ curl -L -A 'OverDrive Media Console' --compressed 'https://ofs.contentreserve.com/MP3LicenseAcquisitionService.svc/ab3d7e4e-90c5-48cc-93f3-adb60078a400?MediaID=348c3c61-b9d4-402e-873f-b2d10f610395-425&ClientID=ID=1.2.0&OS=10.11.6&Hash=K9n1Of/bjq/1tNMC9cxjDvuCt5A='
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.
HTTPS-proxy has similar options --proxy-cacert and --proxy-insecure
Glocktober commented 3 years ago

I just pulled a newer version of curl from brew, 7.79.1, and removed the -k, and downloaded another book just fine.

Definitely relating to the expired certificates and how your version of CURL deals with it.

If you do get the brew version, realize brew doesn't replace the OSX version in /usr/bin/curl so you will need to use curl from /usr/local/opt/curl/bin

I did this by adding: export PATH="/usr/local/opt/curl/bin:$PATH"

to overdirve.sh (the -k was removed for my test.)

choc96208 commented 3 years ago

Thanks for your help Glocktober! It worked after I updated curl from brew running the extra PATHs you and brew itself suggest. I didn't remove the -k in overdrive.sh.

chbrown commented 3 years ago

First off just wanna say I'm chuffed that you all are helping diagnose and debug before I even I get a chance to catch up on my email and poke my head in :)

As @Glocktober mentioned, there's some (temporary?) issue with the SSL certs; not sure what changed in curl, or why macOS's system curl has a problem with them — they look ok as far as SSL Labs is concerned — but tl;dr:

-CURLOPTS=(-s -L -A "$UserAgent" --compressed)
+CURLOPTS=(-s -L -A "$UserAgent" --compressed --insecure)

seems like an entirely reasonable fix; it's not like you're sending any sensitive info or going to be devastated if someone actually MITM's your connection and Rickrolls your audiobook ;)

Glocktober commented 3 years ago

From the output of:

openssl s_client -connect mp3audio-gk.cdn.overdrive.com:443

it looks like overdrive is offering 3 certs that expired 9/30/2021 along with the Lets Encrypt cert (not exactly a best practice.)

The older version of curl in Mojave (7.54) (I'm guessing) failed on the expired certs never getting to the good LE cert.

dhnyny commented 3 years ago

Is it possible to release an updated version for those of us who don't understand enough about how to implement the apparently simple fix outlined above? I gather I need to edit overdrive.sh but all I see is the executable file at /usr/local/Cellar/overdrive/2.1.1/bin/overdrive and I'm too clueless to know how to edit it. Sorry for being so helpless.

ajv43 commented 3 years ago

@dhnyny same. @chbrown from your first post, how would we now copy and past into terminal to still be able to get the downloads working once again.

And also, thank you so much in the first place. Its amazing being able to download these audiobooks and listen to them whenever convenient.

dhnyny commented 3 years ago

For what it's worth, I've got one Mac running Mojave and another running Big Sur. I can download fine on the Mac running Big Sur, which has curl 7.64.1, but not with the one running Mojave and curl 7.54.

Elsto007 commented 2 years ago

Ok. Hi all! I'm a Newbie here who knows nothing about coding so I really need my hand held. Does this work around work with Big Sur 11.6.1? If so, what are the step by step directions. If it works with Big Sur are there any changes to the terminal coding? I switched my user agent to Google Chrome Windows, got my mp3 audiobook download button to appear, downloaded the audiobook, ran the installation step and then came the first error fix:

At this point, if calling overdrive produces the error message -bash: overdrive: command not found, you'll need to add ~/.local/bin to your PATH. One way to do this:

printf 'export PATH=$HOME/.local/bin:$PATH\n' >> ~/.bashrc source ~/.bashrc

Wasn't sure what to do with the above instructions. I didn't want to mess around too much. I did try a few times with the same book loan.

I'd love for this to work for me. I want to still be able to listen to my audiobooks on my 6th generation iPod but to do this I need to continue to download the MP3 Overdrive file to my MacBook Pro and drag it into my Apple Music (aka iTunes).

Thanking is advance for this wonderful program you created and for any help. Happy Holidays!

chbrown commented 2 years ago

Does this work around work with Big Sur 11.6.1?

It should, though the previous commenter noted that Big Sur didn't need the workaround at all.

(Personally, I haven't yet upgraded to Big Sur, so can't say anything from first-hand experience.)

Note that if you're on a fresh install (not an upgrade) of Big Sur, you've probably got zsh as your default shell, so pay attention to the zsh parts of the README.

I switched my user agent to Google Chrome Windows, got my mp3 audiobook download button to appear, downloaded the audiobook, [...]

Oh, good thinking! That's a handy step I really ought to document in the README 😄

[...] then came the first error fix:

At this point, if calling overdrive produces the error message -bash: overdrive: command not found, you'll need to add ~/.local/bin to your PATH.

That's just a quote of the documentation from the README. I don't know if it's relevant to your case. Honestly I don't even know if you actually ran into any problems — you don't explicitly say that, but I'm just kinda assuming something went wrong since you're posting here.

If you are running into that -bash: overdrive: command not found error message in your terminal: first, note that your shell is bash, not zsh. Second, when the README says "One way to do this:" that means that, to fix it, you copy & paste that entire code block into your terminal. (And you might need to execute the final command manually if you didn't happen to capture the final newline when selecting the code block.)

Otherwise, you're gonna have to come back with a lot more details, including:

  1. What have you done so far leading up to the error
  2. What does the error say
  3. What have you tried to fix the error

P.S. There's a YouTube tutorial at https://youtu.be/8Lgnh9To7Fw that might be helpful? I haven't gone through the whole thing — I didn't make it — but it may help you see what the process looks like end-to-end.

Elsto007 commented 2 years ago

Hi. Here is a step by step of what I tried. Hope this is enough detail:

I think the default shell is my issue. If I can get this right I'm hoping it will work for me:

PUBLIC LIBRARY: Borrow Book from LIBRARY LOANS

Switch Agents to access “download button” on Safari

Click Download Audiobook Button : File goes to download file folder as xxxaudiobook.odm

Bring up Github App: https://github.com/chbrown/overdrive#instructions and/or https://www.youtube.com/watch?v=8Lgnh9To7Fw

TERMINAL:

The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.
EllenMaookPro15:~ ellenstock$

Pasted the following code block above: 

mkdir -p ~/.local/bin
curl https://chbrown.github.io/overdrive/overdrive.sh -o ~/.local/bin/overdrive
chmod +x ~/.local/bin/overdrive

QUESTION: I understand I only need to do this one time for all audiobooks?

Got the following:

The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.
EllenMaookPro15:~ ellenstock$ mkdir -p ~/.local/bin
EllenMaookPro15:~ ellenstock$ curl https://chbrown.github.io/overdrive/overdrive.sh -o ~/.local/bin/overdrive
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  8911  100  8911    0     0  80279      0 --:--:-- --:--:-- --:--:-- 80279
EllenMaookPro15:~ ellenstock$ chmod +x ~/.local/bin/overdrive

CLOSED ABOVE TERMINAL WINDOW & OPENED NEW TERMIMAL WINDOW:

Last login: Thu Dec 30 21:15:19 on ttys000

The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.
EllenMaookPro15:~ ellenstock$

TYPE: overdrive download DRAG: downloaded odm audiobook file from my desktop to the terminal HIT: enter

RESULTS:

Last login: Thu Dec 30 21:15:19 on ttys000

The default interactive shell is now zsh.
EllenMaookPro15:~ ellenstock$ overdrive download/Users/ellenstock/Desktop/Origin
EllenMaookPro15:~ ellenstock$ overdrive download/Users/ellenstock/Desktop/Origin
EllenMaookPro15:~ ellenstock$ overdrive download/Users/ellenstock/Desktop/Origin_9780739319307_2968609.odm 
-bash: overdrive: command not found
EllenMaookPro15:~ ellenstock$

GOT THE ERROR MESSAGE 

  -bash: overdrive: command not found

QUESTION: Why does it say “The default interactive shell is now zsh.” When the terminal window says myname – chsh < -bash – 80x12”?

QUESTION: So, does this mean my default shell is bash and I need to change the DEFAULT SHELL from zsh to bash? It’s confusing to me.

THE README SAYS: At this point, if calling overdrive produces the error message -bash: overdrive: command not found, you'll need to add ~/.local/bin to your PATH. One way to do this:

printf 'export PATH=$HOME/.local/bin:$PATH\n' >> ~/.bashrc
source ~/.bashrc

QUESTION: Do I copy and past the above code block into a new terminal window?

COPIED RESULTS COPIED ABOVE CODE BLOCK IN NEW TERMINAL WINDOW:

RESULTS:

Last login: Thu Dec 30 21:21:44 on ttys000

The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.
EllenMaookPro15:~ ellenstock$ printf 'export PATH=$HOME/.local/bin:$PATH\n' >> ~/.bashrc
EllenMaookPro15:~ ellenstock$ source ~/.bashrc
EllenMaookPro15:~ ellenstock$

QUESTION: Does the above results show the default shell is now BASH? Does this step make the default shell stay as BASH or do I have to do this step every time I borrow a book?

QUESTION: "And you might need to execute the final command manually if you didn't happen to capture the final newline when selecting the code block." How and where do I do this?

I feel all I need is a simple tweek to the default shell for this to work for me. I just still don’t get how to get there. Am I leaving out important steps? Hope this makes some sense.

I googled how to change the default shell and there are several ways. They all seem to need my password and then to reboot the computer. Is this another way? I did not want to do this since it’s nowhere in the README ON THE GITHUB instructions.

Thank you for your time and HAPPY NEW YEAR!

chbrown commented 2 years ago

I think the default shell is my issue.

Wish it were that easy, but I doubt it. zsh is a fork of bash and everything we use here works exactly the same way, they just look for certain settings in different files, e.g. .bashrc vs .zshrc.

QUESTION: I understand I only need to do this one time for all audiobooks?

Yes.

QUESTION: Why does it say “The default interactive shell is now zsh.” When the terminal window says myname – chsh < -bash – 80x12”?

Dunno. But you're definitely in bash, which is fine. That's what I use, and that's what the instructions primarily target.

QUESTION: So, does this mean my default shell is bash and I need to change the DEFAULT SHELL from zsh to bash? It’s confusing to me.

It's saying the default is different than your current shell. Ignore it.

QUESTION: Do I copy and past the above code block into a new terminal window?

Yep. Doesn't have to be a new window but that's fine.

QUESTION: Does the above results show the default shell is now BASH? Does this step make the default shell stay as BASH or do I have to do this step every time I borrow a book?

No, but don't worry about it. Just let it stay bash. The important thing is the error message, which clearly starts -bash: [...].

QUESTION: "And you might need to execute the final command manually if you didn't happen to capture the final newline when selecting the code block." How and where do I do this?

Just hit "enter" on your keyboard, but I can see from the "RESULTS" above that it ran.

I feel all I need is a simple tweek to the default shell for this to work for me. I just still don’t get how to get there. Am I leaving out important steps? Hope this makes some sense.

I think you are there :) — it looks like you have everything set up properly at this point, so just run the overdrive download ~/Desktop/Origin_9780739319307_2968609.odm command again.

P.S. if you run into any more errors, it makes it a lot easier to read your comments if you use code fences in your message — TIA. I edited your last comment to make it easier to follow, see those changes for reference.

chbrown commented 2 years ago

Regarding the original issue (last 4 comments above are totally off-topic), I can't reproduce with Homebrewed curl on up-to-date Catalina now, so maybe they fixed it upstream?

If not, I added support for an --insecure option, which simply causes each invocation of curl [...] to call curl --insecure [...] instead. This isn't a fix, per se, but makes the workaround much easier — just call overdrive download MyBook.odm --insecure or whatever.

Elsto007 commented 2 years ago

Again, thanks for your time and attention. I realize your help is as only good as what I am trying to show you and that's not really fair to you as you probably aren't getting the entire picture.

That being said,I tried the process again with a fresh loan of Giver of Stars and added the --insecure command. Got this:

EllenMaookPro15:~ ellenstock$ overdrive download/Users/ellenstock/Downloads/TheGiverofStars_9780525530206_4581755.odm --insecure -bash: overdrive: command not found EllenMaookPro15:~ ellenstock$

You're very generous with your time thanks again! Happy New Year!

dhnyny commented 2 years ago

If you get "command not found," I think that means it's not installed properly. Maybe reinstall.

Also, I think you need to change the directory to the location of your odm file before downloading. Since it's in your Downloads folder, the way you do this is:

cd ~/Downloads

Then your command to download would be:

overdrive download TheGiverofStars_9780525530206_4581755.odm --insecure

Elsto007 wrote on 1/2/22 10:59 pm:

Again, thanks for your time and attention. I realize your help is as only good as what I am trying to show you and that's not really fair to you as you probably aren't getting the entire picture.

That being said,I tried the process again with a fresh loan of Giver of Stars and added the --insecure command. Got this:

EllenMaookPro15:~ ellenstock$ overdrive download/Users/ellenstock/Downloads/TheGiverofStars_9780525530206_4581755.odm --insecure -bash: overdrive: command not found EllenMaookPro15:~ ellenstock$

You're very generous with your time thanks again! Happy New Year!

— Reply to this email directly, view it on GitHub https://github.com/chbrown/overdrive/issues/36#issuecomment-1003855777, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2VVGOV33NBIH4XCN7X2FTUUENJXANCNFSM5E4OVHYQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

chbrown commented 2 years ago

[...] Got this:

EllenMaookPro15:~ ellenstock$ overdrive download/Users/ellenstock/Downloads/TheGiverofStars_9780525530206_4581755.odm --insecure -bash: overdrive: command not found EllenMaookPro15:~ ellenstock$

@Elsto007 the --insecure bit isn't your problem — I was addressing what this issue was originally about. You have a different problem, so you should have created a new issue, but whatever... Your problem now is the same one that like 90% of 'Newbie' users run into, in light of which I just totally revamped the README. So please jump back to the main page (https://github.com/chbrown/overdrive), refresh, and follow the new and much-simplified Instructions section from top to bottom. Then, if you're still stuck, please create a new issue.

And again, code fences would be cool. Thanks!

Elsto007 commented 2 years ago

Hi Christopher,

SUCCESS! The revamped README really helped. Thanks for sticking with me. As a "newbie" I really appreciate your being extremely patient with me.

I even learned about code fences. 😊

Happy New Year!

Ellen

Elsto007 commented 2 years ago

Hi there, Thanks for your kind contribution.   Christopher revamped the README and that was very helpful.  I finally succeeded in learning how to download library books into MP3 files that I can continue to transfer to my ipod on Big Sur. Happy New Year! On Monday, January 3, 2022, 06:42:15 AM EST, dhnyny @.***> wrote:

If you get "command not found," I think that means it's not installed properly. Maybe reinstall.

Also, I think you need to change the directory to the location of your odm file before downloading. Since it's in your Downloads folder, the way you do this is:

cd ~/Downloads

Then your command to download would be:

overdrive download TheGiverofStars_9780525530206_4581755.odm --insecure

Elsto007 wrote on 1/2/22 10:59 pm:

Again, thanks for your time and attention. I realize your help is as only good as what I am trying to show you and that's not really fair to you as you probably aren't getting the entire picture.

That being said,I tried the process again with a fresh loan of Giver of Stars and added the --insecure command. Got this:

EllenMaookPro15:~ ellenstock$ overdrive download/Users/ellenstock/Downloads/TheGiverofStars_9780525530206_4581755.odm --insecure -bash: overdrive: command not found EllenMaookPro15:~ ellenstock$

You're very generous with your time thanks again! Happy New Year!

— Reply to this email directly, view it on GitHub https://github.com/chbrown/overdrive/issues/36#issuecomment-1003855777, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2VVGOV33NBIH4XCN7X2FTUUENJXANCNFSM5E4OVHYQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you commented.Message ID: @.***>