Scrobbler refuses last.fm certificate if built with GnuTLS - Githubissues

Audacious-Bot commented 6 months ago

Author Name: Luís Picciochi Original Redmine Issue: https://redmine.audacious-media-player.org/issues/343 Original Date: 2013-09-05

So, it appears that last.fm has updated its https certificate. Until distributions update their recognised CAs (http://packages.debian.org/ca-certificates ?), the scrobbler is unable to connect, as the certificate is deemed invalid. At least Debian testing is suffering from this.

I haven't chosen a solution yet. Possibilities thought of thus far:

Allow the user to say he doesn't care if the certificate is invalid, and connect anyway. ** This is possibly the worst solution. The scrobbling URL is not configurable, and so we only connect to last.fm's https address. If the certificate is untrusted, a connection to that address should never established.
Allow the user to select whether to connect using https or http.
Allow the user to select whether to connect using https, with possibility of fallback to http, in case the https connection is not possible.

Any of these is sub-optimal, as using plain http leaves the user subject to replay attacks if his traffic is sniffed.

The best solution would be if the certificate was good and recognised by the distro(s). It might be necessary to investigate this further with last.fm and/or to annoy the distros' packagers of CA certificates for this to be correctly addressed.

$ wget  https://last.fm 
--2013-09-05 22:04:53--  https://last.fm/
Resolving last.fm (last.fm)... 195.24.232.203
Connecting to last.fm (last.fm)|195.24.232.203|:443... connected.
ERROR: The certificate of `last.fm' is not trusted.
ERROR: The certificate of `last.fm' hasn't got a known issuer.

$ openssl s_client -showcerts -connect last.fm:443
CONNECTED(00000003)
depth=0 OU = Domain Control Validated, CN = *.last.fm
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 OU = Domain Control Validated, CN = *.last.fm
verify error:num=27:certificate not trusted
verify return:1
depth=0 OU = Domain Control Validated, CN = *.last.fm
verify error:num=21:unable to verify the first certificate
verify return:1
---
(...)
---
Server certificate
subject=/OU=Domain Control Validated/CN=*.last.fm
issuer=/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Domain Validation CA - G2
---
(...)
    Verify return code: 21 (unable to verify the first certificate)

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Luís Picciochi Original Date: 2013-09-05T23:35:20Z

Disregard the previous URLs.

They should be:

$ wget https://ws.audioscrobbler.com/2.0/
--2013-09-05 22:32:56--  https://ws.audioscrobbler.com/2.0/
Resolving ws.audioscrobbler.com (ws.audioscrobbler.com)... 195.24.232.205
Connecting to ws.audioscrobbler.com (ws.audioscrobbler.com)|195.24.232.205|:443... connected.
ERROR: The certificate of `ws.audioscrobbler.com' is not trusted.
ERROR: The certificate of `ws.audioscrobbler.com' hasn't got a known issuer.

$ openssl s_client -showcerts -connect ws.audioscrobbler.com:443
CONNECTED(00000003)
depth=2 C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
 0 s:/OU=Domain Control Validated/CN=*.audioscrobbler.com
   i:/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Domain Validation CA - G2
(...)
 1 s:/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
   i:/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
(...)
 2 s:/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Domain Validation CA - G2
   i:/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
(...)
Server certificate
subject=/OU=Domain Control Validated/CN=*.audioscrobbler.com
issuer=/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Domain Validation CA - G2
(...)
    Verify return code: 19 (self signed certificate in certificate chain)

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Ruei-Yuan Lu Original Date: 2013-09-08T08:09:18Z

Currently I built the scrobbler plugin with libcurl4-openssl-dev, instead of libcurl4-gnutls-dev, to avoid this problem.

It works on my Debian Wheezy machine. :)

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Luís Picciochi Original Date: 2013-09-12T23:47:45Z

Thanks, Ruey-Yuan.

After investigating this, it seems that the issue may be related with the order the certificates are sent by last.fm. OpenSSL seems to be more resilient to server (mis?)configurations. After checking if it really solves this issue, a strong dependency on OpenSSL to avoid compiling against GnuTLS may solve this.

Relating to the above openssl command, if CApath is given on the command line, OpenSSL will validate the certificate by looking at the installed CA certs:

(...)
    Verify return code: 0 (ok)

Doing the same with GnuTLS will not:

Processed 159 CA certificate(s).
Resolving 'ws.audioscrobbler.com'...
Connecting to '195.24.232.205:443'...
*** Verifying server certificate failed...
*** Fatal error: Error in the certificate.
*** Handshake has failed
GnuTLS error: Error in the certificate.

...which may be related to this: ??gnutls is tolerant and if the correct chain is provided in the front of the list then it will verify the chain and not complain. The problem is if there is no proper chain e.g if certificates are thrown in a random order.?? From https://lists.gnu.org/archive/html/help-gnutls/2012-03/msg00038.html

But I'll need further confirmation on this, to get rid of any biases.

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: John Lindgren Original Date: 2013-09-15T00:02:06Z

Trying to load https://last.fm in Firefox on Windows 7 also complains about a bad certificate. I've notified last.fm of the problem. http://www.last.fm/forum/21713/_/2215292

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Luís Picciochi Original Date: 2013-09-16T23:25:10Z

I couldn't reproduce that with an up to date Firefox (23.0.1) and Windows 7 (64-bit) with all updates installed. Did you try with all updates?

Anyway, the address in question is ws.audioscrobbler.com, not last.fm. Can you reproduce that on Windows with ws.audioscrobbler.com?

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: John Lindgren Original Date: 2013-09-19T04:52:15Z

I just now updated Firefox to 24.0 (it was 23.0.1) and that fixed the warning on https://last.fm. Neither 23.0.1 nor 24.0 had a problem with https://ws.audioscrobbler.com.

I hate these problems where none of the symptoms make any sense ...

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Luís Picciochi Original Date: 2013-09-19T23:08:10Z

I'll try to look at this with some more time during the next weekend. No promises as usual, though.

John: on #321 you said that you also see this problem on Windows, with OpenSSL. Does that still happen? Windows updates frequently update certificates. Some needed certificate could been missing (?) but now isn't, coincidentally with you updating Firefox.

This is what we know now: Linux:

scrobbler built with libcurl-gnutls, doesn't recognise the certificate.
scrobbler built with libcurl-openssl, trusting Ruei-Yua's comment (#343-2), recognises the certificate

Windows:

scrobbler built with libcurl-openssl, doesn't recognise the certificate.

This is starting to look more and more a last.fm-side problem.

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Luís Picciochi Original Date: 2013-09-25T22:42:48Z

Here are the results from compiling and using the scrobbler on a Debian testing machine, with all updates as of when I performed the tests (Sep 21st, 2013):

libcurl4-gnutls-dev 7.32.0-1:

scrobbler_communication.c:151 [send_message_to_lastfm]: Could not communicate with last.fm: Peer certificate cannot be authenticated with given CA certificates.

libcurl4-nss-dev 7.32.0-1:

scrobbler_communication.c:151 [send_message_to_lastfm]: Could not communicate with last.fm: Problem with the SSL CA cert (path? access rights?).

libcurl4-openssl-dev 7.32.0-1: Everything works.

I also tested changing the URL to https://google.com and the connection was well established with both GnuTLS and OpenSSL. Obviously, Google didn't like the subsequent requests, but it's cert was recognised, that's all. Using the NSS backend gave the same message as before.

This is leading me to conclude that:

The last.fm certificate is not OK;
NSS support is not well configured on my machine. Apparently some additional configuration is needed to build a certificate DB, which could be built by the distro[1][2] but for some reason it isn't. It makes no sense to force users/installers to do this kind of magic just to scrobble from Audacious.

1 - http://curl.haxx.se/docs/sslcerts.html (see Peer SSL Verification with NSS) 2 - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=537866

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: John Lindgren Original Date: 2013-09-27T01:44:24Z

I still get this on Windows:

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Luís Picciochi Original Date: 2013-09-27T22:49:47Z

I have finally received a confirmation from last.fm that the issue is on their side and that they are working to solve it.

Nevertheless, please leave this issue open until the problem is definitely solved.

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: John Lindgren Original Date: 2013-09-28T00:29:02Z

Luís Picciochi wrote:

Nevertheless, please leave this issue open until the problem is definitely solved. Sure.

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Виктор Пономарёв Original Date: 2013-10-06T16:59:13Z

Luís, what is known about the correct certificate?

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Luís Picciochi Original Date: 2013-10-08T21:14:50Z

I still have no answer from last.fm.

However, yesterday I updated my wget installation, and that came with a dependency on libgnutls28. The previous version was libgnutls26. The libgnutls26 is version 2.12.23-7, libgnutls28 is 3.2.4-4.

wget is now able to connect to https://ws.audioscrobbler.com , although it gets a 400 error subsequently (but that's not an issue):

--2013-10-08 20:02:34--  https://ws.audioscrobbler.com/2.0/
Resolving ws.audioscrobbler.com (ws.audioscrobbler.com)... 195.24.233.55
Connecting to ws.audioscrobbler.com (ws.audioscrobbler.com)|195.24.233.55|:443... connected.
HTTP request sent, awaiting response... 400 Bad Request
2013-10-08 20:02:36 ERROR 400: Bad Request.
$

gnutls-bin now also depends on libgnutls28:

Processed 164 CA certificate(s).
Resolving 'ws.audioscrobbler.com'...
Connecting to '195.24.233.55:443'...

- Certificate type: X.509
- Got a certificate list of 3 certificates.
- Certificate[0] info:
(...)
- Status: The certificate is trusted. 
(...)
- Handshake was completed

This is leading me to think that quite likely, libgnutls28 (v3.2) is capable of dealing with last.fm's apparently crapped-up certificates. I'm planning to test this further by compiling libcurl against this more recent libgnutls, and see how the scrobbler goes. I'll probably do this over the weekend.

If there's anyone on the audience with more free time than me to do this, feel free to reply with your experience.

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Luís Picciochi Original Date: 2013-10-15T00:42:01Z

I just tested the scrobbler with libcurl 7.33.0, which was compiled against GnuTLS 3.2.4. The scrobbler was able to contact last.fm, and submitted the most recent tracks. Due to a restriction on last.fm's side, older plays were lost. I could only submit tracks played on the last week. The others were at least 3 weeks old, and last.fm happily discarded them.

The solution is then to enforce to build against libcurl, which has to have been built either against libopenssl, or against libgnutls >=3.2.4 (this might be trickier than it seems).

jlindgren: when you tested on Windows, what was libcurl built against? libcurl? libopenssl? Which version?

I'll leave this here so it gets noticed: Users will lose their oldest scrobbles due to this issue. If users want to re-scrobble their tracks, they should make a backup of ~/.config/audacious/scrobbler.log before upgrading libgnutls and prepare to re-scrobble them after this is solved. The backed-up scrobbler.log can be opened with any text editor.

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Luís Picciochi Original Date: 2013-10-15T00:44:51Z

Of course, that should read: ??what was libcurl built against? libgnutls? libopenssl???

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: John Lindgren Original Date: 2013-10-16T05:24:13Z

I am using OpenSSL 1.0.1e on Windows. The first problem was that OpenSSL didn't know where to look for CA certificates. After copying some certificates from my Linux partition, I can connect to google.com:443 successfully, but last.fm:443 is giving me the same output as in the original bug description:

depth=0 OU = Domain Control Validated, CN = *.last.fm
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 OU = Domain Control Validated, CN = *.last.fm
verify error:num=27:certificate not trusted
verify return:1
depth=0 OU = Domain Control Validated, CN = *.last.fm
verify error:num=21:unable to verify the first certificate
verify return:1

Edit: I get the same output when I run openssl from the command line in Arch Linux, but in spite of that, the scrobbler can connect. So maybe libcurl+OpenSSL is more tolerant than OpenSSL on its own. I will try some more tests on Windows when I can find the time.

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Luís Picciochi Original Date: 2013-10-16T10:17:38Z

Please don't test against last.fm:443 but ws.audioscrobbler.com:443 .

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Bob Bib Original Date: 2013-10-20T03:21:14Z

Luís Picciochi wrote:

I still have no answer from last.fm.

Have you (or anyone else) tried to contact them one more time?

Due to a restriction on last.fm's side, older plays were lost. I could only submit tracks played on the last week. The others were at least 3 weeks old, and last.fm happily discarded them. ... If users want to re-scrobble their tracks, they should make a backup of before upgrading libgnutls and prepare to re-scrobble them after this is solved

And this looks like separate issue. Do you have any more info on it?

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Luís Picciochi Original Date: 2013-10-23T00:48:57Z

I still have no idea on how to portably ensure libcurl was built against gnutls >= 3.2.4. We could check "libcurl's version string":http://curl.haxx.se/libcurl/c/curl_version.html, but that seems too fragile for the long term. Suggestions are welcome.

Bob Bib wrote:

Luís Picciochi wrote:

I still have no answer from last.fm.

Have you (or anyone else) tried to contact them one more time? I was able to communicate with a last.fm person again today. There's still no progress on their side. The issue is still "in the backlog".

Due to a restriction on last.fm's side, older plays were lost. I could only submit tracks played on the last week. The others were at least 3 weeks old, and last.fm happily discarded them. ... If users want to re-scrobble their tracks, they should make a backup of before upgrading libgnutls and prepare to re-scrobble them after this is solved

And this looks like separate issue. Do you have any more info on it?

Yes and no. I got confirmation from them that +scrobbles older than 2 weeks are bluntly discarded by last.fm+.

The Audacious scrobbler implements their API quite strictly, so where it says:

http://www.last.fm/api/scrobbling wrote:

Lfm error codes that indicate a scrobble request should be retried are:

11. Service Offline - This service is temporarily offline, try again later.

16. The service is temporarily unavailable, please try again.

Additionally this lfm error code indicates that the client should reauthenticate to get a new session key before retrying the request:

9. Invalid session key - Please re-authenticate

All other error codes indicate the scrobble request was incorrectly formed in some way +and should not be retried+.

I have interpreted this as to only retry attempts that got those errors. last.fm returns an "OK" status for discarded tracks, so the scrobbler happily goes on with that and doesn't look back. A workaround can be implemented to keep retrying refused, old tracks. I don't like this solution as it goes against last.fm's spec, but I agree that this is better than losing scrobbles. Once again, I'll try to implement that over the weekend.

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Bob Bib Original Date: 2013-10-23T02:22:43Z

Luís Picciochi wrote:

I got confirmation from them that +scrobbles older than 2 weeks are bluntly discarded by last.fm+.

The Audacious scrobbler implements their API quite strictly, so where it says: ...

I just wonder if it is an old bad behavior of Last.fm or a new "bug feature"; if it's new, it obviously can be not reflected in documentation...

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Luís Picciochi Original Date: 2013-10-24T00:02:02Z

Bob Bib wrote:

I just wonder if it is an old bad behavior of Last.fm or a new "bug feature"; if it's new, it obviously can be not reflected in documentation...

Last.fm has always had a threshold for dropping old scrobbles, and they always documented it. If that threshold was reduced to 2 weeks or 2 days, it's indifferent for the correctness of the specification & implementations.

I'm not seeing a clean way out of this until either last.fm fix their certificates and/or libcurl packagers increase their dependency on libgnutls to libgnutls28.

Until then, I'll probably end up implementing a dirty, version string-based hack to prevent audacious to compile against libcurl with libgnutls < 28.

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: John Lindgren Original Date: 2013-10-24T00:17:49Z

Luís Picciochi wrote:

Until then, I'll probably end up implementing a dirty, version string-based hack to prevent audacious to compile against libcurl with libgnutls < 28.

I don't think it's necessary to add a compile-time check, especially since GnuTLS isn't even a direct dependency. The issue should be documented, though. If you email me a few sentences summarizing the issue, I'll add them to http://audacious-media-player.org/problems.

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Luís Picciochi Original Date: 2013-10-28T02:28:50Z

I started working on a quick fix so no scrobbles are lost. See http://redmine.audacious-media-player.org/issues/260#note-1

Once this is commited, users will have to update their Audacious before last.fm fixes their servers. If not, they will still have to back-up their scrobbler.log file (which I recommend doing ASAP anyway).

jlindgren: I'll hand you a description of these problems + workarounds once I have the code ready for integrating with the main branch (hopefully tomorrow or on Wednesday).

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Виктор Пономарёв Original Date: 2013-10-29T09:04:02Z

Luís, whether correctly I understand that I will have to update a Audacious with your decision before there will be a communication with last.fm?

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: John Lindgren Original Date: 2013-10-30T01:43:30Z

Linking back to http://audacious-media-player.org/news/24-scrobbler-issue for reference.

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Виктор Пономарёв Original Date: 2013-11-16T13:40:22Z

What progress now?

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Jon Hallier Original Date: 2013-12-10T16:40:16Z

Hi from Last.hq,

Sorry this has taken so long, but we've finally addressed the ordering of the certificates and this problem should now be resolved. If it isn't, please let me know (best to send a pm to my last.fm account) and I'll ask the team to take another look.

Cheers, -Jon

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Luís Picciochi Original Date: 2013-12-13T23:35:03Z

Well, better late than never. Thanks for the update, Jon.

I confirm that at least some communication is possible with audacious 3.4.1 (with libcurl built against an old libgnutls). I didn't test all of the interactions yet (including scrobbling), though.

Can someone confirm if the Windows version is also working?

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: John Lindgren Original Date: 2013-12-14T20:24:26Z

I was looking into the Windows version a while back, and it seems that OpenSSL doesn't know how to access the client-side certificates that Windows provides. There are various bits of code posted online that will fetch the certificates and feed them into OpenSSL, but I haven't implemented such a workaround in Audacious (nor will I have time in the near future).

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Виктор Пономарёв Original Date: 2013-12-15T17:14:16Z

Audacious 3.4.2 communicate with Last.fm (I checked by logging into another profile).

Will I lose my listening when I'm connected to Last.fm into base profile?

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Виктор Пономарёв Original Date: 2013-12-15T17:23:05Z

Виктор Пономарёв wrote:

Audacious 3.4.2 communicate with Last.fm (I checked by logging into another profile).

Will I lose my listening when I'm connected to Last.fm into base profile?

I meant Linux version.

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Luís Picciochi Original Date: 2013-12-15T17:37:28Z

+Copy the ~/.config/audacious/scrobbler.log file to a backup location.+

Then start Audacious configured with your profile. Look at your last.fm profile to see all your old plays start appearing.

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Виктор Пономарёв Original Date: 2013-12-16T04:49:03Z

Something went wrong.

I had 12 590 plays on Last.fm.

scrobbler.log contains 3540 songs.

When I listen the song, list of plays loaded on the server. The first batch of sent contained line 3517 - 3540. This can be seen on page http://www.lastfm.ru/user/v_pv/tracks?view=compact&page=58 . First song is "Kick Bong – Voice Of Resurrection", second song is "The Gentle Revolution – Stealth and Cunning", third song is "Hol Baumann – Breathe" etc. The final song is "Cell – Under Your Mind".
The second batch of sent contained unknown positions. I don't know, from what place of the file tracks are loaded. There was a set of repetitions of one track (for example, http://www.lastfm.ru/user/v_pv).
There was a loading of 3402 tracks from 3540. After that Last.fm displayed listenings, but didn't fix it.

Such affairs.

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Luís Picciochi Original Date: 2013-12-16T22:57:57Z

You didn't happen to start audacious with -V and save the output to a file, did you?

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Виктор Пономарёв Original Date: 2013-12-17T06:25:31Z

Luís Picciochi wrote:

You didn't happen to start audacious with -V and save the output to a file, did you?

Yes, of course.

I plan to clear the list of listenings and reload sсrobbler.log. Please remind me the command with -V attribute and output parameter.

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Виктор Пономарёв Original Date: 2013-12-20T19:36:13Z

I cleared the list of listenings. I wanted to load the list anew. 699 compositions were loaded.

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: Luís Picciochi Original Date: 2013-12-20T20:43:51Z

I deleted the log from this issue because it contained your full submissions, including signature - this can allow others to re-scrobble those plays. I should remove that output from the scrobbler's debug mode.

Anyway, thanks for all the input. I'll try to look into this on Sunday. These are busy days around here. :-)

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: John Lindgren Original Date: 2014-01-11T20:26:31Z

Can we close this since the certificate ordering problem is resolved? Further support questions can be asked and answered on the forums.

Audacious-Bot commented 6 months ago

Original Redmine Comment Author Name: John Lindgren Original Date: 2014-01-19T02:16:29Z

Closing.

audacious-media-player / audacious

Scrobbler refuses last.fm certificate if built with GnuTLS #477