search

impliedchaos / mopidy-bandcamp

Mopidy backend for bandcamp
MIT License
21 stars 2 forks source link

Bandcamp rejecting clients with 403 #25

Open jpotier opened 3 months ago

jpotier commented 3 months ago 
Jul 18 10:00:21 nixos mopidy[668276]:   translate_uri error "403 Client Error: Forbidden for url: https://bandcamp.com/api/mobile/24/tralbum_details?band_id=3537931136&tralbum_type=t&tralbum_id=530455317"
Jul 18 10:00:21 nixos mopidy[668276]: WARNING  2024-07-18 10:00:21,193 [668276:Core-7 (_actor_loop)] mopidy.core.tracklist
Jul 18 10:00:21 nixos mopidy[668276]:   Track is not playable: bandcamp:track:3537931136-1106135643-530455317
Jul 18 10:00:21 nixos mopidy[668276]: ERROR    2024-07-18 10:00:21,235 [668276:BandcampBackend-5 (_actor_loop)] mopidy_bandcamp

Anybody seeing similar stuff? I can still connect and play bandcamp content through the website, so it doesn't seem to be some ip-based block.

orontee commented 3 months ago

The token stored in configuration may have expired. Have you checked after generating a new token?

jpotier commented 2 months ago

That's the strange thing though, I don't think I've ever configured and set a token to access bandcamp. Used to "just-work", and that seems to have changed recently.

orontee commented 2 months ago

Is there a bandcamp section in Mopidy's configuration? Does this bandcamp section has a identity key ?

[bandcamp]
identity = 7%09xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx%3D%09%7B%22ex%22%3Ax%2C%22id%22%3Axxxxxxxxxx%7D
jpotier commented 2 months ago

It does not!

jpotier commented 2 months ago

So, I've created myself a "fan" account, and captured the identity field from the cookie, and I still get:

Jul 24 22:03:17 kktdr mopidy[924152]: INFO     2024-07-24 22:03:17,774 [924152:MpdSession-17 (_actor_loop)] mopidy_mpd.session
Jul 24 22:03:17 kktdr mopidy[924152]:   New MPD connection from [::ffff:127.0.0.1]:32820
Jul 24 22:03:17 kktdr mopidy[924152]: ERROR    2024-07-24 22:03:17,902 [924152:BandcampBackend-5 (_actor_loop)] mopidy_bandcamp
Jul 24 22:03:17 kktdr mopidy[924152]:   Bandcamp failed to scrape "https://nodey.bandcamp.com/album/vinasounds-vol1"
Jul 24 22:03:17 kktdr mopidy[924152]: Traceback (most recent call last):
Jul 24 22:03:17 kktdr mopidy[924152]:   File "/nix/store/xdx2cz72p48yvrh7qr7pqi2abv0z49p9-mopidy-with-extensions-3.4.2/lib/python3.12/site-packages/mopidy_bandcamp/library.py", line 312, in lookup
Jul 24 22:03:17 kktdr mopidy[924152]:     resp = self.backend.bandcamp.scrape(url)
Jul 24 22:03:17 kktdr mopidy[924152]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 24 22:03:17 kktdr mopidy[924152]:   File "/nix/store/xdx2cz72p48yvrh7qr7pqi2abv0z49p9-mopidy-with-extensions-3.4.2/lib/python3.12/site-packages/mopidy_bandcamp/bandcamp.py", line 77, in scrape
Jul 24 22:03:17 kktdr mopidy[924152]:     resp.raise_for_status()
Jul 24 22:03:17 kktdr mopidy[924152]:   File "/nix/store/xdx2cz72p48yvrh7qr7pqi2abv0z49p9-mopidy-with-extensions-3.4.2/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
Jul 24 22:03:17 kktdr mopidy[924152]:     raise HTTPError(http_error_msg, response=self)
Jul 24 22:03:17 kktdr mopidy[924152]: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://nodey.bandcamp.com/album/vinasounds-vol1

My identity starts correctly with that "7%09…" and ends with "…%7D". I tried it quoted and unquoted in the mopidy config, neither work.

jpotier commented 2 months ago

Thing is, search is also broken for some reason:

Jul 24 22:10:36 kktdr mopidy[930417]: ERROR    2024-07-24 22:10:36,183 [930417:Core-10 (_actor_loop)] mopidy.core.library
Jul 24 22:10:36 kktdr mopidy[930417]:   FileBackend backend returned bad data: Expected a SearchResult instance, not []
Jul 24 22:10:36 kktdr mopidy[930417]: ERROR    2024-07-24 22:10:36,185 [930417:Core-10 (_actor_loop)] mopidy.core.library
Jul 24 22:10:36 kktdr mopidy[930417]:   StreamBackend backend returned bad data: Expected a SearchResult instance, not []
Jul 24 22:10:36 kktdr mopidy[930417]: ERROR    2024-07-24 22:10:36,313 [930417:BandcampBackend-5 (_actor_loop)] mopidy_bandcamp
Jul 24 22:10:36 kktdr mopidy[930417]:   Bandcamp failed to search.
Jul 24 22:10:36 kktdr mopidy[930417]: Traceback (most recent call last):
Jul 24 22:10:36 kktdr mopidy[930417]:   File "/nix/store/xdx2cz72p48yvrh7qr7pqi2abv0z49p9-mopidy-with-extensions-3.4.2/lib/python3.12/site-packages/mopidy_bandcamp/library.py", line 411, in search
Jul 24 22:10:36 kktdr mopidy[930417]:     resp = self.backend.bandcamp.search(q)
Jul 24 22:10:36 kktdr mopidy[930417]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 24 22:10:36 kktdr mopidy[930417]:   File "/nix/store/xdx2cz72p48yvrh7qr7pqi2abv0z49p9-mopidy-with-extensions-3.4.2/lib/python3.12/site-packages/mopidy_bandcamp/bandcamp.py", line 183, in search
Jul 24 22:10:36 kktdr mopidy[930417]:     js = self._get(f"{self.BASE_URL}/fuzzysearch/1/app_autocomplete?q={query}")
Jul 24 22:10:36 kktdr mopidy[930417]:          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 24 22:10:36 kktdr mopidy[930417]:   File "/nix/store/xdx2cz72p48yvrh7qr7pqi2abv0z49p9-mopidy-with-extensions-3.4.2/lib/python3.12/site-packages/mopidy_bandcamp/bandcamp.py", line 57, in _get
Jul 24 22:10:36 kktdr mopidy[930417]:     resp.raise_for_status()
Jul 24 22:10:36 kktdr mopidy[930417]:   File "/nix/store/xdx2cz72p48yvrh7qr7pqi2abv0z49p9-mopidy-with-extensions-3.4.2/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
Jul 24 22:10:36 kktdr mopidy[930417]:     raise HTTPError(http_error_msg, response=self)
Jul 24 22:10:36 kktdr mopidy[930417]: requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://bandcamp.com/api/fuzzysearch/1/app_autocomplete?q=nodey
Jul 24 22:10:36 kktdr mopidy[930417]: ERROR    2024-07-24 22:10:36,314 [930417:Core-10 (_actor_loop)] mopidy.core.library
Jul 24 22:10:36 kktdr mopidy[930417]:   BandcampBackend backend caused an exception.
Jul 24 22:10:36 kktdr mopidy[930417]: Traceback (most recent call last):
Jul 24 22:10:36 kktdr mopidy[930417]:   File "/nix/store/xdx2cz72p48yvrh7qr7pqi2abv0z49p9-mopidy-with-extensions-3.4.2/lib/python3.12/site-packages/mopidy/core/library.py", line 17, in _backend_error_handling
Jul 24 22:10:36 kktdr mopidy[930417]:     yield
Jul 24 22:10:36 kktdr mopidy[930417]:   File "/nix/store/xdx2cz72p48yvrh7qr7pqi2abv0z49p9-mopidy-with-extensions-3.4.2/lib/python3.12/site-packages/mopidy/core/library.py", line 330, in search
Jul 24 22:10:36 kktdr mopidy[930417]:     result = future.get()
Jul 24 22:10:36 kktdr mopidy[930417]:              ^^^^^^^^^^^^
Jul 24 22:10:36 kktdr mopidy[930417]:   File "/nix/store/xdx2cz72p48yvrh7qr7pqi2abv0z49p9-mopidy-with-extensions-3.4.2/lib/python3.12/site-packages/pykka/_threading.py", line 68, in get
Jul 24 22:10:36 kktdr mopidy[930417]:     raise exc_value
Jul 24 22:10:36 kktdr mopidy[930417]:   File "/nix/store/xdx2cz72p48yvrh7qr7pqi2abv0z49p9-mopidy-with-extensions-3.4.2/lib/python3.12/site-packages/pykka/_actor.py", line 238, in _actor_loop_running
Jul 24 22:10:36 kktdr mopidy[930417]:     response = self._handle_receive(envelope.message)
Jul 24 22:10:36 kktdr mopidy[930417]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 24 22:10:36 kktdr mopidy[930417]:   File "/nix/store/xdx2cz72p48yvrh7qr7pqi2abv0z49p9-mopidy-with-extensions-3.4.2/lib/python3.12/site-packages/pykka/_actor.py", line 349, in _handle_receive
Jul 24 22:10:36 kktdr mopidy[930417]:     return callee(*message.args, **message.kwargs)
Jul 24 22:10:36 kktdr mopidy[930417]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 24 22:10:36 kktdr mopidy[930417]:   File "/nix/store/xdx2cz72p48yvrh7qr7pqi2abv0z49p9-mopidy-with-extensions-3.4.2/lib/python3.12/site-packages/mopidy_bandcamp/library.py", line 414, in search
Jul 24 22:10:36 kktdr mopidy[930417]:     if "results" in resp:
Jul 24 22:10:36 kktdr mopidy[930417]:                     ^^^^
Jul 24 22:10:36 kktdr mopidy[930417]: UnboundLocalError: cannot access local variable 'resp' where it is not associated with a value

When I curl 'https://bandcamp.com/api/fuzzysearch/1/app_autocomplete?q=nodey' it works.

jpotier commented 2 months ago

I'm at a loss why mopidy gets rejected while curl doesn't.

orontee commented 2 months ago

Both requests aren't equivalent. Mopidy-Bandcamp adds headers (a User-Agent header and a Cookie header with value read from the configuration identity=...). You can add those informations to Curl command if you want to reproduce the problem out of Mopidy-Bandcamp.

jpotier commented 2 months ago

I see, I tried:

curl -s -H 'Cookie: identity=7%09...7D' -H 'User-Agent: Mopidy-Bandcamp/v1.1.5' https://bandcamp.com/api/fuzzysearch/1/app_autocomplete\?q\=nodey

And this return proper JSON results and 200 OK.

orontee commented 2 months ago

How did you checked Mopidy's config? By running sudo mopidyctl config (or mopidy config if not running as a systemd unit)?

jpotier commented 2 months ago

I did systemctl cat mopidy to inspect what it was started with, which gave me:

# /etc/systemd/system/mopidy.service
[Unit]
After=network-online.target sound.target
Description=mopidy music player daemon
Wants=network-online.target network-online.target

[Service]
...
ExecStart=/nix/store/xdx2cz72p48yvrh7qr7pqi2abv0z49p9-mopidy-with-extensions-3.4.2/bin/mopidy --config /nix/store/2gpffwrzsc8q3q1z29w76h8gyz107nn2-mopidy.conf
User=mopidy

[Install]
WantedBy=multi-user.target

and then I simply inspected the file at /nix/store/2gpffwrzsc8q3q1z29w76h8gyz107nn2-mopidy.conf

woffs commented 2 months ago

same here, NixOS, too. curl with identity cookie works and returns JSON, mopidy gets 403. Tested with https://bandcamp.com/api/fan/2/collection_summary It worked some weeks ago. Is Bandcamp blocking something recently? I wonder how to reproduce outside mopidy.

woffs commented 2 months ago

Maybe a missing "Accept" header? I'll test.

woffs commented 2 months ago

no, accept-header does not matter. it even succeeds using openssl s_client. It still fails with python3.11 inside mopidy. Did not test with python outside mopidy yet.

woffs commented 2 months ago

maybe rather a bug in mopidy httpclient? But I'm still at 3.4.2

jpotier commented 2 months ago

I went back to ~6 months back nixpkgs (for mopidy, mopidyPackages and all extensions), before the switch to mopidy 3.4.2, and it still gives the same error. I'm pretty sure it used to work correctly then.

woffs commented 2 months ago

I'm not able to reproduce it outside mopidy. curl, openssl s_client and even python httpclient get a result of 200 with exact the same request parameters, which are

GET /api/fan/2/collection_summary HTTP/1.1
Host: bandcamp.com
User-Agent: Mopidy-Bandcamp/1.1.5 Mopidy/3.4.2 CPython/3.11.9
Accept-Encoding: gzip, deflate, br
Accept: */*
Connection: keep-alive
Cookie: identity=7%09xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx%7D

I even checked it by redirecting the request to an own server.

I'm not fluent enough in python for debugging what exactly is happening. I wonder if this is a nixos issue, a python issue or a bandcamp issue.

orontee commented 2 months ago

@woffs I am not convinced it's a Mopidy-Bandcamp issue since visiting my own Bandcamp collection through Iris or Argos works quite well here.

matthias@argos:~ $ python -m pip freeze | grep -i mopidy
Mopidy==3.4.2
Mopidy-Bandcamp==1.1.5
Mopidy-InternetArchive==3.0.1
Mopidy-Iris @ file:///home/matthias/T%C3%A9l%C3%A9chargements/Mopidy-Iris-3.69.3.tar.gz
Mopidy-Listenbrainz==0.2.1
Mopidy-Local==3.2.1
Mopidy-Podcast==3.0.1
Mopidy-SomaFM==2.0.2
matthias@argos:~ $ python --version
Python 3.11.2
matthias@argos:~ $ lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 12 (bookworm)
Release:    12
Codename:   bookworm
woffs commented 2 months ago

even python httpclient get a result of 200

I was wrong, I re-tested with "python requests" using Debian and using NixOS, and the same query gets 403 with NixOS and 200 with Debian. The test query is simply request("GET","https://bandcamp.com/api/fan/2/collection_summary") and the good result should be 200 and response text {"error":true,"error_message":"must be logged in"}

The bad result is 403 and

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
  <head>
    <title>403 Forbidden</title>
  </head>
  <body>
    <h1>Error 403 Forbidden</h1>
    <p>Forbidden</p>
    <h3>Error 54113</h3>
    <p>Details: cache-fra-etou8220062-FRA 1722714486 2138973186</p>
    <hr>
    <p>Varnish cache server</p>
  </body>
</html>
woffs commented 2 months ago

NixOS 23.05 works, NixOS 23.11 does not. Bandcamp definitely did change something, but I'll bisect what change in python packaging relates to that.

woffs commented 2 months ago

git-bisect states that update of python-urllib3 from 1.26.16 to 2.0.4 https://github.com/NixOS/nixpkgs/commit/6beb683e972947a36ab08bad407b9861388524fc was the bad commit. So now we know how to workaround on NixOS, we have to bisect urllib3 to find out what changed

pquentin commented 1 month ago

Hello! urllib3 developer here. Bandcamp.com weirdly requires the TLS cipher suites that are used in urllib3 1.26.x, even if it does not use them. Here's the workaround to continue using urllib3 2.x: https://github.com/urllib3/urllib3/issues/3439#issuecomment-2306400349

woffs commented 1 month ago

based on what @pquentin says I have a workaround with https://gist.github.com/woffs/083f57f6cd0f4bfe49a6e97e1eb2ed60 I am not experienced with python though, so maybe someone wants to beautify this, but at least it works well.