search

snejus / beetcamp

Bandcamp autotagger source for beets (http://beets.io)
GNU General Public License v2.0
64 stars 11 forks source link

KeyError: 'url' #37

Closed mswsn closed 1 year ago

mswsn commented 1 year ago

I'm occasionally getting crashes while using beets on an rpi4 and the culprit seems to be the beetcamp plugin.

Sometimes the hangup seems to be on particular albums and I'm forced to disable the plugin until that one is cleared. I've also noticed that the error always tends to occur if I have left beets alone for a few minutes.

Traceback (most recent call last):
  File "/home/mswsn/.local/bin/beet", line 8, in <module>
    sys.exit(main())
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/ui/__init__.py", line 1285, in main
    _raw_main(args)
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/ui/__init__.py", line 1272, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/ui/commands.py", line 973, in import_func
    import_files(lib, paths, query)
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/ui/commands.py", line 943, in import_files
    session.run()
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/importer.py", line 340, in run
    pl.run_parallel(QUEUE_SIZE)
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/util/pipeline.py", line 446, in run_parallel
    raise exc_info[1].with_traceback(exc_info[2])
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/util/pipeline.py", line 311, in run
    out = self.coro.send(msg)
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/util/pipeline.py", line 193, in coro
    func(*(args + (task,)))
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/importer.py", line 1376, in lookup_candidates
    task.lookup_candidates()
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/importer.py", line 660, in lookup_candidates
    autotag.tag_album(self.items, search_ids=self.search_ids)
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/autotag/match.py", line 461, in tag_album
    for matched_candidate in hooks.album_candidates(items,
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/plugins.py", line 573, in decorated
    for v in generator(*args, **kwargs):
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/autotag/hooks.py", line 629, in album_candidates
    yield from plugins.candidates(items, artist, album, va_likely,
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/plugins.py", line 384, in candidates
    yield from plugin.candidates(items, artist, album, va_likely,
  File "/home/mswsn/.local/lib/python3.10/site-packages/beetsplug/bandcamp/__init__.py", line 214, in candidates
    for res in filter(truth, map(self.get_album_info, results)):
KeyError: 'url'

Here's what I'm getting when using the -v flag:

Sending event: import_task_created
Sending event: import_task_start
Looking up: /styx/music/swinsian/Hear & Now/Aurora Baleare
Tagging Hear & Now - Aurora Baleare
No album ID found.
Search terms: Hear & Now - Aurora Baleare
Album might be VA: False
Searching for MusicBrainz releases with: {'release': 'aurora baleare', 'artist': 'hear & now', 'tracks': '8'}
Requesting MusicBrainz release df62a53e-6071-49c7-8870-95dca450f142
Sending event: import_task_created
Sending event: import_task_created
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: Hear & Now - Aurora Baleare (df62a53e-6071-49c7-8870-95dca450f142)
Computing track assignment...
...done.
Success. Distance: 0.02
Requesting MusicBrainz release 366b692f-854c-4b8a-9778-7f4fd862e8ad
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: Hear & Now - Milvus (366b692f-854c-4b8a-9778-7f4fd862e8ad)
Computing track assignment...
...done.
Success. Distance: 0.54
Requesting MusicBrainz release f115fd29-f4c9-43e5-8447-77d62ebd539d
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: Hear - 5 Pieces (f115fd29-f4c9-43e5-8447-77d62ebd539d)
Computing track assignment...
...done.
Success. Distance: 0.68
Requesting MusicBrainz release f81b3578-7193-42fa-b220-8d70cd4031e7
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: KMLN & Roderic - Aurora (f81b3578-7193-42fa-b220-8d70cd4031e7)
Computing track assignment...
...done.
Success. Distance: 0.62
Requesting MusicBrainz release 5864ea59-d21f-4522-8068-10ed48c89d29
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: mb_track_extract
Sending event: mb_track_extract
Sending event: mb_album_extract
Sending event: albuminfo_received
Candidate: Vintage & Morelli - Aurora (5864ea59-d21f-4522-8068-10ed48c89d29)
Computing track assignment...
...done.
Success. Distance: 0.64
bandcamp: Searching as for Aurora Baleare using {'query': 'Aurora Baleare', 'artist': 'Hear & Now', 'label': '', 'search_type': 'a'}
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: import_task_created
Sending event: albuminfo_received
Candidate: Hear & Now - Aurora Baleare (https://claremont56.bandcamp.com/album/aurora-baleare)
Computing track assignment...
...done.
Success. Distance: 0.17
Sending event: albuminfo_received
Candidate: Hear & Now - Aurora Baleare (https://claremont56.bandcamp.com/album/aurora-baleare#p2376566435)
Computing track assignment...
...done.
Success. Distance: 0.18
Sending event: import_task_created
Traceback (most recent call last):
  File "/home/mswsn/.local/bin/beet", line 8, in <module>
    sys.exit(main())
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/ui/__init__.py", line 1285, in main
    _raw_main(args)
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/ui/__init__.py", line 1272, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/ui/commands.py", line 973, in import_func
    import_files(lib, paths, query)
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/ui/commands.py", line 943, in import_files
    session.run()
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/importer.py", line 340, in run
    pl.run_parallel(QUEUE_SIZE)
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/util/pipeline.py", line 446, in run_parallel
    raise exc_info[1].with_traceback(exc_info[2])
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/util/pipeline.py", line 311, in run
    out = self.coro.send(msg)
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/util/pipeline.py", line 193, in coro
    func(*(args + (task,)))
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/importer.py", line 1376, in lookup_candidates
    task.lookup_candidates()
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/importer.py", line 660, in lookup_candidates
    autotag.tag_album(self.items, search_ids=self.search_ids)
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/autotag/match.py", line 461, in tag_album
    for matched_candidate in hooks.album_candidates(items,
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/plugins.py", line 573, in decorated
    for v in generator(*args, **kwargs):
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/autotag/hooks.py", line 629, in album_candidates
    yield from plugins.candidates(items, artist, album, va_likely,
  File "/home/mswsn/.local/lib/python3.10/site-packages/beets/plugins.py", line 384, in candidates
    yield from plugin.candidates(items, artist, album, va_likely,
  File "/home/mswsn/.local/lib/python3.10/site-packages/beetsplug/bandcamp/__init__.py", line 214, in candidates
    for res in filter(truth, map(self.get_album_info, results)):
KeyError: 'url'
tomauty commented 1 year ago

Following because I get this too. Can't find a pattern as to when. Might look into it codewise later

tomauty commented 1 year ago

Looks like it crashes when it encounters a recaptcha URL. Which explains why this happens more often the more I keep importing. This was a crashing results object from around line 214

results ['https://greenovamusic.bandcamp.com/album/back-ellin-crack', 'https://www.recaptcha.net/recaptcha/enterprise.js']
tomauty commented 1 year ago

@mwassen I was able to temporarily make the crashes stop by setting search_max to 1. Not great as it limits to one possible result, but recaptchas get filtered out.

mswsn commented 1 year ago

Cheers, at least good to have a band-aid for now.

snejus commented 1 year ago

Sorry this took ages to respond to, currently looking into it!

snejus commented 1 year ago

@tomauty you are completely right regarding the recaptcha url! See below for a confirmation of your issue (see results number 7 and 18)

image

snejus commented 1 year ago

Have fixed it, see the equivalent below

image

Will get it released in a moment.

snejus commented 1 year ago

Done, try updating beetcamp to 0.16.1!

marty-oehme commented 1 year ago

I am still constantly getting a url error with beetcamp, have updated to 0.16.1 as far as I know with a direct git clone to the beets environment (is there a cli way of getting the version?)

I am not sure if it is the same issue, due to recaptchas. However, the traceback looks basically identical:

beetcamp stack traceback ``` Traceback (most recent call last): File "/home/marty/.local/bin/beet", line 8, in sys.exit(main()) File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/ui/__init__.py", line 1285, in main _raw_main(args) File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/ui/__init__.py", line 1272, in _raw_main subcommand.func(lib, suboptions, subargs) File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/ui/commands.py", line 973, in import_func import_files(lib, paths, query) File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/ui/commands.py", line 943, in import_files session.run() File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/importer.py", line 340, in run pl.run_parallel(QUEUE_SIZE) File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/util/pipeline.py", line 446, in run_parallel raise exc_info[1].with_traceback(exc_info[2]) File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/util/pipeline.py", line 311, in run out = self.coro.send(msg) File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/util/pipeline.py", line 193, in coro func(*(args + (task,))) File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/importer.py", line 1376, in lookup_candidates task.lookup_candidates() File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/importer.py", line 660, in lookup_candidates autotag.tag_album(self.items, search_ids=self.search_ids) File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/autotag/match.py", line 461, in tag_album for matched_candidate in hooks.album_candidates(items, File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/plugins.py", line 573, in decorated for v in generator(*args, **kwargs): File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/autotag/hooks.py", line 629, in album_candidates yield from plugins.candidates(items, artist, album, va_likely, File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beets/plugins.py", line 384, in candidates yield from plugin.candidates(items, artist, album, va_likely, File "/home/marty/.local/pipx/venvs/beets/lib/python3.10/site-packages/beetsplug/bandcamp/__init__.py", line 215, in candidates for res in filter(truth, map(self.get_album_info, results)): KeyError: 'url' ```

The error happens consistently on this link: https://derleemusic.com/album/off-the-beatn-path-2, while other albums by the same artist worked.

beets verbose output ``` user configuration: /home/marty/.config/beets/config.yaml data directory: /home/marty/.config/beets plugin paths: lastgenre: Loading canonicalization tree ~/.config/beets/lastgenre_canonicallist.yaml inline: adding item field multidisc inline: adding item field artist_differs Sending event: pluginload library database: /home/marty/.local/share/beets/library.db library directory: /home/marty/media/audio/music Sending event: library_opened Sending event: import_begin Sending event: import_task_created Sending event: import_task_start Looking up: /home/marty/inbox/music/Derlee/Off The Beat'n Path Tagging Derlee - Off The Beat'n Path Searching for discovered album ID: 11684198 Requesting MusicBrainz release 11684198 Invalid MBID (11684198). Search terms: Derlee - Off The Beat'n Path Additional search terms: {'year': 2015, 'label': 'Philos Records', 'catalognum': '', 'country': '', 'media': ''} Album might be VA: False Searching for MusicBrainz releases with: {'release': "off the beat'n path", 'artist': 'derlee', 'tracks': '16', 'date': '2015', 'label': 'philos records'} Sending event: import_task_created Sending event: import_task_created Sending event: import_task_created Sending event: import_task_created Sending event: import_task_created Requesting MusicBrainz release fbfcd09e-8a2b-4599-ae86-e1389f7bf49a Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_album_extract Sending event: albuminfo_received Candidate: Dave Koz - Off the Beaten Path (fbfcd09e-8a2b-4599-ae86-e1389f7bf49a) Computing track assignment... ...done. Success. Distance: 0.77 Requesting MusicBrainz release 0824695f-456e-40fe-8c8e-4295c91ff197 Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_album_extract Sending event: albuminfo_received Candidate: The New Breed - Off the Beaten Path (0824695f-456e-40fe-8c8e-4295c91ff197) Computing track assignment... ...done. Success. Distance: 0.71 Requesting MusicBrainz release 7d66b04e-386c-4ff4-9659-785c9c29873a Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_album_extract Sending event: albuminfo_received Candidate: Justin Moore - Off the Beaten Path (7d66b04e-386c-4ff4-9659-785c9c29873a) Computing track assignment... ...done. Success. Distance: 0.73 Requesting MusicBrainz release 126ab765-595a-409b-8b86-94a45b5d5087 Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_album_extract Sending event: albuminfo_received Candidate: Phonique - Beat'n Greet (126ab765-595a-409b-8b86-94a45b5d5087) Computing track assignment... ...done. Success. Distance: 0.86 Requesting MusicBrainz release 4601d64b-bd38-47a7-bddf-04e1cbba10de Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_track_extract Sending event: mb_album_extract Sending event: albuminfo_received Candidate: Justin Moore - Off the Beaten Path (4601d64b-bd38-47a7-bddf-04e1cbba10de) Computing track assignment... ...done. Success. Distance: 0.73 deezer: Searching Deezer for 'album:"Off The Beat'n Path" artist:"Derlee"' deezer: Found 1 result(s) from Deezer for 'album:"Off The Beat'n Path" artist:"Derlee"' deezer: Searching Deezer for album '11684198' Sending event: albuminfo_received Candidate: Derlee - Off The Beat'n Path (11684198) Computing track assignment... ...done. Success. Distance: 0.33 bandcamp: Searching as for Off The Beat'n Path using {'query': "Off The Beat'n Path", 'artist': 'Derlee', 'label': 'Philos Records', 'search_type': 'a'} ```

I tried to look around a tiny bit myself but did not get far with any coherent reasoning unfortunately.

shagr4th commented 1 year ago

Same error here. It's actually due to an artist's use of another domain name : For instance if you search for "Colors in the dark", the first bandcamp result will point to https://3six.net/album/colours-in-the-dark, which is not on bandcamp.com, same as your album link (but use the same backend). Then the regex defined in search.py is not matched, and the url empty, leading to this error...

@snejus imho, I would just change this regex to a simpler one with just 'http' and some query like '?from=search', given that all search results are likely to be bandcamp albums...

snejus commented 1 year ago

Okay, seeing this is re-appearing I am now testing the search against all album names I have in my beets database (1500+).

I can see that the first 300 searches gave me 80 results with a wrong URL (missing or some irrelevant HTML), each of which would cause the issue you above. This gives more than enough data to work with.

@shagr4th thank you for your input - ?from=search is a great idea - see my comments under the PR. Aiming to get the fix out tonight.

tomauty commented 1 year ago

@snejus Sorry to be off track, but what's the table command you're piping to to show the response data?

snejus commented 1 year ago

@tomauty, it's a JSON prettifier based on rich. Since it's now in a somewhat of a more shareable state, see rich-tables. Early version is on pypi, so pip install rich-tables should give you the executable. There aren't any docs or usage so far, but see the tests/json folder in the repo and README to see what they give. Then pipe in some data and see what you get 😉

snejus commented 1 year ago

This should now be fully fixed in 0.16.2!