beetbox / beets

music library manager and MusicBrainz tagger
http://beets.io/
MIT License
12.57k stars 1.8k forks source link

Spotify: Improve song search/matching #5189

Open mgoltzsche opened 2 months ago

mgoltzsche commented 2 months ago

The Spotify plugin doesn't match/find most songs.

Problem

Running this command in verbose (-vv) mode:

$ beet -vv spotify

Led to this problem:

user configuration: /data/beets/config.yaml
data directory: /data/beets
plugin paths: 
inline: adding item field first_artist
lastgenre: Loading canonicalization tree /etc/beets/genre-tree.yaml
lastgenre: Loading canonicalization tree /etc/beets/genre-tree.yaml
Sending event: pluginload
library database: /data/beets/library.db
library directory: /data/music
Sending event: library_opened
Parsed query: AndQuery([TrueQuery()])
Parsed sort: NullSort()
spotify: Processing 27 tracks...
spotify: Searching Spotify for 'Senegal Fast Food artist:Amadou & Mariam album:'
spotify: Found 0 result(s) from Spotify for 'Senegal Fast Food artist:Amadou & Mariam album:'
spotify: Searching Spotify for 'La Realite artist:Amadou & Mariam album:'
spotify: Found 0 result(s) from Spotify for 'La Realite artist:Amadou & Mariam album:'
spotify: Searching Spotify for 'Je pense a toi artist:Amadou & Mariam album:Sou Ni Tile'
spotify: Found 0 result(s) from Spotify for 'Je pense a toi artist:Amadou & Mariam album:Sou Ni Tile'
spotify: Searching Spotify for 'Deep Jungle Walk artist:Astrix album:'
spotify: Found 0 result(s) from Spotify for 'Deep Jungle Walk artist:Astrix album:'
spotify: Searching Spotify for 'Ephemeral artist:Bassnectar album:'
spotify: Found 0 result(s) from Spotify for 'Ephemeral artist:Bassnectar album:'
spotify: Searching Spotify for 'Street Blues artist:Bellaire album:'
spotify: Found 0 result(s) from Spotify for 'Street Blues artist:Bellaire album:'
spotify: Searching Spotify for 'Redemption artist:Boris Brejcha album:'
spotify: Found 0 result(s) from Spotify for 'Redemption artist:Boris Brejcha album:'
spotify: Searching Spotify for 'Vaunce of the Flowers (Waltz of the Flowers) artist:Tchaikovsky album:'
spotify: Found 0 result(s) from Spotify for 'Vaunce of the Flowers (Waltz of the Flowers) artist:Tchaikovsky album:'
spotify: Searching Spotify for 'Walkin' artist:Miles Davis album:Autumn Leaves'
spotify: Found 0 result(s) from Spotify for 'Walkin' artist:Miles Davis album:Autumn Leaves'
spotify: Searching Spotify for 'Autumn Leaves artist:Miles Davis album:Autumn Leaves'
spotify: Found 0 result(s) from Spotify for 'Autumn Leaves artist:Miles Davis album:Autumn Leaves'
spotify: Searching Spotify for 'So What artist:Miles Davis album:Autumn Leaves'
spotify: Found 0 result(s) from Spotify for 'So What artist:Miles Davis album:Autumn Leaves'
spotify: Searching Spotify for ''Round Midnight artist:Miles Davis album:Autumn Leaves'
spotify: Found 0 result(s) from Spotify for ''Round Midnight artist:Miles Davis album:Autumn Leaves'
spotify: Searching Spotify for 'All of You artist:Miles Davis album:Autumn Leaves'
spotify: Found 0 result(s) from Spotify for 'All of You artist:Miles Davis album:Autumn Leaves'
spotify: Searching Spotify for 'Ready or Not (Champion Bootleg) artist:Fugees album:'
spotify: Found 0 result(s) from Spotify for 'Ready or Not (Champion Bootleg) artist:Fugees album:'
spotify: Searching Spotify for 'Amanece artist:EttoreTechnoChannel album:'
spotify: Found 0 result(s) from Spotify for 'Amanece artist:EttoreTechnoChannel album:'
spotify: Searching Spotify for 'Free Tibet (Vini Vici remix) artist:Hilight Tribe album:'
spotify: Found 0 result(s) from Spotify for 'Free Tibet (Vini Vici remix) artist:Hilight Tribe album:'
spotify: Searching Spotify for 'Never Mind artist:Infected Mushroom album:Army of Mushrooms'
spotify: Found 2 result(s) from Spotify for 'Never Mind artist:Infected Mushroom album:Army of Mushrooms'
spotify: Most popular track chosen, count: 2
spotify: Searching Spotify for 'Franks artist:Infected Mushroom album:Legend of the Black Shawarma'
spotify: Found 1 result(s) from Spotify for 'Franks artist:Infected Mushroom album:Legend of the Black Shawarma'
spotify: Spotify track(s) found, count: 1
spotify: Searching Spotify for 'The Rope (Le Youth remix) feat. Polica artist:Lane 8 album:The Rope'
spotify: Found 1 result(s) from Spotify for 'The Rope (Le Youth remix) feat. Polica artist:Lane 8 album:The Rope'
spotify: Spotify track(s) found, count: 1
spotify: Searching Spotify for 'Mishaps Happening artist:Quantic album:'
spotify: Found 0 result(s) from Spotify for 'Mishaps Happening artist:Quantic album:'
spotify: Searching Spotify for 'The Ghost of Tom Joad artist:Rage Against The Machine album:'
spotify: Found 0 result(s) from Spotify for 'The Ghost of Tom Joad artist:Rage Against The Machine album:'
spotify: Searching Spotify for 'Wake Up (Rasticles drum n bass remix) artist:Rage Against the Machine album:'
spotify: Found 0 result(s) from Spotify for 'Wake Up (Rasticles drum n bass remix) artist:Rage Against the Machine album:'
spotify: Searching Spotify for 'The Alcoholic artist:Royksopp album:'
spotify: Found 0 result(s) from Spotify for 'The Alcoholic artist:Royksopp album:'
spotify: Searching Spotify for 'Back To Jazz artist:sometimes jah album:Just Feel'
spotify: Found 1 result(s) from Spotify for 'Back To Jazz artist:sometimes jah album:Just Feel'
spotify: Spotify track(s) found, count: 1
spotify: Searching Spotify for 'Blues artist:STAND HIGH PATROL album:'
spotify: Found 0 result(s) from Spotify for 'Blues artist:STAND HIGH PATROL album:'
spotify: Searching Spotify for 'STAND HIGH PATROL: Boat People artist:STAND HIGH PATROL album:'
spotify: Found 0 result(s) from Spotify for 'STAND HIGH PATROL: Boat People artist:STAND HIGH PATROL album:'
spotify: Searching Spotify for 'Lightweight artist:Total Science album:'
spotify: Found 0 result(s) from Spotify for 'Lightweight artist:Total Science album:'
spotify: 23 track(s) did not match a Spotify ID:
spotify: track: Senegal Fast Food artist:Amadou & Mariam album:
spotify: track: La Realite artist:Amadou & Mariam album:
spotify: track: Je pense a toi artist:Amadou & Mariam album:Sou Ni Tile
spotify: track: Deep Jungle Walk artist:Astrix album:
spotify: track: Ephemeral artist:Bassnectar album:
spotify: track: Street Blues artist:Bellaire album:
spotify: track: Redemption artist:Boris Brejcha album:
spotify: track: Vaunce of the Flowers (Waltz of the Flowers) artist:Tchaikovsky album:
spotify: track: Walkin' artist:Miles Davis album:Autumn Leaves
spotify: track: Autumn Leaves artist:Miles Davis album:Autumn Leaves
spotify: track: So What artist:Miles Davis album:Autumn Leaves
spotify: track: 'Round Midnight artist:Miles Davis album:Autumn Leaves
spotify: track: All of You artist:Miles Davis album:Autumn Leaves
spotify: track: Ready or Not (Champion Bootleg) artist:Fugees album:
spotify: track: Amanece artist:EttoreTechnoChannel album:
spotify: track: Free Tibet (Vini Vici remix) artist:Hilight Tribe album:
spotify: track: Mishaps Happening artist:Quantic album:
spotify: track: The Ghost of Tom Joad artist:Rage Against The Machine album:
spotify: track: Wake Up (Rasticles drum n bass remix) artist:Rage Against the Machine album:
spotify: track: The Alcoholic artist:Royksopp album:
spotify: track: Blues artist:STAND HIGH PATROL album:
spotify: track: STAND HIGH PATROL: Boat People artist:STAND HIGH PATROL album:
spotify: track: Lightweight artist:Total Science album:
spotify: 
https://open.spotify.com/track/5qVNRfUrsJToWrIsdxytec
https://open.spotify.com/track/6zTbuAt90GFVTn3F651K5u
https://open.spotify.com/track/7toata3kAxgvAPer5XCwHE
https://open.spotify.com/track/0lYGEP22F5i29vvFZ8yuMW
Sending event: cli_exit

-> Only 4 of 27 songs matched (14%).

However, there should be a lot more matches, if not all. For instance Miles Davis' "Autumn Leaves" is definitely known by Spotify.

Something seems to be wrong with the way the Spotify search query is constructed. I briefly tried to run some of the generated queries against the Spotify API here and they returned proper results. Not sure why the Beets plugin isn't finding the songs. I suspect some difference in the way the Spotify query is constructed but I didn't spot it yet.

... On the other hand I did not configure Spotify authentication myself but when using the Spotify API on the documentation page, I was logged with my personal account. Is it possible that certain songs can be found only when logged in using a personal account?

Setup

My configuration (output of beet config) is:

beet config # beet config ```yaml importfeeds: relative_to: !!binary | L2RhdGEvaW1wb3J0ZmVlZA== formats: m3u dir: /data/importfeed m3u_name: imported.m3u absolute_path: no directory: /data/music # --------------- Main --------------- library: /data/beets/library.db paths: default: Albums/%title{$albumartist}/$album%aunique{}/$track $title singleton: Singles/%title{$first_artist}/$title comp: Compilations/$album%aunique{}/$track $title # --------------- Plugins --------------- plugins: - describe - web - webm3u - webrouter - beetstream - types - smartplaylist - importfeeds - edit - play - mbsync - spotify - fromfilename - lastgenre - autogenre - ftintitle - inline - rewrite - ihate - chroma - xtractor - mpdstats - convert - bpm - duplicates - bareasc - fuzzy - info - ytimport - badfiles - unimported - random webrouter: host: 0.0.0.0 port: 8337 routes: /: plugin: web /favicon.ico: plugin: webrouter.favicon /subsonic: plugin: beetstream config: never_transcode: yes playlist_dir: /data/playlists /m3u: plugin: webm3u cors: '' cors_supports_credentials: no reverse_proxy: no include_paths: no readonly: yes beetstream: host: 0.0.0.0 port: 8080 never_transcode: yes cors: '*' cors_supports_credentials: yes reverse_proxy: no include_paths: no playlist_dir: '' web: host: 0.0.0.0 port: 8337 reverse_proxy: yes readonly: yes cors: '' cors_supports_credentials: no include_paths: no webm3u: host: 0.0.0.0 port: 8339 cors: '' cors_supports_credentials: no reverse_proxy: no include_paths: no playlist_dir: uri_format: import: write: yes copy: yes move: no link: no hardlink: no incremental: yes quiet: no quiet_fallback: skip log: /data/beets/import.log ytimport: directory: /data/ytimport import: yes reimport: no format: bestaudio/best url_file: '' likes: no max_likes: 15 auth_headers: set: {} min_length: 60 max_length: 7200 max_length_nochapter: 900 split_tracks: yes group_albums: yes spotify: source_weight: 0.7 mode: list tiebreak: popularity show_failures: yes artist_field: albumartist album_field: album track_field: title region_filter: regex: [] client_id: 4e414367a1d14c75a5c5129a627fcab8 client_secret: REDACTED tokenfile: spotify_token.json xtractor: auto: no dry-run: no write: yes threads: 4 force: no quiet: no keep_output: no keep_profiles: no output_path: /tmp/xtractor essentia_extractor: /usr/local/bin/essentia_streaming_extractor_music high_level_targets: genre_rosamerica_probability: path: highlevel.genre_rosamerica.probability type: float genre_electronic: path: highlevel.genre_electronic.value type: string genre_electronic_probability: path: highlevel.genre_electronic.probability type: float timbre: path: highlevel.timbre.value type: string tonal_atonal: path: highlevel.tonal_atonal.value type: string key_edma: path: tonal.key_edma.key type: string key_edma_scale: path: tonal.key_edma.scale type: string danceable: path: highlevel.danceability.all.danceable type: float required: yes gender: path: highlevel.gender.value type: string required: yes is_male: path: highlevel.gender.all.male type: float is_female: path: highlevel.gender.all.female type: float genre_rosamerica: path: highlevel.genre_rosamerica.value type: string required: yes voice_instrumental: path: highlevel.voice_instrumental.value type: string required: yes is_voice: path: highlevel.voice_instrumental.all.voice type: float is_instrumental: path: highlevel.voice_instrumental.all.instrumental type: float mood_acoustic: path: highlevel.mood_acoustic.all.acoustic type: float required: yes mood_aggressive: path: highlevel.mood_aggressive.all.aggressive type: float required: yes mood_electronic: path: highlevel.mood_electronic.all.electronic type: float required: yes mood_happy: path: highlevel.mood_happy.all.happy type: float required: yes mood_sad: path: highlevel.mood_sad.all.sad type: float required: yes mood_party: path: highlevel.mood_party.all.party type: float required: yes mood_relaxed: path: highlevel.mood_relaxed.all.relaxed type: float required: yes mood_mirex: path: highlevel.moods_mirex.value type: string required: yes mood_mirex_cluster_1: path: highlevel.moods_mirex.all.Cluster1 type: float mood_mirex_cluster_2: path: highlevel.moods_mirex.all.Cluster2 type: float mood_mirex_cluster_3: path: highlevel.moods_mirex.all.Cluster3 type: float mood_mirex_cluster_4: path: highlevel.moods_mirex.all.Cluster4 type: float mood_mirex_cluster_5: path: highlevel.moods_mirex.all.Cluster5 type: float extractor_profile: highlevel: svm_models: - /var/lib/essentia/svm-models/beta5/danceability.history - /var/lib/essentia/svm-models/beta5/gender.history - /var/lib/essentia/svm-models/beta5/genre_rosamerica.history - /var/lib/essentia/svm-models/beta5/genre_electronic.history - /var/lib/essentia/svm-models/beta5/mood_acoustic.history - /var/lib/essentia/svm-models/beta5/mood_aggressive.history - /var/lib/essentia/svm-models/beta5/mood_electronic.history - /var/lib/essentia/svm-models/beta5/mood_happy.history - /var/lib/essentia/svm-models/beta5/mood_sad.history - /var/lib/essentia/svm-models/beta5/mood_party.history - /var/lib/essentia/svm-models/beta5/mood_relaxed.history - /var/lib/essentia/svm-models/beta5/moods_mirex.history - /var/lib/essentia/svm-models/beta5/voice_instrumental.history - /var/lib/essentia/svm-models/beta5/tonal_atonal.history - /var/lib/essentia/svm-models/beta5/timbre.history compute: 1 outputFormat: json outputFrames: 0 lowlevel: frameSize: 2048 hopSize: 1024 chromaprint: compute: 0 low_level_targets: bpm: path: rhythm.bpm type: integer required: yes danceability: path: rhythm.danceability type: float beats_count: path: rhythm.beats_count type: integer average_loudness: path: lowlevel.average_loudness type: float required: yes ftintitle: auto: yes drop: no format: feat. {0} item_fields: first_artist: "import re\nm = re.match('^(.+?)\\\\s+(x\\\\s|\\\\(?(feat(\\\\.?|uring)|(ft|vs)\\\\.))', artist, flags=re.IGNORECASE)\nif m:\n return m[1]\nreturn artist\n" bareasc: prefix: '#' fuzzy: threshold: 0.7 prefix: '~' chroma: auto: yes autogenre: pretend: no all: no force: no lastgenre: yes xtractor: yes remix_title: yes parent_genres: yes from_title: yes genre_rosamerica_strong: 0.8 genre_electronic_strong: 0.8 genre_electronic_prepend: 0.5 genre_electronic_append: 0.45 lastgenre: auto: no prefer_specific: yes count: 4 source: album min_weight: 15 canonical: /etc/beets/genre-tree.yaml whitelist: /data/beets/genres.txt fallback: force: yes separator: ', ' title_case: yes ihate: warn: ['artist:rnb'] skip: - genre:Comedy - artist::^Adele$ - artist::^AYLIVA$ - artist::^Sia$ convert: never_convert_lossy_files: yes embed: no hardlink: no dest: pretend: no link: no threads: 16 format: mp3 id3v23: inherit formats: aac: command: ffmpeg -i $source -y -vn -acodec aac -aq 1 $dest extension: m4a alac: command: ffmpeg -i $source -y -vn -acodec alac $dest extension: m4a flac: ffmpeg -i $source -y -vn -acodec flac $dest mp3: ffmpeg -i $source -y -vn -aq 2 $dest opus: ffmpeg -i $source -y -vn -acodec libopus -ab 96k $dest ogg: ffmpeg -i $source -y -vn -acodec libvorbis -aq 3 $dest wma: ffmpeg -i $source -y -vn -acodec wmav2 -vn $dest max_bitrate: auto: no auto_keep: no tmpdir: quiet: no paths: {} no_convert: '' copy_album_art: no album_art_maxwidth: 0 delete_originals: no playlist: duplicates: delete: no album: no checksum: '' copy: '' count: no format: '' full: no keys: [] merge: no move: '' path: no tiebreak: {} strict: no tag: '' bpm: max_strokes: 3 overwrite: yes play: command: play-playlist use_folders: no relative_to: raw: no warning_threshold: 100 bom: no smartplaylist: auto: no output: extm3u fields: [id] playlist_dir: /data/playlists relative_to: /data/playlists playlists: - name: all.m3u query: '' - name: all-by-album-genre.m3u query: albumartist+ year+ album+ disc+ track+ genre+ artist+ title+ - name: jazz-blues.m3u query: - genre::^Jazz - genre::^Blues - genre:'Jazz Fusion' - genre:'Swing Jazz' - name: reggae.m3u query: ['genres:Reggae', 'genres::''^Dub$''', 'genres:Ska'] - name: dub.m3u query: ['genre::''^Dub$'''] - name: ambient.m3u query: ['genre:Ambient'] - name: chillout.m3u query: - genre:Ambient - genre:Downtempo - genre:'Nu Jazz' - genre:'Trip Hop' - genre:Electronica - genre::'^Electro$' - genre:'Easy Listening' - name: idm.m3u query: ['genre:Idm', 'genre:Breakcore'] - name: downtempo.m3u query: ['genre:Downtempo', 'genre:''Nu Jazz'''] - name: nu-jazz.m3u query: ['genre:''Nu Jazz'''] - name: world-music.m3u query: - genre:Blues - genres:African - genres:Afrobeat - genres:'Rhythm And Blues' - genres::'(^|, )Funk(,|$)' - genres::'(^|, )Soul(,|$)' - genres:'Caribbean And Latin American' - genre:Brazilian - genre:Balkan - genre:Klezmer - genres:'East European' - name: african.m3u query: ['genres:African', 'genres:Afrobeat'] - name: afrobeat.m3u query: ['genres:Afrobeat'] - name: balkan-klezmer.m3u query: ['genre:''East European''', 'genre:Balkan', 'genre:Klezmer'] - name: classical.m3u query: ['genre:Classical'] - name: dnb.m3u query: - genres:'Drum And Bass' - genre:Breakbeat - genre:Jungle - genre:Neurofunk - genre:'Liquid Drum And Bass' - genre:'Ragga Drum And Bass' - name: neurofunk.m3u query: ['genre:Neurofunk'] - name: jungle.m3u query: ['genre:Jungle'] - name: uk-garage.m3u query: ['genres:''Uk Garage''', 'genres:Dubstep', 'genre:Grime'] - name: dubstep-grime.m3u query: ['genre:''Dark Dubstep''', 'genre:Dubstep bpm:0..90', 'genre:Grime'] - name: electronic.m3u query: ['genres:Electronic'] - name: house-techno.m3u query: - genre:House - genre:Techno - genre:Minimal - genre:Progressive - name: deep-house.m3u query: ['genre:''Deep House''', 'genre:''Nu Jazz'' genres:House'] - name: house.m3u query: ['genre:House', 'genre:''Nu Jazz'' genres:House'] - name: tech-house-minimal.m3u query: ['genre:''Tech House''', 'genre:Minimal'] - name: techno.m3u query: ['genre:Techno', 'genre:Minimal'] - name: minimal.m3u query: ['genre:Minimal'] - name: trance.m3u query: ['genre:Trance'] - name: goa.m3u query: ['genre:''Goa Trance'''] - name: ska.m3u query: ['genre:Ska'] - name: rock.m3u query: - genre::'^Rock' - genre::'^New Wave' - genre::'^Folk' - genre:'Fun Metal' - name: punk-rock.m3u query: ['genre:Punk', 'genre:''Fun Metal'''] - name: pop.m3u query: - genre:Pop - genre:'New Wave' - genre:'Balkan Beats' - genre::'^Electro$' - genres::'^Electronic Rock, (?!Drum And Bass)' - name: hiphop.m3u query: ['genre:''Hip Hop'''] - name: liked/1 or more.m3u query: play_count:1.. - name: liked/5 or more.m3u query: play_count:5.. - name: liked/loved.m3u query: loved:1 - name: latest/last day.m3u query: added- added:-1d.. - name: latest/last 3 days.m3u query: added- added:-3d.. - name: latest/last 7 days.m3u query: added- added:-7d.. - name: latest/last 14 days.m3u query: added- added:-14d.. - name: latest/last 30 days.m3u query: added- added:-30d.. - name: latest/last 60 days.m3u query: added- added:-60d.. uri_format: forward_slash: no prefix: REDACTED urlencode: no pretend_paths: no unimported: ignore_extensions: [] ignore_subdirectories: [] rewrite: {} pathfields: {} album_fields: {} mpd: music_directory: /data/music strip_path: '' rating: yes rating_mix: 0.75 host: localhost port: 6600 password: REDACTED edit: albumfields: album albumartist itemfields: track title artist album ignore_fields: id path describe: auto: no ```
arsaboo commented 2 months ago

I have had similar experiences. As discussed in #4997, we should move to the Spotipy library, which does a great job of interfacing with Spotify.