Add multithread option - Githubissues

ndragon798 commented 7 years ago

[X] Using latest version as provided on the master branch
[X] Searched for similar issues including closed ones

What is the purpose of your issue?

[ ] Script won't run
[ ] Encountered bug
[X] Feature request
[ ] Question
[ ] Other

System information

Your python version: python 3.5
Your operating system: Ubuntu 17.04

Description

Add a multithreaded option for list downloads to allow for faster downloading.

ritiek commented 7 years ago

This would be a nice thing to have. I am busy with school exams at the moment but if someone wants to give this a try, I'm up for PRs.

linusg commented 7 years ago

I will be busy with the exams startin in late November or so meaning I have some time currently. Will look into it, I think it is not as hard to implement.

linusg commented 7 years ago

Ok, not as easy as I thought. I simply tried starting multiple threads of grab_single in grab_list, which should in theory start downloading all songs at the same time and later on wait until the last is complete by calling Thread.join. My song list is about 20 entries. In practice I get a bunch of:

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/home/linus/.virtualenvs/spotdl/lib/python3.5/site-packages/urllib3/connectionpool.py", line 228, in _get_conn
    conn = self.pool.get(block=self.block, timeout=timeout)
AttributeError: 'NoneType' object has no attribute 'get'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/linus/.virtualenvs/spotdl/lib/python3.5/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/home/linus/.virtualenvs/spotdl/lib/python3.5/site-packages/urllib3/connectionpool.py", line 588, in urlopen
    conn = self._get_conn(timeout=pool_timeout)
  File "/home/linus/.virtualenvs/spotdl/lib/python3.5/site-packages/urllib3/connectionpool.py", line 231, in _get_conn
    raise ClosedPoolError(self, "Pool is closed.")
urllib3.exceptions.ClosedPoolError: HTTPSConnectionPool(host='api.spotify.com', port=443): Pool is closed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "spotdl.py", line 364, in grab_single
    meta_tags = generate_metadata(raw_song)
  File "spotdl.py", line 37, in generate_metadata
    album = spotify.album(meta_tags['album']['id'])
  File "/home/linus/.virtualenvs/spotdl/lib/python3.5/site-packages/spotipy/client.py", line 303, in album
    return self._get('albums/' + trid)
  File "/home/linus/.virtualenvs/spotdl/lib/python3.5/site-packages/spotipy/client.py", line 146, in _get
    return self._internal_call('GET', url, payload, kwargs)
  File "/home/linus/.virtualenvs/spotdl/lib/python3.5/site-packages/spotipy/client.py", line 108, in _internal_call
    r = self._session.request(method, url, headers=headers, proxies=self.proxies, **args)
  File "/home/linus/.virtualenvs/spotdl/lib/python3.5/site-packages/requests/sessions.py", line 502, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/linus/.virtualenvs/spotdl/lib/python3.5/site-packages/requests/sessions.py", line 612, in send
    r = adapter.send(request, **kwargs)
  File "/home/linus/.virtualenvs/spotdl/lib/python3.5/site-packages/requests/adapters.py", line 507, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.spotify.com', port=443): Pool is closed.

and afterwards:

<Thread(Thread-2, started 140505446409984)>
2. MySongHere
Exception in thread Thread-16:ceived. Rate: [4994 KB/s].  ETA: [0 secs]    
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "spotdl.py", line 372, in grab_single
    if download_song(file_name, content):
  File "spotdl.py", line 232, in download_song
    filepath='{0}{1}'.format(os.path.join(args.folder, file_name), args.input_ext))
  File "/home/linus/.virtualenvs/spotdl/lib/python3.5/site-packages/pafy/backend_shared.py", line 653, in download
    os.rename(temp_filepath, filepath)
FileNotFoundError: [Errno 2] No such file or directory: 'Music/MySongHere.m4a.temp' -> 'Music/MySongHere.m4a'

I believe this is due to API call limitations, and indeed setting a time.sleep(5) between each thread start reduces the occurrence of errors, but not eliminates them. Additional the output is unordered (as expected), because downloading and converting different songs takes a different amount of time.

So, currently for downloading the soundtrack of the Nerve movie (which I use apparently whenever I need to test spotdl :grin: ) produces the following output (printing the Thread object after its start and "Thread ended" after its end):

(spotdl) linus@sony-vaio ~/Code/python/projects/spotify-downloader $ time python3 spotdl.py -l nerve-soundtrack.txt
Total songs in list: 16 songs

<Thread(Thread-1, started 140057420797696)>
1. Roy Orbison - You Got It

<Thread(Thread-2, started 140057340016384)>
  3,421,193 Bytes [100.00%] received. Rate: [4965 KB/s].  ETA: [0 secs]    
Converting Roy_Orbison_-_You_Got_It.m4a to mp3

<Thread(Thread-3, started 140057331623680)>
2. BØRNS - Electric Love
  3,507,910 Bytes [100.00%] received. Rate: [1747 KB/s].  ETA: [0 secs]    
Converting BØRNS_-_Electric_Love.m4a to mp3
Fixing meta-tags
3. Into The Night-Benny Mardones

<Thread(Thread-4, started 140057420797696)>
  4,234,613 Bytes [100.00%] received. Rate: [5052 KB/s].  ETA: [0 secs]    
Converting Benny_Mardones_-_Into_The_Night.m4a to mp3

<Thread(Thread-5, started 140057319937792)>
4. Lowell - Ride (feat. Icona Pop) [Nerve 2016 Movie Soundtrack]
  3,112,873 Bytes [100.00%] received. Rate: [3975 KB/s].  ETA: [0 secs]    
Converting Lowell_-_Ride_(feat_Icona_Pop).m4a to mp3
Fixing meta-tags

<Thread(Thread-6, started 140057340016384)>
5. Diplomats of Solid Sound - Gimme one more chance
  3,329,489 Bytes [100.00%] received. Rate: [3739 KB/s].  ETA: [0 secs]    
Converting Diplomats_of_Solid_Sound_-_Gimme_One_More_Chance.m4a to mp3

<Thread(Thread-7, started 140057311020800)>
Fixing meta-tags
6. Halsey - Hurricane (Arty Remix/Audio)
Fixing meta-tags
  3,584,050 Bytes [100.00%] received. Rate: [2979 KB/s].  ETA: [0 secs]    
Converting Halsey_-_Hurricane_-_Arty_Remix.m4a to mp3
7. Pyramid Vritra - Black Belts ft. Pyramid Quince

<Thread(Thread-8, started 140057420797696)>
  3,030,140 Bytes [100.00%] received. Rate: [4826 KB/s].  ETA: [0 secs]    
Converting Pyramid_Vritra_-_Black_Belts.m4a to mp3
Fixing meta-tags

<Thread(Thread-9, started 140057319937792)>
8. Darke Complex - "Invertebrate"
  5,180,715 Bytes [100.00%] received. Rate: [4684 KB/s].  ETA: [0 secs]    
Converting Darke_Complex_-_Invertebrate.m4a to mp3
Fixing meta-tags
Fixing meta-tags

<Thread(Thread-10, started 140057311020800)>
9. Bro Safari - Bender
  4,415,473 Bytes [100.00%] received. Rate: [4897 KB/s].  ETA: [0 secs]    
Converting Bro_Safari_-_Bender.m4a to mp3
10. Tei Shi - "Bassically" (Official Audio)

<Thread(Thread-11, started 140057340016384)>
  4,625,335 Bytes [100.00%] received. Rate: [5078 KB/s].  ETA: [0 secs]    
Converting Tei_Shi_-_Bassically.m4a to mp3

<Thread(Thread-12, started 140057331623680)>
11. Holy Ghost! "Okay" (Official Music Video) - DFA RECORDS
Fixing meta-tags1.47%] received. Rate: [16820 KB/s].  ETA: [0 secs]    
  4,459,270 Bytes [100.00%] received. Rate: [4846 KB/s].  ETA: [0 secs]    
Converting Holy_Ghost_-_Okay.m4a to mp3

<Thread(Thread-13, started 140057420797696)>

<Thread(Thread-14, started 140057296336640)>
12. Alex Winston - Down Low [Official Audio]
Fixing meta-tags
13. Grandtheft - Give Me More [Official Full Stream]
Exception in thread Thread-12:ceived. Rate: [4591 KB/s].  ETA: [0 secs]    
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "spotdl.py", line 372, in grab_single
    if download_song(file_name, content):
  File "spotdl.py", line 232, in download_song
    filepath='{0}{1}'.format(os.path.join(args.folder, file_name), args.input_ext))
  File "/home/linus/.virtualenvs/spotdl/lib/python3.5/site-packages/pafy/backend_shared.py", line 653, in download
    os.rename(temp_filepath, filepath)
FileNotFoundError: [Errno 2] No such file or directory: 'Music/Alex_Winston_-_Down_Low.m4a.temp' -> 'Music/Alex_Winston_-_Down_Low.m4a'

Fixing meta-tagss [38.34%] received. Rate: [1941 KB/s].  ETA: [1 secs]    
  4,230,932 Bytes [100.00%] received. Rate: [2622 KB/s].  ETA: [0 secs]    
Converting Grandtheft_-_Give_Me_More.m4a to mp3

<Thread(Thread-15, started 140057311020800)>
14. Holy ghost!-Crime cutz-(baile)
Exception in thread Thread-14:
Traceback (most recent call last):
  File "/home/linus/.virtualenvs/spotdl/lib/python3.5/site-packages/urllib3/connectionpool.py", line 228, in _get_conn
    conn = self.pool.get(block=self.block, timeout=timeout)
AttributeError: 'NoneType' object has no attribute 'get'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/linus/.virtualenvs/spotdl/lib/python3.5/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/home/linus/.virtualenvs/spotdl/lib/python3.5/site-packages/urllib3/connectionpool.py", line 588, in urlopen
    conn = self._get_conn(timeout=pool_timeout)
  File "/home/linus/.virtualenvs/spotdl/lib/python3.5/site-packages/urllib3/connectionpool.py", line 231, in _get_conn
    raise ClosedPoolError(self, "Pool is closed.")
urllib3.exceptions.ClosedPoolError: HTTPSConnectionPool(host='api.spotify.com', port=443): Pool is closed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "spotdl.py", line 364, in grab_single
    meta_tags = generate_metadata(raw_song)
  File "spotdl.py", line 29, in generate_metadata
    meta_tags = spotify.track(raw_song)
  File "/home/linus/.virtualenvs/spotdl/lib/python3.5/site-packages/spotipy/client.py", line 223, in track
    return self._get('tracks/' + trid)
  File "/home/linus/.virtualenvs/spotdl/lib/python3.5/site-packages/spotipy/client.py", line 146, in _get
    return self._internal_call('GET', url, payload, kwargs)
  File "/home/linus/.virtualenvs/spotdl/lib/python3.5/site-packages/spotipy/client.py", line 108, in _internal_call
    r = self._session.request(method, url, headers=headers, proxies=self.proxies, **args)
  File "/home/linus/.virtualenvs/spotdl/lib/python3.5/site-packages/requests/sessions.py", line 502, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/linus/.virtualenvs/spotdl/lib/python3.5/site-packages/requests/sessions.py", line 612, in send
    r = adapter.send(request, **kwargs)
  File "/home/linus/.virtualenvs/spotdl/lib/python3.5/site-packages/requests/adapters.py", line 507, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.spotify.com', port=443): Pool is closed.

Fixing meta-tags
15. Blood Orange - Forget It (Album Version)

<Thread(Thread-16, started 140057340016384)>
  3,765,621 Bytes [100.00%] received. Rate: [4816 KB/s].  ETA: [0 secs]    
Converting Blood_Orange_-_Forget_It.m4a to mp3
Fixing meta-tags

Thread ended
Thread ended
Thread ended
Thread ended
Thread ended
Thread ended
Thread ended
Thread ended
Thread ended
Thread ended
Thread ended
Thread ended
Thread ended
Thread ended
16. Jungle - Lucky I Got What I Want [HD]
  4,211,032 Bytes [100.00%] received. Rate: [4736 KB/s].  ETA: [0 secs]    
Converting Jungle_-_Lucky_I_Got_What_I_Want.m4a to mp3
Fixing meta-tags
Thread ended
Fixing meta-tags
Thread ended

real    1m32.364s
user    3m11.552s
sys 0m2.536s

Time: 1m32s (including the 5 second sleep every time). Only 14 songs of 23 are present in the Music/ folder afterwards.

Current script without modification:

(spotdl) linus@sony-vaio ~/spotify-downloader $ time python3 spotdl.py -l nerve-soundtrack.txt
Total songs in list: 23 songs

1. Basenji - Can't Get Enough
  3,375,494 Bytes [100.00%] received. Rate: [4973 KB/s].  ETA: [0 secs]    
Converting Basenji_-_Cant_Get_Enough.m4a to mp3
Fixing meta-tags

2. MØ - Kamikaze (Official Video)
  3,493,053 Bytes [100.00%] received. Rate: [4872 KB/s].  ETA: [0 secs]    
Converting MØ_-_Kamikaze.m4a to mp3
Fixing meta-tags

3. Jess Kent - Get Down (Official Video)
  2,904,472 Bytes [100.00%] received. Rate: [4849 KB/s].  ETA: [0 secs]    
Converting Jess_Kent_-_Get_Down.m4a to mp3
Fixing meta-tags

4. Sweetmates - I Don't Dream
  3,336,801 Bytes [100.00%] received. Rate: [4703 KB/s].  ETA: [0 secs]    
Converting Sweetmates_-_I_Dont_Dream.m4a to mp3
Fixing meta-tags

5. Soap - Melanie Martinez (Lyrics)
  3,316,785 Bytes [100.00%] received. Rate: [4641 KB/s].  ETA: [0 secs]    
Converting Melanie_Martinez_-_Soap.m4a to mp3
Fixing meta-tags

6. Crystal Stilts - Darken The Door
  3,684,915 Bytes [100.00%] received. Rate: [4916 KB/s].  ETA: [0 secs]    
Converting Crystal_Stilts_-_Darken_the_Door.m4a to mp3
Fixing meta-tags

7. The Skyliners - Since I Don't Have You
  2,555,998 Bytes [100.00%] received. Rate: [4775 KB/s].  ETA: [0 secs]    
Converting The_Skyliners_-_Since_I_Dont_Have_You.m4a to mp3
Fixing meta-tags

8. Roy Orbison - You Got It
  3,421,193 Bytes [100.00%] received. Rate: [5146 KB/s].  ETA: [0 secs]    
Converting Roy_Orbison_-_You_Got_It.m4a to mp3
Fixing meta-tags

9. BØRNS - Electric Love
  3,507,910 Bytes [100.00%] received. Rate: [4874 KB/s].  ETA: [0 secs]    
Converting BØRNS_-_Electric_Love.m4a to mp3
Fixing meta-tags

10. Into The Night-Benny Mardones
  4,234,613 Bytes [100.00%] received. Rate: [4974 KB/s].  ETA: [0 secs]    
Converting Benny_Mardones_-_Into_The_Night.m4a to mp3
Fixing meta-tags

11. Lowell - Ride (feat. Icona Pop) [Nerve 2016 Movie Soundtrack]
  3,112,873 Bytes [100.00%] received. Rate: [4934 KB/s].  ETA: [0 secs]    
Converting Lowell_-_Ride_(feat_Icona_Pop).m4a to mp3
Fixing meta-tags

12. Diplomats of Solid Sound - Gimme one more chance
  3,329,489 Bytes [100.00%] received. Rate: [4979 KB/s].  ETA: [0 secs]    
Converting Diplomats_of_Solid_Sound_-_Gimme_One_More_Chance.m4a to mp3
Fixing meta-tags

13. Halsey - Hurricane (Arty Remix/Audio)
  3,584,050 Bytes [100.00%] received. Rate: [4536 KB/s].  ETA: [0 secs]    
Converting Halsey_-_Hurricane_-_Arty_Remix.m4a to mp3
Fixing meta-tags

14. Pyramid Vritra - Black Belts ft. Pyramid Quince
  3,030,140 Bytes [100.00%] received. Rate: [4904 KB/s].  ETA: [0 secs]    
Converting Pyramid_Vritra_-_Black_Belts.m4a to mp3
Fixing meta-tags

15. Darke Complex - "Invertebrate"
  5,180,715 Bytes [100.00%] received. Rate: [5156 KB/s].  ETA: [0 secs]    
Converting Darke_Complex_-_Invertebrate.m4a to mp3
Fixing meta-tags

16. Bro Safari - Bender
  4,415,473 Bytes [100.00%] received. Rate: [3544 KB/s].  ETA: [0 secs]    
Converting Bro_Safari_-_Bender.m4a to mp3
Fixing meta-tags

17. Tei Shi - "Bassically" (Official Audio)
  4,625,335 Bytes [100.00%] received. Rate: [4747 KB/s].  ETA: [0 secs]    
Converting Tei_Shi_-_Bassically.m4a to mp3
Fixing meta-tags

18. Holy Ghost! "Okay" (Official Music Video) - DFA RECORDS
  4,459,270 Bytes [100.00%] received. Rate: [4939 KB/s].  ETA: [0 secs]    
Converting Holy_Ghost_-_Okay.m4a to mp3
Fixing meta-tags

19. Alex Winston - Down Low [Official Audio]
  4,030,346 Bytes [100.00%] received. Rate: [5047 KB/s].  ETA: [0 secs]    
Converting Alex_Winston_-_Down_Low.m4a to mp3
Fixing meta-tags

20. Grandtheft - Give Me More [Official Full Stream]
  4,230,932 Bytes [100.00%] received. Rate: [4379 KB/s].  ETA: [0 secs]    
Converting Grandtheft_-_Give_Me_More.m4a to mp3
Fixing meta-tags

21. Holy ghost!-Crime cutz-(baile)
  6,685,189 Bytes [100.00%] received. Rate: [5273 KB/s].  ETA: [0 secs]    
Converting Holy_Ghost_-_Crime_Cutz.m4a to mp3
Fixing meta-tags

22. Blood Orange - Forget It (Album Version)
  3,765,621 Bytes [100.00%] received. Rate: [4784 KB/s].  ETA: [0 secs]    
Converting Blood_Orange_-_Forget_It.m4a to mp3
Fixing meta-tags

23. Jungle - Lucky I Got What I Want [HD]
  4,211,032 Bytes [100.00%] received. Rate: [5175 KB/s].  ETA: [0 secs]    
Converting Jungle_-_Lucky_I_Got_What_I_Want.m4a to mp3
Fixing meta-tags

real    5m34.816s
user    3m55.796s
sys 0m3.500s

Time: 5m34s. All 23 songs were downloaded and converted correctly. But in a much longer time, so I definitively see space for improvement here.

And, last I noticed it kind of breaks my Terminal: typing is not seen unless its length exceeds the length of the prompt, and hitting enter some times produces

(spotdl) linus@sony-vaio ~/Code/python/projects/spotify-downloader $ (spotdl) linus@sony-vaio ~/Code/python/projects/spotify-downloader $ (spotdl) linus@sony-vaio ~/Code/python/projects/spotify-downloader $ (spotdl) linus@sony-vaio ~/Code/python/projects/spotify-downloader $

:cry:

Any help would be appreciated.

linusg commented 7 years ago

Update: I will not work on this any further, since I don't see a practicable way of doing it. @ndragon798 I believe Ritiek has currently not much time either, so if you would work on a solution we would accept your PR. Otherwise we will likely have to close this.

ritiek commented 7 years ago

Let's just keep this open. I have a few days off at the end of this month. I'll try to put some effort on this. :smile:

ndragon798 commented 7 years ago

Keep it open I will look into it over the next few weekends.

On Tue, Sep 26, 2017 at 5:51 AM, Ritiek Malhotra notifications@github.com wrote:

Let's just keep this open. I have a few days off at the end of this month. I'll try to put some effort in this. 😄

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ritiek/spotify-downloader/issues/130#issuecomment-332147238, or mute the thread https://github.com/notifications/unsubscribe-auth/AJ42itSO4s0lOMYExe3tNCwzo_iYBVoTks5smMkygaJpZM4PXq2M .

ritiek commented 7 years ago

Sorry, I am late but I tried to implement basic threading in threading branch. It downloads the song in the main thread and then attempts to convert + metadata in a separate thread. Here are the results with 10 songs:

With Threading:

ritiek@linux:~/Downloads/spotify-downloader $ time python3 spotdl.py -l=anime-music.txt 
Total songs in list: 10 songs

1. [너의 이름은。OST]  RADWIMPS - 스파클 (Movie Ver.) [한국어 자막]/「君の名は。」OST  RADWIMPS - スパークル (Movie Ver.) [ふりがな]
  8,493,149 Bytes [100.00%] received. Rate: [ 306 KB/s].  ETA: [0 secs]    
Converting RADWIMPS_-_Sparkle_-_Movie_Version.m4a to mp3

2. Kimi no na wa OST - Kataware Doki (Piano)
  3,760,462 Bytes [100.00%] received. Rate: [ 491 KB/s].  ETA: [0 secs]    
Converting Theishter_-_Kataware_Doki_(Kimi_No_Na_Wa_Original_Soundtrack).m4a to mp3

Fixing meta-tags
3. Lauren Aquilina - King (Lyrics)
Fixing meta-tags
  3,780,442 Bytes [100.00%] received. Rate: [ 542 KB/s].  ETA: [0 secs]    
Converting Lauren_Aquilina_-_King.m4a to mp3

Fixing meta-tags
4. Most Beautiful OST's of All Time: Kataware Doki
  2,661,462 Bytes [100.00%] received. Rate: [ 217 KB/s].  ETA: [0 secs]    
Converting RADWIMPS_-_Katawaredoki.m4a to mp3

Fixing meta-tags
5. SkiDropz - Indigo | Diversity Release
  4,020,724 Bytes [100.00%] received. Rate: [ 132 KB/s].  ETA: [0 secs]    
Converting SkiDropz_-_Indigo.m4a to mp3

6. RADWIMPS - Yumetourou [Your name./AMV version] ซับไทย
Fixing meta-tags
  2,017,563 Bytes [100.00%] received. Rate: [ 643 KB/s].  ETA: [0 secs]    
Converting RADWIMPS_-_Yumetourou.m4a to mp3

Fixing meta-tags
7. RADWIMPS - Nandemonaiya [Your name./Movie version] ซับไทย
  5,402,385 Bytes [100.00%] received. Rate: [ 321 KB/s].  ETA: [0 secs]    
Converting RADWIMPS_-_Nandemonaiya_-_Movie_Version.m4a to mp3

8. 前前前世 (movie ver.) RADWIMPS MV
Fixing meta-tags
  4,644,498 Bytes [100.00%] received. Rate: [ 761 KB/s].  ETA: [0 secs]    
Converting RADWIMPS_-_Zenzenzense_-_Movie_Version.m4a to mp3

Fixing meta-tags
9. RADWIMPS - Nandemonaiya [Your name./Movie edit] ซับไทย
  3,153,415 Bytes [100.00%] received. Rate: [ 897 KB/s].  ETA: [0 secs]    
Converting RADWIMPS_-_Nandemonaiya_-_Movie_Edit.m4a to mp3

Fixing meta-tags
10. Theme Of Mitsuha —— 三葉のテーマ —— 三叶的主题
  3,825,068 Bytes [100.00%] received. Rate: [ 553 KB/s].  ETA: [0 secs]    
Converting RADWIMPS_-_Mitsuha_No_Theme.m4a to mp3

Fixing meta-tags

real    7m33.584s
user    4m53.080s
sys 0m5.508s

Without Threading:

ritiek@linux:~/Downloads/spotify-downloader $ time python3 spotdl.py -l=anime-music.txt 
Total songs in list: 10 songs

1. [너의 이름은。OST]  RADWIMPS - 스파클 (Movie Ver.) [한국어 자막]/「君の名は。」OST  RADWIMPS - スパークル (Movie Ver.) [ふりがな]
  8,493,149 Bytes [100.00%] received. Rate: [ 372 KB/s].  ETA: [0 secs]    
Converting RADWIMPS_-_Sparkle_-_Movie_Version.m4a to mp3
Fixing meta-tags

2. Kimi no na wa OST - Kataware Doki (Piano)
  3,760,462 Bytes [100.00%] received. Rate: [ 379 KB/s].  ETA: [0 secs]    
Converting Theishter_-_Kataware_Doki_(Kimi_No_Na_Wa_Original_Soundtrack).m4a to mp3
Fixing meta-tags

3. Lauren Aquilina - King (Lyrics)
  3,780,442 Bytes [100.00%] received. Rate: [ 362 KB/s].  ETA: [0 secs]    
Converting Lauren_Aquilina_-_King.m4a to mp3
Fixing meta-tags

4. Most Beautiful OST's of All Time: Kataware Doki
  2,661,462 Bytes [100.00%] received. Rate: [ 157 KB/s].  ETA: [0 secs]    
Converting RADWIMPS_-_Katawaredoki.m4a to mp3
Fixing meta-tags

5. SkiDropz - Indigo | Diversity Release
  4,020,724 Bytes [100.00%] received. Rate: [ 203 KB/s].  ETA: [0 secs]    
Converting SkiDropz_-_Indigo.m4a to mp3
Fixing meta-tags

6. RADWIMPS - Yumetourou [Your name./AMV version] ซับไทย
  2,017,563 Bytes [100.00%] received. Rate: [ 110 KB/s].  ETA: [0 secs]    
Converting RADWIMPS_-_Yumetourou.m4a to mp3
Fixing meta-tags

7. RADWIMPS - Nandemonaiya [Your name./Movie version] ซับไทย
  5,402,385 Bytes [100.00%] received. Rate: [ 203 KB/s].  ETA: [0 secs]    
Converting RADWIMPS_-_Nandemonaiya_-_Movie_Version.m4a to mp3
Fixing meta-tags

8. 前前前世 (movie ver.) RADWIMPS MV
  4,644,498 Bytes [100.00%] received. Rate: [ 182 KB/s].  ETA: [0 secs]    
Converting RADWIMPS_-_Zenzenzense_-_Movie_Version.m4a to mp3
Fixing meta-tags

9. RADWIMPS - Nandemonaiya [Your name./Movie edit] ซับไทย
  3,153,415 Bytes [100.00%] received. Rate: [ 275 KB/s].  ETA: [0 secs]    
Converting RADWIMPS_-_Nandemonaiya_-_Movie_Edit.m4a to mp3
Fixing meta-tags

10. Theme Of Mitsuha —— 三葉のテーマ —— 三叶的主题
  3,825,068 Bytes [100.00%] received. Rate: [ 983 KB/s].  ETA: [0 secs]    
Converting RADWIMPS_-_Mitsuha_No_Theme.m4a to mp3
Fixing meta-tags

real    11m19.416s
user    4m23.036s
sys 0m4.712s

Threads improved the downloading times by nearly 4 minutes in my case. However, my download speed keeps throttling, so these are no genuine results. As it may occur, the benefit of threading depends on how balanced your hardware and download speed are.

I did not get any weird errors so far. I wonder how you implemented it @linusg?

linusg commented 7 years ago

grab_list: @ritiek

def grab_list(text_file):
    """Download all songs from the list."""
    with open(text_file, 'r') as listed:
        lines = (listed.read()).splitlines()
    # ignore blank lines in text_file (if any)
    try:
        lines.remove('')
    except ValueError:
        pass
    print(u'Total songs in list: {0} songs'.format(len(lines)))
    print('')
    threads = []
    # nth input song
    number = 1
    for raw_song in lines:
        try:
            thread = threading.Thread(target=grab_single, args=(raw_song,), kwargs={'number': number})
            thread.start()
            threads.append(thread)
            print(thread)
            time.sleep(5)
            # grab_single(raw_song, number=number)
        # token expires after 1 hour
        except spotipy.oauth2.SpotifyOauthError:
            # refresh token when it expires
            new_token = misc.generate_token()
            global spotify
            spotify = spotipy.Spotify(auth=new_token)
            grab_single(raw_song, number=number)
        # detect network problems
        except (urllib.request.URLError, TypeError, IOError):
            lines.append(raw_song)
            # remove the downloaded song from .txt
            misc.trim_song(text_file)
            # and append it to the last line in .txt
            with open(text_file, 'a') as myfile:
                myfile.write(raw_song + '\n')
            print('Failed to download song. Will retry after other songs.')
            # wait 0.5 sec to avoid infinite looping
            time.sleep(0.5)
            continue
        except KeyboardInterrupt:
            misc.grace_quit()
        finally:
            print('')
        misc.trim_song(text_file)
        number += 1
    for thread in threads:
        thread.join()
        print('Thread ended')

I'm half out of the door, I can send you the whole file later. Guess this is most I changed :thinking:

netllama commented 6 years ago

Any one have time to resume work on this effort?

jerbob commented 5 years ago

I'm interested in wokring on this issue, if no one else is working on it at the moment?

jerbob commented 5 years ago

I'll fork the repository, and make a PR as soon as I can.

magnusmaynard commented 5 years ago

Here's my dodgy bash script if anyone wants it. It downloads a playlist in separate terminals (and therefore threads). It gets the occasional conflict, but is definitely faster than single threaded:

#!/bin/bash
# 1 - playlist link
# 2 - playlist name

# download playlist text
spotdl -p $1

# reverse text for most recent song first.
tac $2.txt > $2_rev.txt 

LINES="$(wc -l $2.txt | awk '{ print $1 }')"
THREADS=3 # threads too high causes failures.
LINES_PER_THREAD="$(($LINES/$THREADS+1))"

# split text
split -l $LINES_PER_THREAD -d $2.txt $2 --additional-suffix split.txt

# execute download in seperate terminal
for file in *split.txt; do
    # use custom search pattern, the default sucks
    gnome-terminal -- bash -c "spotdl --list $file -f $(pwd) --search-format "'"{artist} - {track_name}"'"; exec bash"
done

ritiek commented 5 years ago

I've been reading about related stuff lately. I wonder whether async will be a better choice versus multithreading?

playXE commented 4 years ago

I also got small program that allows downloading in multiple processes, it's based on @magnusmaynard shell script but allows downloading in 8,10,16 processes

use std::fs::{File, OpenOptions};
use std::io::{BufRead, BufReader, Read, Write};

fn main() -> std::io::Result<()> {
    const THREADS: usize = 8;
    let file = File::open("new-playlist.txt")?;
    let f = BufReader::new(file);
    let lines = f
        .lines()
        .map(|x| x.unwrap().to_string())
        .collect::<Vec<String>>();

    let lines_per_thread = lines.len() / THREADS + 1;
    let mut c = 0;
    let mut cmds = vec![];
    for i in 1..=THREADS {
        let first_one = &lines[c..lines_per_thread * i - 1];
        let mut f = OpenOptions::new()
            .write(true)
            .create(true)
            .open(&format!("file-{}.txt", i))
            .unwrap();
        f.set_len(0)?;
        let mut buf = String::new();
        for l in first_one.iter() {
            buf.push_str(&format!("{}\n", l));
        }
        f.write_all(buf.as_bytes())?;
        c += lines_per_thread;

        cmds.push(
            std::process::Command::new("spotdl")
                .arg("--list")
                .arg(&format!("file-{}.txt", i))
                .spawn()
                .unwrap(),
        );
    }

    Ok(())
}

ochen1 commented 4 years ago

For now, I think I will just stick to a wrapper script (Python) for spotdl, spotdl-multithreaded. Super impractical, as it just takes an input playlist, splits the playlist file, then opens up a gnome-terminal -- spotdl. I program in Python but am too lazy to actually go and find out and understand how spotdl works XD

ghost commented 4 years ago

This is done. Now using multiprocessing.

spotDL / spotify-downloader

Add multithread option #130

What is the purpose of your issue?

System information

Description