Ubuntu 18.04.1 - Poetry Issue? setup.py

[undaunt]

I seem to be getting stuck during configuration. Originally I could not run the requirements configuration, but was able to resolve that but installing python-tk manually. Then I was getting an error during setup.py regarding python3-venv, so I also manually installed that.

Now, running through config steps fresh, I get held up during some kind of poetry call.

sudo apt install python3 python-pip python3-pip python-tk python3-venv

cd Automatic-Youtube-Downloader
pip3 install -r requirements.txt

python3 setup.py

When I hit 3, I get through some of the resolving dependencies, then it fails out when trying to install python-xlib.

What would you like to do?   
3
Checking Dependencies....  
Python Path: /usr/bin/python3
Retrieving Poetry metadata

Latest version already installed.
Updating dependencies
Resolving dependencies... (0.8s)

Package operations: 2 installs, 0 updates, 0 removals

  - Installing python-xlib (0.25)

[EnvCommandError]                                                                                                            
Command ['/home/USER/.cache/pypoetry/virtualenvs/automatic-youtube-downloader-py3.6/bin/python', '-m', 'pip', 'install', '--no-deps', 'python-  
xlib==0.25'] errored with the following return code 1, and output:                                                                        
/home/USER/.cache/pypoetry/virtualenvs/automatic-youtube-downloader-py3.6/bin/python: No module named pip                                 

update [--no-dev] [--dry-run] [--lock] [--] [<packages>]...

An error occurred with running poetry  

[PizzaWaffles]

So poetry is the dependency manager for this project and it creates its own virtual environment for package management. It looks like pip either isn't installed or isn't in your PATH. Start by first going to

/home/USER/.cache/pypoetry/virtualenvs/automatic-youtube-downloader-py3.6

and delete that directory, then run these commands to make sure pip is installed

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py python3 get-pip.py --force-reinstall

Try running the config again, if you still get an error run these commands and post your results here

/home/USER/.cache/pypoetry/virtualenvs/automatic-youtube-downloader-py3.6/bin/python -v ls -a /home/USER/.cache/pypoetry/virtualenvs/automatic-youtube-downloader-py3.6/bin/

[undaunt]

That was it. I re-ran the install as you suggested (had to append --user), added the /home/user/.local/bin to the path, and all set.

Unrelated, but is there anywhere to put granular quality selections for the youtube-dl video and/or audio types? Based on some reading here: https://www.reddit.com/r/DataHoarder/comments/c6fh4x/after_hoarding_over_50k_youtube_videos_here_is/

I wanted to go with something like the following:

-f "(bestvideo[vcodec^=av01][height>=2160][fps>30]/bestvideo[vcodec=vp9.2][height>=2160][fps>30]/bestvideo[vcodec=vp9][height>=2160][fps>30]/bestvideo[vcodec^=av01][height>=2160]/bestvideo[vcodec=vp9.2][height>=2160]/bestvideo[vcodec=vp9][height>=2160]/bestvideo[height>=2160]/(bestvideo[vcodec^=av01][height>=1440][fps>30]/bestvideo[vcodec=vp9.2][height>=1440][fps>30]/bestvideo[vcodec=vp9][height>=1440][fps>30]/bestvideo[vcodec^=av01][height>=1440]/bestvideo[vcodec=vp9.2][height>=1440]/bestvideo[vcodec=vp9][height>=1440]/bestvideo[height>=1440]/(bestvideo[vcodec^=av01][height>=1080][fps>30]/bestvideo[vcodec=vp9.2][height>=1080][fps>30]/bestvideo[vcodec=vp9][height>=1080][fps>30]/bestvideo[vcodec^=av01][height>=1080]/bestvideo[vcodec=vp9.2][height>=1080]/bestvideo[vcodec=vp9][height>=1080]/bestvideo[height>=1080]/bestvideo[vcodec^=av01][height>=720][fps>30]/bestvideo[vcodec=vp9.2][height>=720][fps>30]/bestvideo[vcodec=vp9][height>=720][fps>30]/bestvideo[vcodec^=av01][height>=720]/bestvideo[vcodec=vp9.2][height>=720]/bestvideo[vcodec=vp9][height>=720]/bestvideo[height>=720]/bestvideo)+(bestaudio[ext=m4a]/bestaudio)/best"

VIDEO_QUALITY_LIST in AYD.py, but not sure on syntax?

Thanks so much! Super excited to try it out.

Edit: During running of main.py I also saw these:

WARNING: You have requested multiple formats but ffmpeg or avconv are not installed. The formats won't be merged. WARNING: Requested formats are incompatible for merge and will be merged into mkv.

ffmpeg subfolder appears to exist in the directory. Do I need to add those to path as well? Thanks!

[PizzaWaffles]

Interesting, currently the program just checks for webm formats and if there is none it will default to "best" Try changing line 575 in AYD.py so instead of 'format': usable_format_code_video + "+" + usable_format_code_audio it will be 'format': usable_format_code then add the variable before line 557 on the same margin so it would be

                usable_format_code = [your quality code setting here]
                if not skip_download:

Let me know if that works for you and Ill add it to the project

Also for ffmpeg it should be installed by poetry but if your getting that error it wasn't try running this (the ffmpeg folder in the project just has the windows binaries in it): sudo apt-get install ffmpeg

[undaunt]

Confirmed that a manual add of sudo apt install ffmpeg removed that issue. Now I'm getting mkv and jpg posters as expected.

After a successful run, (during config I set it to run once per day at 2am, but I then manually called the script to test) I observed it appears to hang on "Starting on run number 1". Does it not like that I called it manually even though its on a schedule? I tested this on my local Mac and linux server. Mac works on run once, changed it to schedule and it hangs, back to run once and it is ok. Output from interrupt:

Starting on run number 1

^CTraceback (most recent call last):
  File "AYD.py", line 756, in <module>
    start()
  File "AYD.py", line 753, in start
    sch.run()
  File "AYD.py", line 397, in run
    time.sleep(self.minutes_to_wait * 60)
KeyboardInterrupt
Traceback (most recent call last):
  File "main.py", line 56, in <module>
    proc = subprocess.call(cmd)
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 341, in call
    return p.wait(timeout=timeout)
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 1019, in wait
    return self._wait(timeout=timeout)
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 1653, in _wait
    (pid, sts) = self._try_wait(0)
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 1611, in _try_wait
    (pid, sts) = os.waitpid(self.pid, wait_flags)
KeyboardInterrupt

Re: the formatting (which I did not modify until after the above test run, I had to force quit the process after leaving it for 10-15 mins) - I don't think it liked it.

Failed to Download
[youtube] DBJ-In8BCwo: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by <HTTPError 429: 'Too Many Requests'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Edit: Seems like I may have a rate limit ban actually, so I'd like to re-test the above, but the below points are still pertinent. Hopefully it's not that long.

Would it be possibly better to try specifically setting these in audio or video specific areas such as around line 510? IE: Could I change

usable_format_code_audio = 'bestaudio'

to

usable_format_code_audio = 'bestaudio[ext=m4a]/bestaudio'

In that case, my question becomes, if I want to try and specify a codec as part of the video 'best' call, and not specify an extension, can I modify or comment out

usable_extension = 'webm'

Other Edit: I seem to need to keep incrementing the number of videos to download by 1, even if I delete the prior video, to make a new video download to test. Are the prior videos being logged somewhere even without leveraging the --download-archive functionality of youtube-dl?

Thanks!

[PizzaWaffles]

For your first point the RUN_ONCE setting is none blocking and the program will find and download all available videos then quit. This is useful when using a cron job (this is how I have it setup on my system.) The TIME_OF_DAY setting will WAIT till the specified hour comes then execute. The DELAY setting will WAIT for a specified amount. If you want it to run in the background append a & to the end of the command like so: python3 main.py & it will be attached to your shell so if it dies (you close the terminal) the program will be stopped. I recommend either setting up a simple crontab job and using the RUN_ONCE setting or you can setup a systemd service using TIME_OF_DAY of DELAY, when you are ready for long-term usage.

For the second point you could change it on line 510 but the code that follows it may overwrite it as its trying to match up a quality profile to a format code. If you want to do it this way, comment out or delete lines 514-555.

And finally yes AYD keeps its own separate log file for successful downloads located at data/log.txt. For testing purposes you can delete the data/log.txt file after each test.

[PizzaWaffles]

I had some free time so I implemented something that will also work if someone would like to choose their quality settings. Im going to push my changes soon.

EDIT: 684ca4be5c0abeaa815016ca2304bca51bca6a56 Updated AYD.py with suggested changes

And open up your data/config and change the VIDEO_FORMAT setting to 'best' like so: VIDEO_FORMAT=best

[undaunt]

Regarding the run method, got it - so if I am testing the script I should set it to run once, even if eventually I'll use the time of day run, as otherwise it will just sit and wait. Also, thanks for the tip regarding the offscreen run with &, I wasn't sure if I could run python stuff in screen - that's my go to.

Also understood now regarding the data/log.txt, thank you!

I think I accidentally tested the code changes after a git pull without setting my format to "best" so I'll have to test again when I'm unblocked. Thanks for implementing that. Does it matter that the format is still set to webm on line 509? I wasn't sure webm vs mp4 and webm vs m4a worked for things such as the video codec (av01, vp9, etc.) or audio codec (opus, etc). Are we forcing webm extension instead of m4a/mp4 with that line?

Thanks again! Will report back when I'm un-temp-banned.