Handle GitHub changes to fetch latest `yt-dlp`

richardsondev commented 2 years ago

Issue

GitHub has removed the response body when redirecting a user during release downloads which has broken the yt-dlp runtime download. This is causing first time executions of the application to not download yt-dlp and return "Error! Binaries missing/corrupted" in the UI.

Background

The approach to grab the latest version of yt-dlp was relying on GitHub to return the redirect URL in the body response when fetching the latest binary. GitHub has changed to no longer return the URL in the response body and is only emitting it in the Location header now.

This PR

This PR is changing the logic to read the version from the Location header during the redirect.

In my testing the application now correctly detects the latest version number for yt-dlp and downloads it correctly.

After screenshot

ytdlVersion file

{"version":"2022.05.18","ytdlp":true}

Tests

olegshulyakov commented 2 years ago

@richardsondev Please also update modules/Filepaths.js set binary names like following:

case "darwin":
  this.ytdl = this.app.isPackaged ? path.join(this.unpackedPrefix, "binaries/yt-dlp_macos") : "binaries/yt-dlp_macos";
  ...
  break;
case "linux":
  this.ytdl = this.app.isPackaged ? path.join(this.persistentPath, "yt-dlp") : "binaries/yt-dlp";
  ...
  break;

StefanLobbenmeier commented 2 years ago

@richardsondev Please also update modules/Filepaths.js set binary names like following:

That would also require a change to getRemoteVersion. I agree that we should also get yt-dlp_macos and yt-dlp_macos_legacy now and I can make a pull request to also fix that, but until then we should merge this as soon as possible since it resolves so many issues.

StefanLobbenmeier commented 2 years ago

I can confirm that this fixes the issue :smile: LGTM

42Willow commented 1 year ago

How can I download this? I need to download a youtube video by tomorrow because I need to be able to view it offline...

StefanLobbenmeier commented 1 year ago

@42Willow the quickest solution is to use https://github.com/jely2002/youtube-dl-gui/releases/tag/v2.3.1 instead, that version still packages youtube_dl.

Altnernatively I merged this PR in my own fork and made a release for macOS and Linux. Unfortunately the Windows Build is broken: https://github.com/StefanLobbenmeier/youtube-dl-gui/releases/tag/v2.4.2

jrowberg commented 1 year ago

How can I download this? I need to download a youtube video by tomorrow because I need to be able to view it offline...

In the meantime, you can also download the yt-dlp binary manually and put it where it's supposed to be; I just did this successfully on my Windows installation. You can find yt-dlp here:

https://github.com/yt-dlp/yt-dlp/releases

On Windows, the yt-dlp.exe file should be placed in C:\Users\[YOURUSER]\AppData\Local\Programs\youtube-dl-gui\resources\app.asar.unpacked\binaries, assuming a standard installation.

claudiusraphael commented 1 year ago

Dunno if this is the right place, but stumbled over the same error in Windows 11 and therefore went to build this branch (richardson:handlegithubchangesytdlp) via npm under Node Current (v18.8.0):

PS C:\Users\claudiusraphael\Downloads\youtube-dl-gui-handlegithubchangesytdlp\youtube-dl-gui-handlegithubchangesytdlp> npm install
npm WARN old lockfile
npm WARN old lockfile The package-lock.json file was created with an old version of npm,
npm WARN old lockfile so supplemental metadata must be fetched from the registry.
npm WARN old lockfile
npm WARN old lockfile This is a one-time fix-up, please be patient...
npm WARN old lockfile
npm WARN deprecated popper.js@1.16.1: You can find the new Popper v2 at @popperjs/core, this package is dedicated to the legacy v1
npm WARN deprecated core-js@3.18.3: core-js@<3.23.3 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Some versions have web compatibility issues. Please, upgrade your dependencies to the actual version of core-js.

added 693 packages, and audited 694 packages in 1m

53 packages are looking for funding
  run `npm fund` for details

8 vulnerabilities (7 moderate, 1 critical)

To address issues that do not require attention, run:
  npm audit fix

To address all issues (including breaking changes), run:
  npm audit fix --force`

Run `npm audit` for details.
npm notice
npm notice New minor version of npm available! 8.18.0 -> 8.19.1
npm notice Changelog: https://github.com/npm/cli/releases/tag/v8.19.1
npm notice Run npm install -g npm@8.19.1 to update!
npm notice
PS C:\Users\claudiusraphael\Downloads\youtube-dl-gui-handlegithubchangesytdlp\youtube-dl-gui-handlegithubchangesytdlp> npx electron-builder --win
  • electron-builder  version=22.9.1 os=10.0.22000
  • loaded configuration  file=package.json ("build" field)
  • writing effective config  file=dist\builder-effective-config.yaml
  • packaging       platform=win32 arch=x64 electron=11.5.0 appOutDir=dist\win-unpacked
  • downloading     url=https://github.com/electron/electron/releases/download/v11.5.0/electron-v11.5.0-win32-x64.zip size=78 MB parts=8
  • downloaded      url=https://github.com/electron/electron/releases/download/v11.5.0/electron-v11.5.0-win32-x64.zip duration=11.77s
  • downloading     url=https://github.com/electron-userland/electron-builder-binaries/releases/download/winCodeSign-2.6.0/winCodeSign-2.6.0.7z size=5.6 MB parts=1
  • downloaded      url=https://github.com/electron-userland/electron-builder-binaries/releases/download/winCodeSign-2.6.0/winCodeSign-2.6.0.7z duration=3.465s
  • building        target=nsis file=dist\Open Video Downloader Setup 2.4.0.exe archs=x64 oneClick=true perMachine=false
  • downloading     url=https://github.com/electron-userland/electron-builder-binaries/releases/download/nsis-resources-3.4.1/nsis-resources-3.4.1.7z size=731 kB parts=1
  • downloaded      url=https://github.com/electron-userland/electron-builder-binaries/releases/download/nsis-resources-3.4.1/nsis-resources-3.4.1.7z duration=1.487s
  • downloading     url=https://github.com/electron-userland/electron-builder-binaries/releases/download/nsis-3.0.4.1/nsis-3.0.4.1.7z size=1.3 MB parts=1
  • downloaded      url=https://github.com/electron-userland/electron-builder-binaries/releases/download/nsis-3.0.4.1/nsis-3.0.4.1.7z duration=1.683s
  • building block map  blockMapFile=dist\Open Video Downloader Setup 2.4.0.exe.blockmap

Build went fine and can confirm that the error doesn't appear anymore and that downloading videos works as it should.

If this comment should be applied as a "Review" instead, let me know - i wasn't sure about that.

StefanLobbenmeier commented 1 year ago

@claudiusraphael no worries, this repo is anarchy now anyway 😄

Good to know that the windows build still works locally, the CI build has some strange powershell in it that fails:

Run echo  > env.b64
  echo  > env.b64
  certutil -decode env.b64 .env
  del env.b64
  shell: C:\Program Files\PowerShell\7\pwsh.EXE -command ". '{0}'"
Write-Output: D:\a\_temp\b171[2](https://github.com/StefanLobbenmeier/youtube-dl-gui/runs/7121445209?check_suite_focus=true#step:5:2)3ec-0f5e-40cd-a6e9-03771331ce46.ps1:2
Line |
   2 |  echo  > env.b64
     |  ~~~~~~~~~~~~~~~
     | Cannot process command because of one or more missing mandatory parameters: InputObject.

Error: Process completed with exit code 1.

So maybe we can also fix the CI build. Is there actually anyone interested continuing dev in an active fork since @jely2002 does not seem to respond anymore? I could offer mine, but I can only review PRs and not develop myself

StefanLobbenmeier commented 1 year ago

Inspired by that it is actually supposed to be able to build on windows I fixed the CI (removed anything related to Sentry) and made a new release here: https://github.com/StefanLobbenmeier/youtube-dl-gui/releases/tag/v2.4.3

Feel free to link that in future issues 😄

claudiusraphael commented 1 year ago

Inspired by that it is actually supposed to be able to build on windows I fixed the CI (removed anything related to Sentry) and made a new release here: https://github.com/StefanLobbenmeier/youtube-dl-gui/releases/tag/v2.4.3

Feel free to link that in future issues 😄

Just to confirm, tried the installer (for Windows: Open-Video-Downloader-Setup-2.4.3.exe ) and it works as expected under Windows 11 with latest Updates!

[EDIT] P.s.: Can also confirm AppImage (Open-Video-Downloader-2.4.3.AppImage ) under Fedora 35 Workstation + Ubuntu Desktop 22.04 and 20.04 and DMG (Open-Video-Downloader-2.4.3.dmg ) under Monterey with latest Updates (tested on MBP 13" Late 2015) , work as well!

claudiusraphael commented 1 year ago

@claudiusraphael no worries, this repo is anarchy now anyway 😄

Good to know that the windows build still works locally, the CI build has some strange powershell in it that fails:
Run echo  > env.b64
  echo  > env.b64
  certutil -decode env.b64 .env
  del env.b64
  shell: C:\Program Files\PowerShell\7\pwsh.EXE -command ". '{0}'"
Write-Output: D:\a\_temp\b171[2](https://github.com/StefanLobbenmeier/youtube-dl-gui/runs/7121445209?check_suite_focus=true#step:5:2)3ec-0f5e-40cd-a6e9-03771331ce46.ps1:2
Line |
   2 |  echo  > env.b64
     |  ~~~~~~~~~~~~~~~
     | Cannot process command because of one or more missing mandatory parameters: InputObject.

Error: Process completed with exit code 1.
So maybe we can also fix the CI build. Is there actually anyone interested continuing dev in an active fork since @jely2002 does not seem to respond anymore? I could offer mine, but I can only review PRs and not develop myself

If you find a minute would you be so kind to post an excerpt/diff here of the changes made? I think it might be helpful to understand the details (for Learners), especially regarding PowerShell. Thx!

StefanLobbenmeier commented 1 year ago

If you find a minute would you be so kind to post an excerpt/diff here of the changes made? I think it might be helpful to understand the details (for Learners), especially regarding PowerShell. Thx!

no problem - you can find all my changes here: https://github.com/StefanLobbenmeier/youtube-dl-gui/commits/master

I had to remove sentry from the Linux build and the windows build here: https://github.com/StefanLobbenmeier/youtube-dl-gui/commit/eba0da19c0d594990f1b15a632a89832eeefccde

The reason those sentry related lines are in there are because this is a web service for error reporting, where I would need the secret of the owner to access his project. I could also create a new sentry project and use my own secret there, but I was not able to get that working. For now removing it is the easier solution.

jely2002 commented 1 year ago

Hi folks, will take a look at some open PR's in a couple hours. I need to test them before I can confidently release a new version and that takes time as it has been a while.

claudiusraphael commented 1 year ago

Hi folks, will take a look at some open PR's in a couple hours. I need to test them before I can confidently release a new version and that takes time as it has been a while.

I don't know what the actual status is, but if you need a tester for a new release on Windows 11 Pro, Windows 10 LTSC Enterprise, Macos Monterey, Fedora Workstation 36, Ubuntu 22.04, let me know.

If of help provide me detailed steps for debugging/creating reports.

To inform/ping me you can do so on discord: claudiusraphael#2172

odhiambo commented 1 year ago

Version 2.4.3 is running fine on my W11Pro.

StefanLobbenmeier commented 1 year ago

Just to avoid confusion @odhiambo you are talking about a release from https://github.com/StefanLobbenmeier/youtube-dl-gui/releases/? There is no 2.4.3 in https://github.com/jely2002/youtube-dl-gui/releases

odhiambo commented 1 year ago

Sorry for the confusion, but I was referring to the one created by @StefanLobbenmeier

MrFobwatch commented 1 year ago

So is there going to be active development on this repo anymore? I am using @StefanLobbenmeier 's fork now on my machine but was trying to figure out if I should put in a pull request to try and redirect the homebrew cask to the new repo as it is currently pulling the broken version from this repo.

jely2002 / youtube-dl-gui