search

e-alfred / ocdownloader

ocDownloader - AGPL-licensed multi-protocol download manager for Nextcloud using ARIA2, youtube-dl and Curl (supports Youtube, BitTorrent, HTTP, FTP)
https://github.com/e-alfred/ocdownloader
GNU Affero General Public License v3.0
375 stars 85 forks source link

Unable to retrieve true YouTube video URL #165

Closed geosphere closed 4 years ago

geosphere commented 4 years ago

I'm use Nextcloud 18 in container with Aria2 and youtube-dl in some container. Aria2 work fine, but youtube-dl work only manually.

Steps to reproduce

  1. Trying to download a Youtube video and catch error "Unable to retrieve true YouTube video URL"
  2. No matching log entries
  3. Open console and connect to docker: docker exec -it cloud sh
  4. Trying to download manually: /usr/local/bin/youtube-dl https://..... - and this work fine

Expected behaviour

Actual behaviour

Server configuration detail

Operating system: Linux 5.5.0-amd64 #1 SMP Debian 5.5.17-1 (2020-04-21) x86_64

Webserver: nginx/1.17.8 (fpm-fcgi)

Database: pgsql PostgreSQL 9.6.5 on x86_64-pc-linux-gnu, compiled by gcc (Debian 7.2.0-1) 7.2.0, 64-bit

PHP version:

7.3.14 Modules loaded: Core, date, libxml, openssl, pcre, sqlite3, zlib, ctype, curl, dom, fileinfo, filter, gd, hash, iconv, json, mbstring, SPL, PDO, pdo_sqlite, bz2, posix, readline, Reflection, session, SimpleXML, standard, tokenizer, xml, xmlreader, xmlwriter, mysqlnd, cgi-fcgi, apcu, Phar, exif, ftp, gmp, imap, intl, ldap, mysqli, pcntl, pdo_mysql, pdo_pgsql, pgsql, zip, imagick, inotify, redis, smbclient, libsmbclient, Zend OPcache

Nextcloud version: 18.0.4 - 18.0.4.2

Updated from an older Nextcloud/ownCloud or fresh install: updated

Where did you install Nextcloud from: unknown

Signing status Array ( )
List of activated apps ``` Enabled: - accessibility: 1.4.0 - activity: 2.11.0 - admin_audit: 1.8.0 - analytics: 2.2.3 - announcementcenter: 3.7.0 - apporder: 0.9.0 - bookmarks: 3.0.10 - breezedark: 18.0.14 - bruteforcesettings: 1.6.0 - calendar: 2.0.3 - camerarawpreviews: 0.7.3 - checksum: 0.4.4 - cloud_federation_api: 1.1.0 - comments: 1.8.0 - contacts: 3.3.0 - cospend: 0.4.9 - dav: 1.14.0 - deck: 0.8.2 - drawio: 0.9.5 - external: 3.5.0 - federatedfilesharing: 1.8.0 - federation: 1.8.0 - files: 1.13.1 - files_accesscontrol: 1.8.1 - files_downloadactivity: 1.7.0 - files_external: 1.9.0 - files_frommail: 0.4.0 - files_markdown: 2.2.0 - files_pdfviewer: 1.7.0 - files_retention: 1.7.0 - files_rightclick: 0.15.2 - files_sharing: 1.10.1 - files_texteditor: 2.11.0 - files_trackdownloads: 1.7.0 - files_trashbin: 1.8.0 - files_versions: 1.11.0 - files_videoplayer: 1.7.0 - firstrunwizard: 2.7.0 - flowupload: 0.1.8 - gpxedit: 0.0.13 - gpxmotion: 0.0.11 - gpxpod: 4.2.1 - groupfolders: 6.0.6 - impersonate: 1.5.0 - issuetemplate: 0.6.0 - logreader: 2.3.0 - lookup_server_connector: 1.6.0 - mail: 1.3.4 - maps: 0.1.6 - metadata: 0.11.1 - music: 0.14.0 - nextcloud_announcements: 1.7.0 - notes: 3.3.0 - notifications: 2.6.0 - oauth2: 1.6.0 - occweb: 0.0.7 - ocdownloader: 1.7.7 - ocsms: 2.1.7 - passman: 2.3.5 - password_policy: 1.8.0 - phonetrack: 0.6.2 - photos: 1.0.0 - previewgenerator: 2.3.0 - privacy: 1.2.0 - provisioning_api: 1.8.0 - qownnotesapi: 20.1.0 - quota_warning: 1.7.0 - recommendations: 0.6.0 - serverinfo: 1.8.0 - settings: 1.0.0 - sharebymail: 1.8.0 - sharingpath: 0.2.2 - smb_test: 0.2.4 - sociallogin: 2.4.6 - socialsharing_diaspora: 2.0.0 - socialsharing_email: 2.0.0 - socialsharing_facebook: 2.0.0 - socialsharing_twitter: 2.0.0 - spreed: 8.0.8 - support: 1.1.0 - survey_client: 1.6.0 - systemtags: 1.8.0 - tasks: 0.12.2 - telephoneprovider: 1.0.3 - text: 2.0.0 - theming: 1.9.0 - theming_customcss: 1.5.0 - twofactor_backupcodes: 1.7.0 - twofactor_totp: 4.1.3 - updatenotification: 1.8.0 - viewer: 1.2.0 - w2g2: 3.0.1 - workflow_script: 1.3.1 - workflowengine: 2.0.0 Disabled: - appointments - circles - dropit - encryption - files_automatedtagging - files_fulltextsearch - fulltextsearch - richdocuments - sharerenamer - user_ldap - user_saml - user_usage_report ```
Configuration (config/config.php) ``` { "datadirectory": "***REMOVED SENSITIVE VALUE***", "apps_paths": [ { "path": "\/nextcloud\/apps", "url": "\/apps", "writable": false }, { "path": "\/apps2", "url": "\/apps2", "writable": true } ], "memcache.local": "\\OC\\Memcache\\APCu", "memcache.locking": "\\OC\\Memcache\\Redis", "redis": { "host": "***REMOVED SENSITIVE VALUE***", "port": 0, "timeout": 0, "dbindex": 1 }, "instanceid": "***REMOVED SENSITIVE VALUE***", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "logtimezone": "Europe\/Kiev", "logdateformat": "Y-m-d H:i:s", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "***REMOVED SENSITIVE VALUE***" ], "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***", "overwritehost": "***REMOVED SENSITIVE VALUE***", "overwriteprotocol": "https", "dbtype": "pgsql", "version": "18.0.4.2", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "mail_smtpmode": "smtp", "mail_smtpauthtype": "LOGIN", "mail_smtpauth": 1, "mail_domain": "***REMOVED SENSITIVE VALUE***", "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "mail_smtpsecure": "tls", "mail_smtpport": "587", "mail_smtpname": "***REMOVED SENSITIVE VALUE***", "mail_smtppassword": "***REMOVED SENSITIVE VALUE***", "maintenance": false, "updater.release.channel": "stable", "loglevel": 0, "data-fingerprint": "ae85ed2460dfc8f1a3100b51adc85aa0", "ldapIgnoreNamingRules": false, "ldapProviderFactory": "\\OCA\\User_LDAP\\LDAPProviderFactory", "log_type": "file", "log_rotate_size": 1048576, "debug": false, "customclient_desktop": "https:\/\/nextcloud.com\/install\/#install-clients", "customclient_android": "https:\/\/play.google.com\/store\/apps\/details?id=com.nextcloud.client", "allow_user_to_change_display_name": true, "theme": "", "enabledPreviewProviders": [ "OC\\Preview\\Image", "OC\\Preview\\MP3", "OC\\Preview\\TXT", "OC\\Preview\\MarkDown", "OC\\Preview\\Epub", "OC\\Preview\\FB2", "OC\\Preview\\PDF" ], "has_rebuilt_cache": true } ```

Are you using external storage, if yes which one: local/smb/sftp/...

Are you using encryption:

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...

Client configuration

Browser: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0

Operating system: Linux

Logs

Web server error log ``` No similar entries ```
Nextcloud log ``` No similar entries ```
Browser log No similar entries
MoucceuWildfire commented 4 years ago

Similar error. FreeNAS-11.3-U1 + Nextcloud 18.0.4 + ARIA2. A couple of weeks ago was fine. Youtube-dl latest 2020.05.03 version and ocDownloader v1.7.7. PHP7.2 Manual downloading from command line works fine.

e-alfred commented 4 years ago

Did you set the right path in the admin setting section of ocDownloader for youtube-dl?

geosphere commented 4 years ago

/usr/local/bin/youtube-dl This is the default path. And yes, this path corresponds to the entry in the admin section.

geosphere commented 4 years ago

It looks strange, but the download from YT works when I use the plugin for Firefox ocDownloader v1.6.2. Global settings Audio/Video Format are ignoring, but the download is successful. And the same link in UI /apps/ocdownloader/add brings an error

e-alfred commented 4 years ago

Can you try to revert to 1.7.6 to test if this PR (https://github.com/e-alfred/ocdownloader/pull/158) might be the problem? It is strange because on a non-docker installation using the UI or extension works flawlessly.

Emiliaaah commented 4 years ago

I am running Nextcloud with ocdownloader on ubuntu 20.04 LTS but I also have the same issue. Could be a newer version thats causing this since I just installed it on a seperate VM instead of in a Docker container.

EDIT: I am running this in combination with nextcloud and when trying to download via the chrome extension I get the same error

MoucceuWildfire commented 4 years ago

Similar error. FreeNAS-11.3-U1 + Nextcloud 18.0.4 + ARIA2. A couple of weeks ago was fine. Youtube-dl latest 2020.05.03 version and ocDownloader v1.7.7. PHP7.2 Manual downloading from command line works fine.

I downgraded app to ver 1.7.6 and this version works fine. Forget to mention - Nextcloud works in Jail of FreeNAS.

e-alfred commented 4 years ago

Can you try to revert this change if you have ocDownloader 1.7.7 installed and retry a Youtube download: https://github.com/e-alfred/ocdownloader/commit/fc18d4368d715ad94c486186bc248ec217e376dc

Emiliaaah commented 4 years ago

I'm running 1.7.7 but how do I revert the a earlier version?

MoucceuWildfire commented 4 years ago

Can you try to revert this change if you have ocDownloader 1.7.7 installed and retry a Youtube download: fc18d43

I have to edit this controller/lib/youtube.php file in my installation according to link above?

MoucceuWildfire commented 4 years ago

Can you try to revert this change if you have ocDownloader 1.7.7 installed and retry a Youtube download: fc18d43

I have to edit this controller/lib/youtube.php file in my installation according to link above?

this didn`t help

e-alfred commented 4 years ago

@RelyIz You can download earlier versions from https://github.com/e-alfred/ocdownloader/releases and extract them to the apps folder of your Nextcloud installation.

@MoucceuWildfire Yes, trying to change lines 72-76 in this file. As it didn't help, I have to look further into this.

MoucceuWildfire commented 4 years ago

Hi e-alfred. Want to thank you for this app at first. Maybe you can tell me how to change the source to add some logging. Just for debugging purposes. I want to help to find the problem. I successfully can use 1.7.6 ver but...

e-alfred commented 4 years ago

To debug the Youtube download link extraction, please insert throw new \Exception( "\$Output = $Output" ); at line 61 in controller/lib/youtube.php and post the results from your Nextcloud log here to see if the link is extracted correctly.

Please also check the following settings to see if they match with yours:

image

MoucceuWildfire commented 4 years ago

Well my steps:

  1. Upgrade ocDownloader to 1.7.7
  2. Write the proper value in YouTube DL Audio Format bestaudio[abr<=75]
  3. Write in field YouTube DL Video Format best[width<=1280]
  4. App works well now.

Mentioned field was empty before. I also restarted my instance of Nextcloud and App still works fine. So problem was in empty field YouTube DL Audio Format and YouTube DL Video Format in "Additional settings" of Nextcloud. OcDownloader failed to fill this field with default value. Thanks for very nice app Alfred!

If you want me to help investigate the bug further, let me know how.

e-alfred commented 4 years ago

The values for youtube-dl weren't correctly set initially for local downloads, this explains why it worked with the API. Fixed by https://github.com/e-alfred/ocdownloader/commit/cb6ced0b7a70c8e8c3b3097fd1c1db2b556e7044.

NewRedsquare commented 4 years ago

Hello and sorry for re-opening this issue, i added the line to get debug output and here is what i get ( i have the Unable to retrieve true YouTube video URL problem ) :

{"reqId":"24wmOJ7J9VnKyfJWJ6SF","level":3,"time":"2020-06-09T16:40:57+00:00","remoteAddr":"***HIDDEN IP***","user":"***HIDDEN USER***","app":"index","method":"POST","url":"/apps/ocdownloader/ytdownloader/add","message":{"Exception":"Exception","Message":"$Output = ","Code":0,"Trace":[{"file":"/var/www/cloud/apps/ocdownloader/controller/ytdownloader.php","line":150,"function":"getVideoData","class":"OCA\\ocDownloader\\Controller\\Lib\\YouTube","type":"->","args":[]},{"file":"/var/www/cloud/lib/private/AppFramework/Http/Dispatcher.php","line":170,"function":"add","class":"OCA\\ocDownloader\\Controller\\YTDownloader","type":"->","args":[]},{"file":"/var/www/cloud/lib/private/AppFramework/Http/Dispatcher.php","line":100,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OCA\\ocDownloader\\Controller\\YTDownloader"},"Add"]},{"file":"/var/www/cloud/lib/private/AppFramework/App.php","line":137,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OCA\\ocDownloader\\Controller\\YTDownloader"},"Add"]},{"file":"/var/www/cloud/lib/private/AppFramework/Routing/RouteActionHandler.php","line":47,"function":"main","class":"OC\\AppFramework\\App","type":"::","args":["YTDownloaderController","Add",{"__class__":"OC\\AppFramework\\DependencyInjection\\DIContainer"},{"_route":"ocdownloader.YTDownloader.Add"}]},{"function":"__invoke","class":"OC\\AppFramework\\Routing\\RouteActionHandler","type":"->","args":[{"_route":"ocdownloader.YTDownloader.Add"}]},{"file":"/var/www/cloud/lib/private/Route/Router.php","line":297,"function":"call_user_func","args":[{"__class__":"OC\\AppFramework\\Routing\\RouteActionHandler"},{"_route":"ocdownloader.YTDownloader.Add"}]},{"file":"/var/www/cloud/lib/base.php","line":1007,"function":"match","class":"OC\\Route\\Router","type":"->","args":["/apps/ocdownloader/ytdownloader/add"]},{"file":"/var/www/cloud/index.php","line":37,"function":"handleRequest","class":"OC","type":"::","args":[]}],"File":"/var/www/cloud/apps/ocdownloader/controller/lib/youtube.php","Line":61,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.142 Safari/537.36","version":"19.0.0.12"}

So apparently no issue but i get an infinite loading while adding youtube video with webUI. With webextension i have the same Unable to retrieve true YouTube video URL.

My config : The last 1.7.8 version of Ocdownloader Ubuntu 18.04.4 LTS Mariadb Ver 15.1 Distrib 10.1.44-MariaDB Nextcloud 19.0.0.12

blob42 commented 4 years ago

Hi,

I have same issue as @NewRedsquare !

EDIT: Fixed the issue. The log was showing the wrong youtube-dl path /usr/bin/youtube-dl even though on the admin interface it looked good by default.

The fix:

config: ocDownloader 1.7.8 Nextcloud 19 official docker image

xspeed1989 commented 4 years ago

Hi,

I have same issue as @NewRedsquare !

EDIT: Fixed the issue. The log was showing the wrong youtube-dl path /usr/bin/youtube-dl even though on the admin interface it looked good by default.

The fix:

  • Delete the youtube-dl path field and click outside to save the setting
  • Refill the path again
  • Works !

config: ocDownloader 1.7.8 Nextcloud 19 official docker image

same as u but it doesn't work fine Unable to retrieve true YouTube video URL

thiccpad commented 3 years ago

Also have this issue on Ubuntu 20.04 Nextcloud 20.0.4 PHP 7.4.3 ocdownoader 1.7.10

Delete the youtube-dl path field and click outside to save the setting Refill the path again Works !

Didn't work. Also tried installing it with pip instead of apt-get and manually.

etyrnal commented 3 years ago

this app should be taken out of the Nextcloud store. There are SO MANY reports of this not working. It's clear that it's not mature, and is barely an alpha version.