search

owncloud / core

:cloud: ownCloud web server core (Files, DAV, etc.)
https://owncloud.com
GNU Affero General Public License v3.0
8.4k stars 2.05k forks source link

Cannot download big files from google drive #38328

Open jvillafanez opened 3 years ago

jvillafanez commented 3 years ago
### Steps to reproduce 1. Using a standard ownCloud 10.6 setup (512 MB of memory for php) and having a 1GB+ file stored in gdrive 2. Connect ownCloud to gdrive 3. Download the 1GB file ### Expected behaviour File is downloaded ### Actual behaviour The whole file is stored into memory. The request crashes due to insufficient memory. ### Server configuration **Operating system**: ubuntu 20.04 **Web server:** apache 2.4 **Database:** mysql **PHP version:** 7.4 **ownCloud version:** 10.6.0 **Updated from an older ownCloud or fresh install:** fresh install **Where did you install ownCloud from:** **Signing status (ownCloud 9.0 and above):** ``` Login as admin user into your ownCloud and access http://example.com/index.php/settings/integrity/failed paste the results into https://gist.github.com/ and puth the link here. ``` **The content of config/config.php:** ``` Log in to the web-UI with an administrator account and click on 'admin' -> 'Generate Config Report' -> 'Download ownCloud config report' This report includes the config.php settings, the list of activated apps and other details in a well sanitized form. or If you have access to your command line run e.g.: sudo -u www-data php occ config:list system from within your ownCloud installation folder *ATTENTION:* Do not post your config.php file in public as is. Please use one of the above methods whenever possible. Both, the generated reports from the web-ui and from occ config:list consistently remove sensitive data. You still may want to review the report before sending. If done manually then it is critical for your own privacy to dilligently remove *all* host names, passwords, usernames, salts and other credentials before posting. You should assume that attackers find such information and will use them against your systems. ``` **List of activated apps:** ``` If you have access to your command line run e.g.: sudo -u www-data php occ app:list from within your ownCloud installation folder. ``` **Are you using external storage, if yes which one:** local/smb/sftp/... **Are you using encryption:** yes/no **Are you using an external user-backend, if yes which one:** LDAP/ActiveDirectory/Webdav/... #### LDAP configuration (delete this part if not used) ``` With access to your command line run e.g.: sudo -u www-data php occ ldap:show-config from within your ownCloud installation folder Without access to your command line download the data/owncloud.db to your local computer or access your SQL server remotely and run the select query: SELECT * FROM `oc_appconfig` WHERE `appid` = 'user_ldap'; Eventually replace sensitive data as the name/IP-address of your LDAP server or groups. ``` ### Client configuration **Browser:** **Operating system:** ### Logs #### Web server error log ``` owncloud_1 | [Fri Jan 22 08:37:30.371396 2021] [php7:error] [pid 238] [client 10.0.2.27:41634] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 1329476752 bytes) in /var/www/owncloud/apps/files_external/3rdparty/guzzlehttp/streams/src/Stream.php on line 135 ``` #### ownCloud log (data/owncloud.log) ``` Insert your ownCloud log here ``` #### Browser log ``` Insert your browser log here, this could for example include: a) The javascript console log b) The network log c) ... ```
jvillafanez commented 3 years ago

It seems the problem is more complex, and it's likely related to the google api client.

Array
(
    [0] => Array
        (
            [file] => /var/www/owncloud/apps/files_external/3rdparty/guzzlehttp/ringphp/src/Core.php
            [line] => 233
            [function] => __toString
            [class] => GuzzleHttp\Stream\Stream
            [type] => ->
        )

    [1] => Array
        (
            [file] => /var/www/owncloud/lib/composer/guzzlehttp/ringphp/src/Client/CurlFactory.php
            [line] => 237
            [function] => body
            [class] => GuzzleHttp\Ring\Core
            [type] => ::
        )

    [2] => Array
        (
            [file] => /var/www/owncloud/lib/composer/guzzlehttp/ringphp/src/Client/CurlFactory.php
            [line] => 203
            [function] => applyBody
            [class] => GuzzleHttp\Ring\Client\CurlFactory
            [type] => ->
        )

    [3] => Array
        (
            [file] => /var/www/owncloud/lib/composer/guzzlehttp/ringphp/src/Client/CurlFactory.php
            [line] => 30
            [function] => applyMethod
            [class] => GuzzleHttp\Ring\Client\CurlFactory
            [type] => ->
        )

    [4] => Array
        (
            [file] => /var/www/owncloud/lib/composer/guzzlehttp/ringphp/src/Client/CurlHandler.php
            [line] => 84
            [function] => __invoke
            [class] => GuzzleHttp\Ring\Client\CurlFactory
            [type] => ->
        )

    [5] => Array
        (
            [file] => /var/www/owncloud/lib/composer/guzzlehttp/ringphp/src/Client/CurlHandler.php
            [line] => 68
            [function] => _invokeAsArray
            [class] => GuzzleHttp\Ring\Client\CurlHandler
            [type] => ->
        )

    [6] => Array
        (
            [file] => /var/www/owncloud/lib/composer/guzzlehttp/ringphp/src/Client/Middleware.php
            [line] => 54
            [function] => __invoke
            [class] => GuzzleHttp\Ring\Client\CurlHandler
            [type] => ->
        )

    [7] => Array
        (
            [file] => /var/www/owncloud/lib/composer/guzzlehttp/ringphp/src/Client/Middleware.php
            [line] => 30
            [function] => GuzzleHttp\Ring\Client\{closure}
            [class] => GuzzleHttp\Ring\Client\Middleware
            [type] => ::
        )

    [8] => Array
        (
            [file] => /var/www/owncloud/lib/composer/guzzlehttp/guzzle/src/RequestFsm.php
            [line] => 129
            [function] => GuzzleHttp\Ring\Client\{closure}
            [class] => GuzzleHttp\Ring\Client\Middleware
            [type] => ::
        )

    [9] => Array
        (
            [file] => /var/www/owncloud/lib/composer/guzzlehttp/guzzle/src/Client.php
            [line] => 165
            [function] => __invoke
            [class] => GuzzleHttp\RequestFsm
            [type] => ->
        )

    [10] => Array
        (
            [file] => /var/www/owncloud/apps/files_external/3rdparty/google/auth/src/HttpHandler/Guzzle5HttpHandler.php
            [line] => 53
            [function] => send
            [class] => GuzzleHttp\Client
            [type] => ->
        )

    [11] => Array
        (
            [file] => /var/www/owncloud/apps/files_external/3rdparty/google/apiclient/src/Http/REST.php
            [line] => 83
            [function] => __invoke
            [class] => Google\Auth\HttpHandler\Guzzle5HttpHandler
            [type] => ->
        )

    [12] => Array
        (
            [function] => doExecute
            [class] => Google\Http\REST
            [type] => ::
        )

    [13] => Array
        (
            [file] => /var/www/owncloud/apps/files_external/3rdparty/google/apiclient/src/Task/Runner.php
            [line] => 181
            [function] => call_user_func_array
        )

    [14] => Array
        (
            [file] => /var/www/owncloud/apps/files_external/3rdparty/google/apiclient/src/Http/REST.php
            [line] => 66
            [function] => run
            [class] => Google\Task\Runner
            [type] => ->
        )

    [15] => Array
        (
            [file] => /var/www/owncloud/apps/files_external/3rdparty/google/apiclient/src/Client.php
            [line] => 898
            [function] => execute
            [class] => Google\Http\REST
            [type] => ::
        )

    [16] => Array
        (
            [file] => /var/www/owncloud/apps/files_external/3rdparty/google/apiclient/src/Service/Resource.php
            [line] => 238
            [function] => execute
            [class] => Google\Client
            [type] => ->
        )

    [17] => Array
        (
            [file] => /var/www/owncloud/apps/files_external/3rdparty/google/apiclient-services/src/Google/Service/Drive/Resource/Files.php
            [line] => 194
            [function] => call
            [class] => Google\Service\Resource
            [type] => ->
        )

    [18] => Array
        (
            [file] => /var/www/owncloud/apps/files_external/lib/Lib/Storage/Google.php
            [line] => 171
            [function] => get
            [class] => Google_Service_Drive_Resource_Files
            [type] => ->
        )

    [19] => Array
        (
            [file] => /var/www/owncloud/apps/files_external/lib/Lib/Storage/Google.php
            [line] => 178
            [function] => getInternalDriveFile
            [class] => OCA\Files_External\Lib\Storage\Google
            [type] => ->
        )

    [20] => Array
        (
            [file] => /var/www/owncloud/apps/files_external/lib/Lib/Storage/Google.php
            [line] => 152
            [function] => getInternalDriveFile
            [class] => OCA\Files_External\Lib\Storage\Google
            [type] => ->
        )

    [21] => Array
        (
            [file] => /var/www/owncloud/apps/files_external/lib/Lib/Storage/Google.php
            [line] => 450
            [function] => getDriveFile
            [class] => OCA\Files_External\Lib\Storage\Google
            [type] => ->
        )

    [22] => Array
        (
            [file] => /var/www/owncloud/lib/private/Files/Storage/Wrapper/Wrapper.php
            [line] => 229
            [function] => file_exists
            [class] => OCA\Files_External\Lib\Storage\Google
            [type] => ->
        )

    [23] => Array
        (
            [file] => /var/www/owncloud/lib/private/Files/Storage/Wrapper/Wrapper.php
            [line] => 229
            [function] => file_exists
            [class] => OC\Files\Storage\Wrapper\Wrapper
            [type] => ->
        )

    [24] => Array
        (
            [file] => /var/www/owncloud/lib/private/Files/Storage/Wrapper/Availability.php
            [line] => 240
            [function] => file_exists
            [class] => OC\Files\Storage\Wrapper\Wrapper
            [type] => ->
        )

    [25] => Array
        (
            [file] => /var/www/owncloud/lib/private/Files/Storage/Wrapper/Wrapper.php
            [line] => 229
            [function] => file_exists
            [class] => OC\Files\Storage\Wrapper\Availability
            [type] => ->
        )

    [26] => Array
        (
            [file] => /var/www/owncloud/lib/private/Files/Storage/Wrapper/Wrapper.php
            [line] => 229
            [function] => file_exists
            [class] => OC\Files\Storage\Wrapper\Wrapper
            [type] => ->
        )

    [27] => Array
        (
            [file] => /var/www/owncloud/lib/private/Files/Storage/Wrapper/Wrapper.php
            [line] => 229
            [function] => file_exists
            [class] => OC\Files\Storage\Wrapper\Wrapper
            [type] => ->
        )

    [28] => Array
        (
            [file] => /var/www/owncloud/lib/private/Files/View.php
            [line] => 1401
            [function] => file_exists
            [class] => OC\Files\Storage\Wrapper\Wrapper
            [type] => ->
        )

    [29] => Array
        (
            [file] => /var/www/owncloud/lib/private/Files/View.php
            [line] => 1447
            [function] => getCacheEntry
            [class] => OC\Files\View
            [type] => ->
        )

    [30] => Array
        (
            [file] => /var/www/owncloud/apps/dav/lib/Connector/Sabre/ObjectTree.php
            [line] => 179
            [function] => getFileInfo
            [class] => OC\Files\View
            [type] => ->
        )

    [31] => Array
        (
            [file] => /var/www/owncloud/apps/dav/lib/Tree.php
            [line] => 70
            [function] => getNodeForPath
            [class] => OCA\DAV\Connector\Sabre\ObjectTree
            [type] => ->
        )

    [32] => Array
        (
            [file] => /var/www/owncloud/lib/composer/sabre/dav/lib/DAV/Server.php
            [line] => 521
            [function] => getNodeForPath
            [class] => OCA\DAV\Tree
            [type] => ->
        )

    [33] => Array
        (
            [file] => /var/www/owncloud/apps/dav/lib/Connector/Sabre/CorsPlugin.php
            [line] => 79
            [function] => getAllowedMethods
            [class] => Sabre\DAV\Server
            [type] => ->
        )

    [34] => Array
        (
            [file] => /var/www/owncloud/apps/dav/lib/Connector/Sabre/CorsPlugin.php
            [line] => 141
            [function] => getExtraHeaders
            [class] => OCA\DAV\Connector\Sabre\CorsPlugin
            [type] => ->
        )

    [35] => Array
        (
            [file] => /var/www/owncloud/lib/composer/sabre/event/lib/WildcardEmitterTrait.php
            [line] => 89
            [function] => setCorsHeaders
            [class] => OCA\DAV\Connector\Sabre\CorsPlugin
            [type] => ->
        )

    [36] => Array
        (
            [file] => /var/www/owncloud/lib/composer/sabre/dav/lib/DAV/Server.php
            [line] => 456
            [function] => emit
            [class] => Sabre\DAV\Server
            [type] => ->
        )

    [37] => Array
        (
            [file] => /var/www/owncloud/lib/composer/sabre/dav/lib/DAV/Server.php
            [line] => 253
            [function] => invokeMethod
            [class] => Sabre\DAV\Server
            [type] => ->
        )

    [38] => Array
        (
            [file] => /var/www/owncloud/apps/dav/lib/Server.php
            [line] => 330
            [function] => start
            [class] => Sabre\DAV\Server
            [type] => ->
        )

    [39] => Array
        (
            [file] => /var/www/owncloud/apps/dav/appinfo/v2/remote.php
            [line] => 31
            [function] => exec
            [class] => OCA\DAV\Server
            [type] => ->
        )

    [40] => Array
        (
            [file] => /var/www/owncloud/remote.php
            [line] => 165
            [args] => Array
                (
                    [0] => /var/www/owncloud/apps/dav/appinfo/v2/remote.php
                )

            [function] => require_once
        )

)

The debug trace shows that the problem is in https://github.com/owncloud/core/blob/master/apps/files_external/lib/Lib/Storage/Google.php#L530 , requesting the file, but not directly getting the body response. The __toString() method that end up being called is what brings the whole file into memory.

So far, I haven't seen an option in the google api client to prevent this from happening.

mmattel commented 3 years ago

Q1: is there an updated version of the GD API available which does not have this issue? Q2: in case Q1 is "false", would´nt it be good to report this to Google and ask for a fix?