search

owncloud / core

:cloud: ownCloud web server core (Files, DAV, etc.)
https://owncloud.com
GNU Affero General Public License v3.0
8.37k stars 2.06k forks source link

[External storage - Dropbox] Allowed memory size exhausted #4040

Closed enoch85 closed 8 years ago

enoch85 commented 11 years ago

I don´t know why I get this messages but external storage is working really bad for me. Should I allow a certain OAuth adress in the app i created in Dropbox, or what seems to be the error?

Fatal    PHP     Allowed memory size of 536870912 bytes exhausted (tried to allocate 408230542 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   12.07.2013 13:29:22
Fatal    PHP     Allowed memory size of 536870912 bytes exhausted (tried to allocate 408230542 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   10.07.2013 10:45:28
Fatal    PHP     Call to a member function t() on a non-object at /var/www/owncloud/apps/files_mv/ajax/autocompletedir.php#48    09.07.2013 23:24:33
Fatal   PHP Call to a member function t() on a non-object at /var/www/owncloud/apps/files_mv/ajax/autocompletedir.php#48    09.07.2013 23:24
Fatal   PHP Call to a member function t() on a non-object at /var/www/owncloud/apps/files_mv/ajax/autocompletedir.php#48    09.07.2013 23:24

I can also add that I have 22 GB on my Dropbox as we speak, and I´m not able to download the content from either WebDAV or OC web interferance. In OC web interferance I get "server error message". In WebDAV it loads in like 10 kB/s, thou I have a 100 MBit/s internet.

OS: Windows 7 Ultimate Server: Ubuntu Server 12.04 LTS OC: 5.0.8 WebDAV client: Netdrive

j-ed commented 11 years ago

The given PHP error refers to the amount of RAM which can/cannot be allocated by PHP on your PC and not to the available storage space on your Dropbox share. See https://drupal.org/node/76156 for further information. I think it's more important to clarify why 512M of RAM is not sufficient to work with and why it claims to request additional 391M.

karlitschek commented 11 years ago

agreed. This amount of ram should not be necessary. Something is strange here.

enoch85 commented 11 years ago

I changed the RAM to 3 GB instead of 2 GB as I had before, and I still get this:

Fatal    PHP     Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:47:22
Fatal    PHP     Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:45:30
Fatal    PHP     Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:43:18
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:43
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:41
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:39
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:37
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:35
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:33
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:31
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:29
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:27
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:25
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:23
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:21
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:19
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:17
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:15
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:13
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:11
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:09
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:07
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:05
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:03
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 14:01
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 13:59
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 13:58
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 13:56
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 13:53
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 13:51
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 13:49
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 13:47
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 13:45
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 13:43
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 13:41
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 13:39
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 13:37
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 13:35
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 13:33
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 13:31
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 13:29
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 13:27
Fatal   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 527961378 bytes) at /var/www/owncloud/3rdparty/Dropbox/OAuth/Curl.php#102   20.07.2013 13:25

Now I´m using ownCloud 5.0.9

enoch85 commented 11 years ago

Is this interesting maybe? http://forum.owncloud.org/viewtopic.php?f=23&t=13578

Is it possible to raise the memory_limit from a owncloud point of view directly, or is this a server configuration?

enoch85 commented 11 years ago

The row that it refers to is this: $response=curl_exec($ch);

This shows my php.ini file:

;;;;;;;;;;;;;;;;;;;
; Resource Limits ;
;;;;;;;;;;;;;;;;;;;

; Maximum execution time of each script, in seconds
; http://php.net/max-execution-time
; Note: This directive is hardcoded to 0 for the CLI SAPI
max_execution_time = 30

; Maximum amount of time each script may spend parsing request data. It's a good
; idea to limit this time on productions servers in order to eliminate unexpectedly
; long running scripts.
; Note: This directive is hardcoded to -1 for the CLI SAPI
; Default Value: -1 (Unlimited)
; Development Value: 60 (60 seconds)
; Production Value: 60 (60 seconds)
; http://php.net/max-input-time
max_input_time = 60

; Maximum execution time of each script, in seconds
; http://php.net/max-execution-time
; Note: This directive is hardcoded to 0 for the CLI SAPI
max_execution_time = 30

; Maximum amount of time each script may spend parsing request data. It's a good
; idea to limit this time on productions servers in order to eliminate unexpectedly
; long running scripts.
; Note: This directive is hardcoded to -1 for the CLI SAPI
; Default Value: -1 (Unlimited)
; Development Value: 60 (60 seconds)
; Production Value: 60 (60 seconds)
; http://php.net/max-input-time
max_input_time = 60

; Maximum input variable nesting level
; http://php.net/max-input-nesting-level
;max_input_nesting_level = 64

; How many GET/POST/COOKIE input variables may be accepted
; max_input_vars = 1000

; Maximum amount of memory a script may consume (128MB)
; http://php.net/memory-limit
memory_limit = -1

-------------------------------

I hope it helps. :)

enoch85 commented 11 years ago

And still, in 5.0.10 i get this:

Fatal    PHP     Allowed memory size of 536870912 bytes exhausted (tried to allocate 18446744071754983150 bytes) at /var/www/owncloud/apps/files_encryption/lib/crypt.php#233    2013-08-18T00:34:09+00:00
Fatal    PHP     Allowed memory size of 536870912 bytes exhausted (tried to allocate 18446744071754983150 bytes) at /var/www/owncloud/apps/files_encryption/lib/crypt.php#233    2013-08-18T00:34:15+00:00
Fatal    PHP     Allowed memory size of 536870912 bytes exhausted (tried to allocate 18446744071754983150 bytes) at /var/www/owncloud/apps/files_encryption/lib/crypt.php#233    2013-08-18T00:34:21+00:00
enoch85 commented 11 years ago

@MTGap @schiesbn ?

enoch85 commented 11 years ago

Just found the row that the error points to:

(Fatal PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 18446744071754983150 bytes) at /var/www/owncloud/apps/files_encryption/lib/crypt.php#233 2013-08-18T00:34:21+00:00)
                if ($plainContent = openssl_decrypt($encryptedContent, 'AES-128-CFB', $passphrase, false, $iv)) {

                        return $plainContent;

Maybe it helps. :)

enoch85 commented 11 years ago

[SOLVED]

I changed the /etc/php5/apache2/php.ini: memory_limit = 1450MB upload_max_filesize = 1000MB post_max_size = 1200MB max_input_time = 260 (seconds) max_execution_time = 130 (seconds)

And i hashed out the settings in .htaccess referring to those lines in php.ini

php_value:

php_value:

php_value:

I don't know if this will solve it for good. But it works atm. I still think that you should have a look at this issue because i've seen this issue a lot when I googled the problem.

karlitschek commented 10 years ago

We think this should be fixed in 5.0.13. Can you test please?

enoch85 commented 10 years ago

I haven´t seen it for a while now, so I think it´s gone. I´ll reopen this if it´s still an issue in the future. Thank you.

margartiaville commented 10 years ago

@enoch85 You saved my life. Thank you.

enoch85 commented 10 years ago

Haha, glad to help! :) @margartiaville

@karlitschek I´ve still the same error in OC 6.0.0a when i´m trying to sync files with the 1.5.0 client without changing any settings in .htaccess:

Error   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 370420957 bytes) at /var/www/owncloud/apps/files_external/3rdparty/Dropbox/OAuth/Curl.php#102   2013-12-19T16:40:09+00:00
Error   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 423624705 bytes) at /var/www/owncloud/apps/files_external/3rdparty/Dropbox/OAuth/Curl.php#102   2013-12-19T16:37:03+00:00
Error   files_external  Curl error: (18) transfer closed with 202551996 bytes remaining to read     2013-12-19T16:10:37+00:00
Error   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 353714595 bytes) at /var/www/owncloud/apps/files_external/3rdparty/Dropbox/OAuth/Curl.php#102   2013-12-19T15:50:17+00:00
Error   PHP Allowed memory size of 536870912 bytes exhausted (tried to allocate 521669907 bytes) at /var/www/owncloud/apps/files_external/3rdparty/Dropbox/OAuth/Curl.php#102

So I´m sorry to say that the problem isn´t fixed yet.

Jas2Ma commented 9 years ago

I've the error, too.

2014-12-10 22:05:52: (mod_fastcgi.c.2676) FastCGI-stderr: PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 71677512 bytes) in /media/USB/owncloud/apps/files_external/3rdparty/Dropbox/OAuth/Curl.php on line 102

It is happening, when the client is trying to download a relatively large file (68,36MB). [71677512 bytes = 68,36MB] So I think the script is trying to download the file to allocatable ram of the php process, which I think is a bad idea. Is there a possibility to download large files to tmp directory?

Client Version: 1.7.0 Server Version: 7.0.3 (Webserver is lighttpd)

enoch85 commented 9 years ago

@PVince81 ?

arkascha commented 9 years ago

Looks like this is still not fixed in OC-7.0.4?

MorrisJobke commented 9 years ago

https://github.com/owncloud/core/issues/13874#issuecomment-72836055

PVince81 commented 9 years ago

@enoch85 I discovered that the old Dropbox library used by ownCloud doesn't stream files. When downloading, it puts the whole file in memory. This will break with bigger files.

We should switch to a better library, maybe the official Dropbox lib or use Flysystem's Dropbox plugin.

Let's keep this ticket to track the library update.

enoch85 commented 9 years ago

@PVince81 I´m really glad you found the reason here. Finally we can fix it. Thanks!

PVince81 commented 9 years ago

We're past feature freeze, will not update the dropbox lib at this point => 9.0. Ideally we need a new better lib, maybe the official one from Dropbox, which might need rewriting the storage backend class.

PVince81 commented 8 years ago

Something to look at: whether Flysystem's Dropbox library is properly managing memory. If yes, then might be easier to switch to using Flysystem's Dropbox impl.

CC @Xenopathic if you're looking for a challenge :wink:

PVince81 commented 8 years ago

Or use a direct URL like it was done in https://github.com/owncloud/core/pull/21732

PVince81 commented 8 years ago

Fix for download here: https://github.com/owncloud/core/pull/23272

PVince81 commented 8 years ago

I had a look at the library and from what I see it doesn't store the file in memory on upload, it even uses chunking when possible.

PVince81 commented 8 years ago

Reopening as there were issues when testing downloading big files (timeouts).

Alternative approach with stream download: https://github.com/owncloud/core/pull/23516

PVince81 commented 8 years ago

This already made it into 9.0.1: https://github.com/owncloud/core/pull/23522

Closing

lock[bot] commented 5 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.