tempdirectory option in config.php does not work

[urbenlegend]

I have OwnCloud 8.2.2 deployed on a Fedora 23 machine. By default, its /tmp is a tmpfs mount, which means that any OwnCloud file uploads larger than available system RAM will fail since it uses /tmp as a temporary upload directory. I've been trying to set the upload directory to something different via the tempdirectory option in config.php, but it doesn't seem to have an effect. All uploads still go to /tmp.

Steps to reproduce

1. Create tmp directory in owncloud folder. Set rw permissions to apache user.
2. Set 'tempdirectory' => '/var/www/html/owncloud/tmp' in config.php
3. Upload a file using Web GUI or WebDav. Notice that owncloud still uses /tmp as the temporary upload directory

   Expected behaviour

/var/www/html/owncloud/tmp should be used as the temporary upload directory instead for Web GUI and WebDav

Actual behaviour

Uploads go to /tmp and larger uploads will eventually fail due to lack of space in RAM.

Server configuration

Operating system: Fedora 23

Web server: Apache/2.4.18

Database: Maria DB 1:10.0.21-1.fc23

PHP version: PHP/5.6.17

ownCloud version: 8.2.2

Updated from an older ownCloud or fresh install: Updated from 8.1

List of activated apps:

Enabled:
  - activity: 2.1.3
  - calendar: 0.8.1
  - contacts: 0.5.0.0
  - documents: 0.11.0
  - files: 1.2.0
  - files_sharing: 0.7.0
  - files_texteditor: 2.0
  - files_trashbin: 0.7.0
  - files_versions: 1.1.0
  - files_videoviewer: 0.1.3
  - firstrunwizard: 1.1
  - galleryplus: 14.3.0
  - music: 0.3.8
  - provisioning_api: 0.3.0
  - templateeditor: 0.1
Disabled:
  - encryption
  - external
  - files_external
  - files_pdfviewer
  - gallery
  - notifications
  - user_external
  - user_ldap

The content of config/config.php:

{
    "system": {
        "instanceid": "ocpleicxwygq",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "benxiao.me"
        ],
        "datadirectory": "\/var\/www\/html\/owncloud\/data",
        "tempdirectory": "\/var\/www\/html\/owncloud\/tmp",
        "overwrite.cli.url": "https:\/\/benxiao.me\/owncloud",
        "dbtype": "mysql",
        "version": "8.2.2.2",
        "dbname": "owncloud_db",
        "dbhost": "localhost",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "mail_smtpmode": "smtp",
        "mail_from_address": "benxiao.me",
        "mail_domain": "gmail.com",
        "mail_smtpsecure": "tls",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpauth": 1,
        "mail_smtphost": "smtp.gmail.com",
        "mail_smtpport": "587",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "theme": "",
        "maintenance": false,
        "loglevel": 2,
        "appstore.experimental.enabled": false,
        "trashbin_retention_obligation": "auto"
    }
}

Are you using external storage, if yes which one: no

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: Chrome 48

Operating system: Fedora 23

Logs

Web server error log

Insert your webserver log here

ownCloud log (data/owncloud.log)

Insert your ownCloud log here

Browser log

Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log 
c) ...

[PVince81]

@icewind1991 @mmattel (you touched temp stuff recently IIRC, maybe you know)

[1Fopresta]

Hi, i have the same problem.

Server configuration

Operating system: Debian 8 Web server: Apache/2.4.10 Database: MySQL 5.5.46 PHP version: PHP/7.0.2 ownCloud version: 8.2.2 Updated from an older ownCloud or fresh install: fresh install 8.2.2

[PVince81]

since it uses /tmp as a temporary upload directory

This is not 100% correct. In a regular situation without any external storages, the chunk files are stored in "data/$user/cache/" and part files directly on the storage when uploading through Webdav.

The only situation where ownCloud explicitly will use the tmp storage is when uploading/download to/from external storages that don't support streaming (or where streaming wasn't implemented), in which case it first creates the file into tmp, then calls the storage's API and tells it to upload that file. (ex: GDrive)

There are two other situations where PHP might use the temporary storage:

• uploading through the web UI, which uses HTTP POST. PHP will first upload to a temporary location before giving control to ownCloud. In this case you might need to tweak your php.ini to make it store uploaded files somewhere else.
• if you use php-fpm, it uses the tmp folder for uploads even for HTTP PUT (Webdav), this is not controlled by ownCloud but by your env configuration. See https://github.com/owncloud/core/issues/9832#issuecomment-162837853

[urbenlegend]

if you use php-fpm, it uses the tmp folder for uploads even for HTTP PUT (Webdav), this is not controlled by ownCloud but by your env configuration.

I do not have php-fpm. In fact the php-fpm package is not even installed. I am also using a regular folder in the owncloud directory for data storage. However WebDav is still using /tmp.

[PVince81]

@urbenlegend can you post the file names of the temp files that you are seeing there ? Might help to find out which part of Apache/PHP/ownCloud is generating them.

[urbenlegend]

Here's one example upload: /tmp/systemd-private-cd35d18c66a1405abbb96bf4631c0292-httpd.service-5OyEA4/tmp/phpEAG5yY

[PVince81]

This looks like the temporary file that PHP would create when uploading with the web UI.

Was this a web UI upload or Webdav upload ? Webdav shouldn't go there. Web UI does because we use POST for uploads.

You could try changing the value of "upload_tmp_dir" in your php.ini.

Now thinking of this, maybe there's a way for ownCloud to set this value programmatically to make PHP use the configured temp folder.

[urbenlegend]

This was a WebDAV upload using a mount in Nautilus.

I thought about changing upload_tmp_dir but there's no way to set it on a per site basis. It applies to my entire Apache server which is hosting two WordPress blogs as well and I don't want to change tmp dir for those.

[PVince81]

Can you change it on one site just to confirm that it is indeed this problem ?

Setting this to enhancement to look whether it's possible to make ownCloud set this php.ini value programmatically.

[rullzer]

From what I found out you can't do this from within owncloud.

But https://mediatemple.net/community/products/dv/204644200/how-do-i-set-the-upload_tmp_dir has a solution to have a per vhost temp dir.

<Directory /path/to/vhosts/example.com/httpdocs>
php_admin_value upload_tmp_dir /tmp
</Directory>

That sounds like what you want.

[mmattel]

@RealRancor

The only situation where ownCloud explicitly will use the tmp storage is when uploading/download to/from external storages that don't support streaming (or where streaming wasn't implemented), in which case it first creates the file into tmp, then calls the storage's API and tells it to upload that file. (ex: GDrive)

There are two other situations where PHP might use the temporary storage:

uploading through the web UI, which uses HTTP POST. PHP will first upload to a temporary location before giving control to ownCloud. In this case you might need to tweak your php.ini to make it store uploaded files somewhere else. if you use php-fpm, it uses the tmp folder for uploads even for HTTP PUT (Webdav), this is not controlled by ownCloud but by your env configuration. See #9832 (comment)

From: https://github.com/owncloud/documentation/blob/master/admin_manual/configuration_server/config_sample_php_parameters.rst

Override where ownCloud stores temporary files. Useful in situations where the system temporary directory is on a limited space ramdisk or is otherwise restricted, or if external storages which do not support streaming are in use.

The Web server user must have write access to this directory.

We should adopt the wording in the documentation to be more clear and add the information from @rullzer above to complete. I have marked the corresponding text pieces from @PVince81 to think about bold.

What do you think?

[ghost]

@mmattel Sorry, don't have any insights about such PHP internals and how it is handling that stuff.

[urbenlegend]

@rullzer Your solution works great! That applies the upload_tmp_dir per directory. My wordpress and my owncloud are on the same vhost, but using that config option I can set owncloud's upload directory to be different from wordpress. I suggest we put this solution into the documentation as it works better than owncloud's tempdirectory setting.

[urbenlegend]

@mmattel I am definitely not using php-fpm and its still writing to the /tmp folder for WebDav file transfers.

[cdamken]

In the owncloud documentation is written that the php.ini file should have:

upload_tmp_dir = /var/big_temp_file/

Ref: https://doc.owncloud.com/server/8.2/admin_manual/configuration_files/big_file_upload_configuration.html?highlight=big%20files#configuring-php

However, if we have the parameter in the config.php then should work.

@MorrisJobke

00005219

[MorrisJobke]

However, if we have the parameter in the config.php then should work.

What do you want to tell us? Is it not working? What is not working? What should work?

[urbenlegend]

Yes, the configuration option basically doesn't work for a lot of use cases.

[utnalove]

I am having the same problem as described here. Nginx nginx/1.6.2 PHP 5.6.20-0+deb8u1 (cli) $OC_VersionString = '9.0.2';

The value 'tempdirectory' is ignored

[PVince81]

For web UI uploads, the POST method is used for which PHP will first use its own "upload_tmp_dir" setting to put the file there. And only then it will start the ownCloud code. There is nothing that ownCloud can do to prevent this, and it is also not possible for ownCloud to automatically replace this php.ini setting programmatically because the code runs too late in this workflow.

The issue with web UI POST will be resolved in the future when switching to PUT: https://github.com/owncloud/core/issues/4380

That said, I would like to know for which of you the wrong temporary folder is used for Webdav only (no web UI).

[PVince81]

Have you been able to adjust your PHP's "upload_tmp_dir" value ?

Or try again with master where PUT is used.

[PVince81]

please try with 10.0 beta which now uses Webdav PUT

[pako81]

this is still an issue in 10.0.3. Basically tempdirectory in config.php does not apply at all

[PVince81]

@pako81 any details ? network console report ? is it not using PUT when uploading ?

[pako81]

Yes, it is using PUT when uploading but still the tmp folder seems to be ignored:

[PVince81]

@pako81 where is it storing the files and how do these look like ? Are they related to the upload ?

Webdav PUT operations should not use any temporary folders as they are streamed directly.

One exception is when using certain types of external storages which might store the file to temp space before uploading to the final storage.

[pako81]

I am uploading via WebUI in my home folder. No external storage involved.

However, when using the "upload_tmp_dir" setting in php.ini I can see the the tmp folder is correctly used. So Webdav PUT operations seem to use temporary folders..

[PVince81]

Argh... Maybe you're using php-fpm and that one is configured to stream to temp space first ? (there were such cases before)

[PVince81]

the point is: ownCloud doesn't have any code that tells it to write anything to temp space when doing an upload to home storage. So there is nothing that ownCloud can do here as it's not ownCloud that is writing the file there but PHP or php-fpm, which means an ownCloud setting wouldn't be able to affect PHP/php-fpm.

[pako81]

You are absolutely right, I am using php-fpm. Sorry for the noise

[mmattel]

@PVince81 Question, sorry if stupid, but cant we use ini_set('upload_tmp_dir', tempdirectory); to define it? See https://github.com/owncloud/core/blob/f10d105ea2f2d0822693313ddcd6f9b2eaac6ebe/lib/private/TempManager.php#L209

In my php-fpm/php.ini file upload_tmp_dir is not set but described

; Temporary directory for HTTP uploaded files (will use system default if not specified).
; http://php.net/upload-tmp-dir
;upload_tmp_dir =

Maybe a note to be added to the documentation in the tempdirectory description if not possible? https://github.com/owncloud/documentation/blob/master/admin_manual/configuration/server/config_sample_php_parameters.rst

[PVince81]

@mmattel we can try but I think it will not work because it requires OC code to run first to be able to set that value. My understanding is that the value is only set for the current PHP request.

However during an upload with php-fpm, it will first put the file into temp space before running any OC script, and only then OC code will run.

@pako81 can you try with https://github.com/owncloud/core/pull/28937 ? I don't think it will work because of the above reasons.

[PVince81]

We can then update config.sample.php to mention the php-fpm scenarios and that the value cannot override it and needs to be adjusted by the admin.

[cdamken]

@pako81 can you try with #28937 ? I don't think it will work because of the above reasons.

I will try to do this, but probably the best recommendation is to change it directly where @mmattel described.

[mmattel]

I did some research, here is an update. It seems that ini_set('upload_tmp_dir', $tempDir) can´t be used because of

http://php.net/manual/en/ini.list.php

Name	Default	Changeable
upload_tmp_dir	NULL	PHP_INI_SYSTEM

and

http://php.net/manual/en/configuration.changes.modes.php

Mode	Meaning
PHP_INI_USER	Entry can be set in user scripts (like with ini_set()) or in the Windows registry. Since PHP 5.3, entry can be set in .user.ini
PHP_INI_PERDIR	Entry can be set in php.ini, .htaccess, httpd.conf or .user.ini (since PHP 5.3)
PHP_INI_SYSTEM	Entry can be set in php.ini or httpd.conf
PHP_INI_ALL	Entry can be set anywhere

What leads to the point that we can´t solve this programatically but have to properly document this.

In which conditions the ownCloud config.php parameter tempdirectory gets triggered/used. And in which conditions the php.ini upload_tmp_dir parameter gets triggered/used (php / php-fpm).

Hopefully that this helped.

[PVince81]

The config.php parameter is used by the TempManager which itself is used by code like external storage. Some external storages stored files to temp space before uploading to the real storage.

[PVince81]

it is also likely that LDAP stores stuff there, I remember seeing oc_tmp files lying around in my /tmp

[mmattel]

In which cases, where oC drives it, the TempManager (config.php parameter) is involved. And when is php.ini (indirect also via the OS) in charge of a temp dir of php. If we nail that down, we can derive proper description cases. Maybe as a first start to get the differenciation between php driven and oC driven tempdir.

[pako81]

I totally agree with the comment above. The tempdirectory in config.php does not apply at all when uploading to home storage, neither with php-fpm nor with mod_php.

Did I get it correctly that this config switch has only effect when uploading to certain type of external storages not having streaming implemented ?

If yes, then https://doc.owncloud.org/server/latest/admin_manual/configuration/server/config_sample_php_parameters.html#all-other-configuration-options needs a little of rework as the description seems to suggest this config switch can always overwrite the upload_tmp_dir setting in php.ini

[PVince81]

Did I get it correctly that this config switch has only effect when uploading to certain type of external storages not having streaming implemented ?

Yes, but it might also apply to LDAP when not using memcache and any third party app that might require a temp storage.

[pako81]

@PVince81 many thanks. So do you think is worth having our doc slightly modified to make this clearer ? /cc @settermjd

[PVince81]

We could add a note about the fact that "tempdirectory" does not affect uploads to home storage and that the php.ini value needs to be adjusted accordingly.