search

owncloud / core

:cloud: ownCloud web server core (Files, DAV, etc.)
https://owncloud.com
GNU Affero General Public License v3.0
8.37k stars 2.06k forks source link

Make cli version of PHP used by Updater configurable #25973

Closed timreeves closed 8 years ago

timreeves commented 8 years ago

Steps to reproduce

  1. Install (or update to) 9.0.4 OR 9.1.0 on a server running Ubuntu 12.04 LTS OR 14.04 LTS
  2. It must also be running under Plesk 12.5 with alternative PHP version 7.0
  3. The GUI is powered by Nginx + PHP-FPM, using the Plesk-PHP 7.0.9
  4. Call Updater from the Web Admin page

    Expected behaviour

The Updater should work, and use the PHP 7.0.9, which is available on the host

Actual behaviour

The Updater says: [UnexpectedValueException] Could not parse a response for config:list. Please check if the current shell user can run occ command. Raw output: Not allowed

So it looks to me as if the Updater is using the standard OS-PHP, which no doubt cannot understand the PHP of OC.

My wish/suggestion is: To be able to add a config statement re the version of PHP to use for executing external PHP commands. No doubt you know that here in Germany many Hosters like HostEurope, Strato, Hetzner etc. offer Plesk, so it would be great to have this option. Otherwise, I have to tell my clients that they need a new server - something they don't want to hear )-: UPDATE: Turns out, on later trials with Ubuntu 14.04, that PHP 5.3 was NOT the problem.

Server configuration

Ubuntu 12.04 OR 14.04 LTS

Web server: Nginx (from Plesk - 1.11.1)

Database: MySQL

PHP version: OS = 5.3.10 / 5.5.9, Plesk = 7.0.9 (at /opt/plesk/php/7.0/bin/php)

ownCloud version: 9.0.4 OR 9.1.0

Updated from an older ownCloud or fresh install: Updated

Where did you install ownCloud from: From Source-Zip, via CLI

Signing status (ownCloud 9.0 and above): No errors have been found.

List of activated apps: Enabled:

Disabled:

The content of config/config.php: { "system": { "instanceid": "oc452bf87cea", "passwordsalt": "_REMOVED SENSITIVE VALUE_", "trusteddomains": [ "owncloud.biblische-reisen.de" ], "datadirectory": "\/var\/www\/vhosts\/biblische-reisen.de\/oc-data", "dbtype": "mysql", "version": "9.0.4.1", "dbname": "owncloudbde", "dbhost": "localhost", "dbtableprefix": "oc", "dbuser": "_REMOVED SENSITIVE VALUE_", "dbpassword": "_REMOVED SENSITIVE VALUE_", "installed": true, "loglevel": 4, "forcessl": true, "theme": "", "maintenance": false, "mail_from_address": "owncloud", "mail_domain": "biblische-reisen.de", "mail_smtpmode": "php", "mail_smtpname": "_REMOVED SENSITIVE VALUE_", "mail_smtppassword": "_REMOVED SENSITIVE VALUE_", "default_language": "en", "secret": "_REMOVED SENSITIVE VALUE_", "log_rotate_size": 10485760, "check_for_working_wellknown_setup": false, "check_for_working_htaccess": false, "memcache.local": "\OC\Memcache\APCu", "filelocking.enabled": "true", "memcache.locking": "\OC\Memcache\Redis", "redis": { "host": "\/var\/run\/redis\/redis.sock", "port": 0, "timeout": 0, "dbindex": 1 }, "trashbin_retention_obligation": "auto", "updater.secret": "_REMOVED SENSITIVE VALUE_" } }

Are you using external storage No

Are you using encryption: No

Are you using an external user-backend, if yes which one: No

Client configuration

Browser: Firefox

Operating system: Windows 10

Logs

Web server error log

Insert your webserver log here
Log is empty.

#### ownCloud log (data/owncloud.log)
Log is empty.

#### Browser log
17:55:21.680 Content Security Policy: Die Direktive 'frame-src'  sollte nicht mehr verwendet werden. Bitte verwenden Sie stattdessen die Direktive 'child-src'.(unbekannt)
17:55:23.842 Content Security Policy: Die Einstellungen der Seite haben das Laden einer Ressource auf self blockiert ("script-src https://owncloud.biblische-reisen.de 'unsafe-eval'").admin
DeepDiver1975 commented 8 years ago

The Updater says: [UnexpectedValueException] Could not parse a response for config:list. Please check if the current shell user can run occ command. Raw output: Not allowed

@VicDeo THX

VicDeo commented 8 years ago

@timreeves Could you please adjust your nginx configuration according to https://github.com/owncloud/documentation/pull/2551 and recheck?

ghost commented 8 years ago

The nginx config doesn't need to be updated here. The linked issue would cause the updater app to shown but with missing styles / graphics. Its more about this:

Raw output: Not allowed

where the shared hoster is limited some methods or similar needed for the updater app.

timreeves commented 8 years ago

I agree with @RealRancor , the "Not allowed" is the real clue. I already know the thread about updater CSS/JS, and added my own comment to the original thread Resync nginx config. My Nginx config is good, the updater screen is shown with all resources. It just does'nt run.

Neither am I at a shared hoster - it's an own VPS at HostEurope, managed with Plesk. And both Plesk and all system packages are fully up to date. I'm using HTTP/2, so tried a fallback to HTTP/1.1, but the problem remained unchanged.

So I come back to my original thought: Which version of PHP is the Updater using? Can that be configured anywhere, if not officially then in some arcane manner?

ghost commented 8 years ago

I think that question is not relevant for your "not allowed" issue. Furthermore this looks more like a configuration issue (blocked HTTP verbs or other restrictions) which are better discussed at https://central.owncloud.org

timreeves commented 8 years ago

I have now checked this on OC 9.1.0 AND on an Ubuntu 14.04 LTS server (VPS), there using a self-compiled PHP 7.0.5 / 7.0.10 and not the Plesk one. Always the same - Not allowed. So it seems like a bug to me - or something in my environment which I have (mis)configured the same on both maschines (e.g. Nginx-Config). If anyone can tell me what/how to investigate / debug, I'm game to have a go - but at the moment I don't have a clue where or what I should be looking for!

ghost commented 8 years ago
  1. Open updater in your browser
  2. Open Web Developer / Firebug in Firefox
  3. Open network console of web developer / firefox
  4. Click update in updater

One of the requests going to your server is very likely coming back with an "405 Method not allowed" or similar.

timreeves commented 8 years ago

Thanks for the instructions! My FF-Console says:

11:10:44.548 Content Security Policy: Die Direktive 'frame-src' sollte nicht mehr verwendet werden. Bitte verwenden Sie stattdessen die Direktive 'child-src'.(unbekannt) 11:11:13.987 Content Security Policy: Die Einstellungen der Seite haben das Laden einer Ressource auf self blockiert ("script-src https://owncloud.reeves.one 'unsafe-eval'").admin

This is my Nginx-Config, which passes PHP to a self-compiled PHP 7.0.10 via PHP-FPM. Sorry about so much comment - I'm 62 and my memory was never the best and aint getting better, I simply need the comments:

Put this file in the Plesk webspace root, NOT the document root of ownCloud. Why?

1) that way it's update-safe; and (2) ownCloud code-signing would complain.

Plesk: Websites & Domains | {Your OC Domain} | Apache & nginx Settings:

All defaults, ALL checkboxes off, Additional nginx directives:

include "/var/www/vhosts/{plesk-webspace}/nginx-for-own-phpfpm-owncloud9.conf";

set $sockname ##YOUR-POOL##;

Plesk Web Server settings:

PHP support: Plesk-PHP 7.0.X as FPM application server by nginx

Smart static files processing = Nein

Serve static files directly by nginx = Nein

After each modification: service nginx restart

Nginx Server Error log: /var/log/nginx/error.log

Nginx Domain Error log: /var/www/vhosts/system/{domain}/logs/error_log

FPM-Log: /var/log/php5-fpm.log bzw. /var/log/plesk-php70-fpm/error.log

http://wiki.nginx.org/Pitfalls

http://www.php.net/manual/de/regexp.introduction.php

The following directives are included at the end of each Server-Block, i.e. in "server { ... }" context.

This config is a merge of my previous effort and this:

"OCNX": https://doc.owncloud.org/server/9.1/admin_manual/installation/nginx_owncloud_9x.html

Plesk's "location /" sends the resource request to Apache via Ports 7080/7081

Plesk's "location /" is the ONLY prefix string location in the Plesk config,

which means it's always the winning prefix location, and gets noted as such.

- UNLESS we add any longer prefix string locations...

Implying that if we want to use Nginx for everything, then we must add

"location" statements matching ALL resource types we expect, with "try_files" or "rewrite".

Security via e.g. "try_files" is NOT an issue at "location /", that's then Apache's job.

Force SSL (if you use http, then comment out)

if ($server_port = 80) { rewrite ^ https://$server_name$request_uri? permanent; }

Adds the specified charset to the “Content-Type” response header field.

Context: http, server, location, if in location

charset utf-8;

Add security related headers

Before enabling Strict-Transport-Security headers please read into this topic first

add_header Strict-Transport-Security "max-age=15768000; preload;"; add_header X-Content-Type-Options nosniff; add_header X-Frame-Options "SAMEORIGIN"; add_header X-XSS-Protection "1; mode=block"; add_header X-Robots-Tag none; add_header X-Download-Options noopen; add_header X-Permitted-Cross-Domain-Policies none;

set max upload size

Nginx says client_max_body_size already set - I only see it in plesk.conf.d/webmail.conf

fastcgi_buffers 64 4K;

location = / { index index.php; } error_page 403 /core/templates/403.php; error_page 404 /core/templates/404.php;

error_log /var/log/nginx/error.log debug;

location = /robots.txt { allow all; log_not_found off; access_log off; }

Then comes regex processing, * which terminates on the first match *, and the

corresponding location configuration is used. If no match with a regular expression

is found, then the configuration of the prefix location remembered earlier is used.

SO the order in which regex locations appear is IMPORTANT!

(?: ... ) non-capturing, siehe https://de.wikipedia.org/wiki/Regul%C3%A4rer_Ausdruck

ownCloud ab 8 nutzt PATH-INFO to .js and .php, e.g. /index.php/core/js/oc.js,

/index.php/core/ajax/share.php, /index.php/apps/files/ajax/list.php, /index.php/avatar/TimReeves/128

See: https://doc.owncloud.org/server/9.1/admin_manual/installation/nginx_owncloud_9x.html

location = /.well-known/carddav { return 301 $scheme://$host/remote.php/dav; } location = /.well-known/caldav { return 301 $scheme://$host/remote.php/dav; } location /.well-known/acme-challenge { }

OCNX has a "location /" - which we cannot, as Plesk has already used it, to pass thru to Apache

The OCNX version - location / { rewrite ^ /index.php$uri; } - means:

rewrite the URI, appending it (as PATH_INFO) to /index.php; and

it ONLY happens as a final fallback, being a prefix string.

It must be replaced by a regex - at the END, after all other location directives have had their chance.

The Apache .htaccess provided by OC also does: RewriteRule ^remote/(.*) remote.php [QSA,L]

But not needed, see here: https://github.com/owncloud/documentation/issues/2014

These catch the worst flagrant attempts to call disallowed locations from the web

location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ { deny all; return 404; } location ~ ^/(?:.|autotest|occ|issue|indie|db_|console) { deny all; return 404; }

This is careful: It matches all the EXPECTED ".php" calls and passes them to PHP-FPM

There is NO pendant in the Apache .htaccess

location ~ ^/(?:index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|ocs-provider/.+|core/templates/40[34]|apc|info|ocp).php(?:$|/) {

See http://wiki.nginx.org/HttpFastcgiModule\#fastcgi_split_path_info

Directive populates $fastcgi_script_name + $fastcgi_path_info

BUT there's a problem: http://trac.nginx.org/nginx/ticket/321

fastcgi_split_path_info ^((?U).+.php)(/?.+)$; set $path_info $fastcgi_path_info;

Requested PHP Scripts MUST exist exactly as named

try_files $fastcgi_script_name =404;

try_files has emptied $fastcgi_path_info

fastcgi_param PATH_INFO $path_info;

You can add any application-special headers to $_SERVER here

fastcgi_param PATH "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"; fastcgi_pass "unix:/usr/local/$sockname.sock";

Following include sets SCRIPT_FILENAME to $document_root$fastcgi_script_name

Meaning that $fastcgi_script_name MUST be correct now!

include /etc/nginx/fastcgi.conf;

Avoid sending the security headers twice

fastcgi_param modHeadersAvailable true;

This and the above are both ownCloud-specific

fastcgi_param front_controller_active true;

http://nginx.org/en/docs/http/ngx_http_fastcgi_module.html\#fastcgi_intercept_errors

fastcgi_intercept_errors on;

http://nginx.org/en/docs/http/ngx_http_fastcgi_module.html\#fastcgi_request_buffering

fastcgi_request_buffering off;

This is the one branch we DO want to log

access_log off;

}

Beginning "updater" or "ocs-provider" and ending with nothing more, or continuing with "/" (and potentially more behind it)

In fact, clicking on "Open updater" requests "/updater/", which matches this location, which is why we need the "index" -

the index addition goes into a new parsing round and then matches the location above, for ".php" files.

The HTML source file returned actually contains:

which do NOT match the ".php" location above, thus are handled here

~ = regex case-sensitive

location ~ ^/(?:updater|ocs-provider)(?:$|/) { index index.php; try_files $uri $uri/ =404;

try_files: If none of the files were found, an internal redirect to the uri specified in the last parameter is made.

}

Adding the cache control header for js and css files

Make sure it is BELOW the PHP block

~* = regex case-insensitive

location ~* .(?:css|js)$ { try_files $uri /index.php$uri$is_args$args; add_header Cache-Control "public, max-age=7200";

Add headers to serve security related headers (It is intended to duplicate the ones above). Nginx allows several add_header directives.

These directives are inherited from the previous level if and only if there are no add_header directives defined on the current level.

add_header Strict-Transport-Security "max-age=15768000; preload;"; add_header X-Content-Type-Options nosniff; add_header X-Frame-Options "SAMEORIGIN"; add_header X-XSS-Protection "1; mode=block"; add_header X-Robots-Tag none; add_header X-Download-Options noopen; add_header X-Permitted-Cross-Domain-Policies none; access_log off; }

location ~* .(?:svg|gif|png|html|ttf|woff|ico|jpg|jpeg)$ { try_files $uri /index.php$uri$is_args$args; access_log off; }

location ~ ^/core/doc/[^\/]+/$ { rewrite ^(/core/doc/[^\/]+/)$ $1/index.html; access_log off; }

location = /fpmstatus { fastcgi_pass "unix:/usr/local/$sockname.sock"; include /etc/nginx/fastcgi.conf; } location = /fpmping { fastcgi_pass "unix:/usr/local/$sockname.sock"; include /etc/nginx/fastcgi.conf; }

Nothing should arrive here EXCEPT the things needing redirecting to be handled by index.php

location ~ ^/.+ { rewrite ^ /index.php$uri; access_log off; }

Finally, switch gzipping on

Seems fixed since OC8.1 - https://github.com/owncloud/core/pull/15227, https://github.com/owncloud/core/issues/9005

Background: https://github.com/owncloud/client/wiki/Etags-and-file-ids

Client + Server: https://github.com/owncloud/client/issues/3946

gzip on; gzip_proxied any; gzip_min_length 100; gzip_buffers 8 16k; # number size, default 32 4k|16 8k gzip_types text/css text/plain text/javascript application/javascript application/json application/x-javascript application/xml application/xml+rss application/xhtml+xml application/x-font-ttf application/x-font-opentype application/vnd.ms-fontobject image/svg+xml image/x-icon application/rss+xml application/atom_xml; gzip_vary on; gzip_comp_level 9; gzip_http_version 1.0; gzip_disable "MSIE [1-6].(?!.*SV1)";

VicDeo commented 8 years ago

@timreeves your config doesn't work for me at all unless I change

location ~ ^/(?:.|autotest|occ|issue|indie|db_|console) { deny all; return 404; }

to

location ~ ^/(?:autotest|occ|issue|indie|db_|console) { deny all; return 404; }

But this is not the case IMO.

Can you grep nginx access.log and error.log for requests to /index.php/occ?

User agent for these requests will be Guzzle/5.3.0

timreeves commented 8 years ago

Many thanks @VicDeo for taking the trouble to install my nginx config! I can't see why removing the ".|" from the start of the deny location should make a difference - it matches a period as the first char of the URI and is there on purpose, to prevent access e.g. to .htaccess and .user.ini - and it works fine for me. And the rewrite to add index.php means we get prettier URLs than with the Apache .htaccess.

In the meantime I'm going nuts, having noticed that although the GUI works without complaints (except for the Updater problem), the file sync via Webdav coming from Mirall (Windows) causes error messages like this in the OC log:

{"reqId":"WU+ZpbyU3VOT44yqW3MN","remoteAddr":"188.192.17.159","app":"PHP","message":"realpath(): open_basedir restriction in effect. File(/) is not within the allowed path(s): (/var/www/vhosts/reeves.one/:/tmp/:/dev/urandom) at /var/www/vhosts/reeves.one/owncloud.reeves.one/lib/private/Files/Storage/Local.php#56","level":3,"time":"2016-08-31T14:13:28+00:00","method":"PUT","url":"/remote.php/webdav/Zed/Server/2015-2016-Server.odt","user":"TimReeves"}

However, these log messages seem to be "false positives" - the file uploads arrive on the OC server. I have noted this here.

I have tried OC 9.1.0 with

Always the same result:

Can't see anything in the nginx error log that looks relevant to the Updater problem (there are some entries, but more to do with all my switching environments while testing).

grep occ access_ssl_log 176.28.16.171 - - [01/Sep/2016:22:06:38 +0200] "POST /index.php/occ/config:list HTTP/1.0" 200 1307 "-" "Guzzle/5.3.0 curl/7.35.0 PHP/7.0.9" 176.28.16.171 - - [01/Sep/2016:22:35:10 +0200] "POST /index.php/occ/config:list HTTP/1.0" 200 1335 "-" "Guzzle/5.3.0 curl/7.35.0 PHP/5.6.24"

So that shows it is happening and formally succeeding - the error is noted in the JSON returned, so the problem seems to be in the PHP code, not in the client-server communication.

Any help much appreciated - I really don't want to fall back to 9.0.4, inasmuch as a number of my data files have changed since then and the backup DB is outdated. Thanks!

VicDeo commented 8 years ago

@timreeves I might be wrong but a dot matches ANY character and \. matches a dot so location ~ ^/(?:.|word) matches _anyCharacter_some_more_characters_ or word _some_more_characters.

timreeves commented 8 years ago

@VicDeo sorry we were, in a way, both right: My original nginx config has a backslash in front of the period in question, which was deleted in the display of my comment by the markdown processing. I have now edited my comment above with the nginx config, and doubled up the backslashes in FIVE places - should work better now :)

timreeves commented 8 years ago

Now the good news: I found something relevant to the Updater problem in the PHP 5.6 error log (maybe the log protocol settings differ in the various PHP versions I have used):

[01-Sep-2016 22:44:21] WARNING: [pool owncloud.reeves.one] child 1848 said into stderr: "[error] " [01-Sep-2016 22:44:21] WARNING: [pool owncloud.reeves.one] child 1848 said into stderr: "" [01-Sep-2016 22:44:21] WARNING: [pool owncloud.reeves.one] child 1848 said into stderr: " " [01-Sep-2016 22:44:21] WARNING: [pool owncloud.reeves.one] child 1848 said into stderr: " [UnexpectedValueException] " [01-Sep-2016 22:44:21] WARNING: [pool owncloud.reeves.one] child 1848 said into stderr: " Could not parse a response for config:list. Please check if the current shell user can run occ command. Raw output: " [01-Sep-2016 22:44:21] WARNING: [pool owncloud.reeves.one] child 1848 said into stderr: " Not allowed " [01-Sep-2016 22:44:21] WARNING: [pool owncloud.reeves.one] child 1848 said into stderr: " " [01-Sep-2016 22:44:21] WARNING: [pool owncloud.reeves.one] child 1848 said into stderr: "" [01-Sep-2016 22:44:21] WARNING: [pool owncloud.reeves.one] child 1848 said into stderr: "" [01-Sep-2016 22:44:21] WARNING: [pool owncloud.reeves.one] child 1848 said into stderr: "Exception trace:" [01-Sep-2016 22:44:21] WARNING: [pool owncloud.reeves.one] child 1848 said into stderr: " () at /var/www/vhosts/reeves.one/owncloud.reeves.one/updater/src/Utils/OccRunner.php:83" [01-Sep-2016 22:44:21] WARNING: [pool owncloud.reeves.one] child 1848 said into stderr: " Owncloud\Updater\Utils\OccRunner->runJson() at /var/www/vhosts/reeves.one/owncloud.reeves.one/updater/src/Utils/ConfigReader.php:106" [01-Sep-2016 22:44:21] WARNING: [pool owncloud.reeves.one] child 1848 said into stderr: " Owncloud\Updater\Utils\ConfigReader->load() at /var/www/vhosts/reeves.one/owncloud.reeves.one/updater/src/Utils/ConfigReader.php:50" [01-Sep-2016 22:44:21] WARNING: [pool owncloud.reeves.one] child 1848 said into stderr: " Owncloud\Updater\Utils\ConfigReader->init() at /var/www/vhosts/reeves.one/owncloud.reeves.one/updater/src/Console/Application.php:154" [01-Sep-2016 22:44:21] WARNING: [pool owncloud.reeves.one] child 1848 said into stderr: " Owncloud\Updater\Console\Application->doRun() at /var/www/vhosts/reeves.one/owncloud.reeves.one/updater/vendor/symfony/console/Application.php:120" [01-Sep-2016 22:44:21] WARNING: [pool owncloud.reeves.one] child 1848 said into stderr: " Symfony\Component\Console\Application->run() at /var/www/vhosts/reeves.one/owncloud.reeves.one/updater/src/Controller/IndexController.php:150" [01-Sep-2016 22:44:21] WARNING: [pool owncloud.reeves.one] child 1848 said into stderr: " Owncloud\Updater\Controller\IndexController->ajaxAction() at /var/www/vhosts/reeves.one/owncloud.reeves.one/updater/src/Controller/IndexController.php:88" [01-Sep-2016 22:44:21] WARNING: [pool owncloud.reeves.one] child 1848 said into stderr: " Owncloud\Updater\Controller\IndexController->dispatch() at /var/www/vhosts/reeves.one/owncloud.reeves.one/updater/index.php:29" [01-Sep-2016 22:44:21] WARNING: [pool owncloud.reeves.one] child 1848 said into stderr: "" [01-Sep-2016 22:44:21] WARNING: [pool owncloud.reeves.one] child 1848 said into stderr: "" [01-Sep-2016 22:44:21] WARNING: [pool owncloud.reeves.one] child 1848 said into stderr: ""

Really hope this helps to find the problem!

VicDeo commented 8 years ago

@timreeves I guess mention of pool in your logs means that CLI php is not used and the request is processed via web stack (nginx/php-fpm).

VicDeo commented 8 years ago

@timreeves I suspect that the following validation is failing:

        if (!in_array($this->request->getRemoteAddress(), ['::1', '127.0.0.1', 'localhost'])) {
            throw new \UnexpectedValueException('Web executor is not allowed to run from a different host');
        }

Please add logging below this line https://github.com/owncloud/core/blob/stable9/core/controller/occcontroller.php#L126

as follows

            \OC::$server->getLogger()->warning(
                    'occ controller requested from an unknown location "{remoteAddress}"',
                    [
                        'app' => 'core',
                        'remoteAddress' => $this->request->getRemoteAddress()
                    ]
            );
timreeves commented 8 years ago

@VicDeo thx for your patient support!

Yes, I am calling the Updater from the web GUI, in my case then via nginx/php-fpm. The nature of the work to be done, and the error message involved, caused me to assume that the GUI component would be making CLI calls in the background to actually do the job - sorry if I confused things.

I added the log warning you requested above, but nothing appears from it in the OC log. But now I did find this in my php-fpm error log:

[05-Sep-2016 18:35:35] WARNING: [pool reeves.one] child 9978 said into stderr: "[error] " [05-Sep-2016 18:35:35] WARNING: [pool reeves.one] child 9978 said into stderr: "" [05-Sep-2016 18:35:35] WARNING: [pool reeves.one] child 9978 said into stderr: " " [05-Sep-2016 18:35:35] WARNING: [pool reeves.one] child 9978 said into stderr: " [UnexpectedValueException] " [05-Sep-2016 18:35:35] WARNING: [pool reeves.one] child 9978 said into stderr: " Could not parse a response for config:list. Please check if the current shell user can run occ command. Raw output: " [05-Sep-2016 18:35:35] WARNING: [pool reeves.one] child 9978 said into stderr: " Not allowed " [05-Sep-2016 18:35:35] WARNING: [pool reeves.one] child 9978 said into stderr: " " [05-Sep-2016 18:35:35] WARNING: [pool reeves.one] child 9978 said into stderr: "" [05-Sep-2016 18:35:35] WARNING: [pool reeves.one] child 9978 said into stderr: "" [05-Sep-2016 18:35:35] WARNING: [pool reeves.one] child 9978 said into stderr: "Exception trace:" [05-Sep-2016 18:35:35] WARNING: [pool reeves.one] child 9978 said into stderr: " () at /var/www/vhosts/reeves.one/owncloud.reeves.one/updater/src/Utils/OccRunner.php:83" [05-Sep-2016 18:35:35] WARNING: [pool reeves.one] child 9978 said into stderr: " Owncloud\Updater\Utils\OccRunner->runJson() at /var/www/vhosts/reeves.one/owncloud.reeves.one/updater/src/Utils/ConfigReader.php:106" [05-Sep-2016 18:35:35] WARNING: [pool reeves.one] child 9978 said into stderr: " Owncloud\Updater\Utils\ConfigReader->load() at /var/www/vhosts/reeves.one/owncloud.reeves.one/updater/src/Utils/ConfigReader.php:50" [05-Sep-2016 18:35:35] WARNING: [pool reeves.one] child 9978 said into stderr: " Owncloud\Updater\Utils\ConfigReader->init() at /var/www/vhosts/reeves.one/owncloud.reeves.one/updater/src/Console/Application.php:154" [05-Sep-2016 18:35:35] WARNING: [pool reeves.one] child 9978 said into stderr: " Owncloud\Updater\Console\Application->doRun() at /var/www/vhosts/reeves.one/owncloud.reeves.one/updater/vendor/symfony/console/Application.php:120" [05-Sep-2016 18:35:35] WARNING: [pool reeves.one] child 9978 said into stderr: " Symfony\Component\Console\Application->run() at /var/www/vhosts/reeves.one/owncloud.reeves.one/updater/src/Controller/IndexController.php:150" [05-Sep-2016 18:35:35] WARNING: [pool reeves.one] child 9978 said into stderr: " Owncloud\Updater\Controller\IndexController->ajaxAction() at /var/www/vhosts/reeves.one/owncloud.reeves.one/updater/src/Controller/IndexController.php:88" [05-Sep-2016 18:35:35] WARNING: [pool reeves.one] child 9978 said into stderr: " Owncloud\Updater\Controller\IndexController->dispatch() at /var/www/vhosts/reeves.one/owncloud.reeves.one/updater/index.php:29" [05-Sep-2016 18:35:35] WARNING: [pool reeves.one] child 9978 said into stderr: "" [05-Sep-2016 18:35:35] WARNING: [pool reeves.one] child 9978 said into stderr: "" [05-Sep-2016 18:35:35] WARNING: [pool reeves.one] child 9978 said into stderr: ""

Looks like it's in the Updater itself, in this function in src/Utils/OccRunner.php

public function runJson($command, $args = []){ $plain = $this->run($command, $args, true); // trim response to always be a valid json. Capture everything between the first and the last curly brace preg_match_all('!(\{.*\})!ms', $plain, $matches); $clean = isset($matches[1][0]) ? $matches[1][0] : ''; $decoded = json_decode($clean, true); if (!is_array($decoded)){ throw new \UnexpectedValueException('Could not parse a response for ' . $command . '. Please check if the current shell user can run occ command. Raw output: ' . PHP_EOL . $plain); } return $decoded; }

Personally I have'nt a clue what that code does - but I rather assume you do - hope this helps!

Tim

VicDeo commented 8 years ago

@timreeves

Looks like it's in the Updater itself

yes. It expects a valid json as a response when requesting index.php/occ/config:list from core but gets "Not allowed" instead. So real root is located in core, namely inside the core/occcontroller.php Because it sends Not allowed instead of the expected reply

VicDeo commented 8 years ago

@timreeves You can temporary change this line to reply with a real reason instead of generic Not allowed: https://github.com/owncloud/core/blob/stable9/core/controller/occcontroller.php#L113

change it to 'response' => $e->getMessage(), and your php-fpm log write it down

timreeves commented 8 years ago

Your suspicion was correct: Raw output: Web executor is not allowed to run from a different host

Also, I added your code from further above now ABOVE the line throwing the exception and get this in my OC log:

{"reqId":"LanE7kdpDc8MVQJmFTPX","remoteAddr":"176.28.16.171","app":"core","message":"occ controller requested from location \"176.28.16.171\"","level":2,"time":"2016-09-05T17:12:08+00:00","method":"POST","url":"\/index.php\/occ\/config:list ","user":"--"}

HA: 176.28.16.171 is the external IP of my Server - is there some way you (or the core author involved) can find that out and allow it through also? But beware: I have FIVE IP's on that server, assigned via Plesk to different domains, so I can use reverse DNS on them for optimal SEO. On that server, luckily ownCloud is on the basic IP of the server, but other users may have it on a secondary IP - I think basically we need either the IP of the Domain in use, failing that any IP assigned to the server.

timreeves commented 8 years ago

And here is my $_SERVER as comes through nginx - php-fpm, in case it helps:

$_SERVER: array ( ... 'front_controller_active' => 'true', 'modHeadersAvailable' => 'true', 'REDIRECT_STATUS' => '200', ... 'SERVER_ADDR' => '176.28.16.171', ... 'SERVER_SOFTWARE' => 'nginx/1.11.1', ... ... 'FCGI_ROLE' => 'RESPONDER', ... )

VicDeo commented 8 years ago

@DeepDiver1975 looks like we need to allow occ controller execution for requests originating from $_SERVER['SERVER_ADDR'] (if it exists)

VicDeo commented 8 years ago

@timreeves thanks for these details, but I think you'd better cut them a bit (or even completely) This request is blocked as potentially unsafe one.
Since $_SERVER['SERVER_ADDR'] is server-controlled it should be safe to allow queries from this IP too.

VicDeo commented 8 years ago

@timreeves for now it should be enough to replace if (!in_array($this->request->getRemoteAddress(), ['::1', '127.0.0.1', 'localhost'])) { with if (!in_array($this->request->getRemoteAddress(), ['::1', '127.0.0.1', 'localhost', $this->request->server['SERVER_ADDR'])) { in core/controller/occcontroller.php

timreeves commented 8 years ago

I have tried the suggestion - which looks reasonable to me - on my test server, which is Ubuntu 12.04 LTS with OS-PHP 5.3.10 (but the GUI is using Plesk PHP 7.0.9). I don't dare doing it on my production server, because the Updater is only offered if I select "daily" - probably not a good idea for production.

What I get is this:

OC-Log:

{"reqId":"UxScOx90RfW4JS6r/So7","remoteAddr":"","app":"index","message":"Exception: {"Exception":"ParseError","Message":"syntax error, unexpected ')', expecting ']'","Code":0,"Trace":"

0 /var/www/vhosts//lib/composer/composer/ClassLoader.php(310): ComposerAutoloadincludeFile('/var/www/vhosts...')

1 [internal function]: ComposerAutoloadClassLoader->loadClass('OCCoreControl...')

2 /var/www/vhosts//core/Application.php(130): spl_autoload_call('OCCoreControl...')

3 /var/www/vhosts//3rdparty/pimple/pimple/src/Pimple/Container.php(113): OCCoreApplication->OCCore{closure}(Object(OCAppFrameworkDependencyInjectionDIContainer))

4 /var/www/vhosts//lib/private/AppFramework/Utility/SimpleContainer.php(102): PimpleContainer->offsetGet('OccController')

5 /var/www/vhosts//lib/private/AppFramework/App.php(93): OCAppFrameworkUtilitySimpleContainer->query('OccController')

6 /var/www/vhosts//lib/private/AppFramework/Routing/RouteActionHandler.php(46): OCAppFrameworkApp::main('OccController', 'execute', Object(OCAppFrameworkDependencyInjectionDIContainer), Array)

7 [internal function]: OCAppFrameworkRoutingRouteActionHandler->__invoke(Array)

8 /var/www/vhosts//lib/private/Route/Router.php(280): call_user_func(Object(OCAppFrameworkRoutingRouteActionHandler), Array)

9 /var/www/vhosts//lib/base.php(891): OCRouteRouter->match('/occ/config:lis...')

10 /var/www/vhosts//index.php(39): OC::handleRequest()

11 {main}","File":"/var/www/vhosts//core/Controller/OccController.php","Line":128}",

"level":3,"time":"2016-09-05T19:33:12+00:00","method":"POST","url":"/index.php/occ/config:list","user":"--" }

FPM-Log:

[05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: "[error] " [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: "" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: " " [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: " [GuzzleHttp\Exception\ServerException] " [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: " Server error response [url] https://owncloud./index.php/occ/config:list [status code] 500 [reason phrase] Internal Server Error " [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: " " [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: "" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: "" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: "Exception trace:" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: " () at /var/www/vhosts//owncloud./updater/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:89" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: " GuzzleHttp\Exception\RequestException::create() at /var/www/vhosts//owncloud./updater/vendor/guzzlehttp/guzzle/src/Subscriber/HttpError.php:32" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: " GuzzleHttp\Subscriber\HttpError->onComplete() at /var/www/vhosts//owncloud./updater/vendor/guzzlehttp/guzzle/src/Event/Emitter.php:109" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: " GuzzleHttp\Event\Emitter->emit() at /var/www/vhosts//owncloud./updater/vendor/guzzlehttp/guzzle/src/RequestFsm.php:91" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: " GuzzleHttp\RequestFsm->invoke() at /var/www/vhosts//owncloud./updater/vendor/guzzlehttp/guzzle/src/RequestFsm.php:132" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: " GuzzleHttp\RequestFsm->GuzzleHttp{closure}() at /var/www/vhosts//owncloud./updater/vendor/react/promise/src/FulfilledPromise.php:25" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: " React\Promise\FulfilledPromise->then() at /var/www/vhosts//owncloud./updater/vendor/guzzlehttp/ringphp/src/Future/CompletedFutureValue.php:55" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: " GuzzleHttp\Ring\Future\CompletedFutureValue->then() at /var/www/vhosts//owncloud./updater/vendor/guzzlehttp/guzzle/src/Message/FutureResponse.php:43" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: " GuzzleHttp\Message\FutureResponse::proxy() at /var/www/vhosts//owncloud./updater/vendor/guzzlehttp/guzzle/src/RequestFsm.php:134" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: " GuzzleHttp\RequestFsm->invoke() at /var/www/vhosts//owncloud./updater/vendor/guzzlehttp/guzzle/src/Client.php:165" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: " GuzzleHttp\Client->send() at /var/www/vhosts//owncloud./updater/src/Utils/OccRunner.php:103" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: " Owncloud\Updater\Utils\OccRunner->runAsRequest() at /var/www/vhosts//owncloud./updater/src/Utils/OccRunner.php:70" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: " Owncloud\Updater\Utils\OccRunner->run() at /var/www/vhosts//owncloud./updater/src/Utils/OccRunner.php:77" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: " Owncloud\Updater\Utils\OccRunner->runJson() at /var/www/vhosts//owncloud./updater/src/Utils/ConfigReader.php:106" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: " Owncloud\Updater\Utils\ConfigReader->load() at /var/www/vhosts//owncloud./updater/src/Utils/ConfigReader.php:50" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: " Owncloud\Updater\Utils\ConfigReader->init() at /var/www/vhosts//owncloud./updater/src/Console/Application.php:154" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: " Owncloud\Updater\Console\Application->doRun() at /var/www/vhosts//owncloud./updater/vendor/symfony/console/Application.php:120" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: " Symfony\Component\Console\Application->run() at /var/www/vhosts//owncloud./updater/src/Controller/IndexController.php:150" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: " Owncloud\Updater\Controller\IndexController->ajaxAction() at /var/www/vhosts//owncloud./updater/src/Controller/IndexController.php:88" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: " Owncloud\Updater\Controller\IndexController->dispatch() at /var/www/vhosts//owncloud./updater/index.php:29" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: "" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: "" [05-Sep-2016 21:59:33] WARNING: [pool owncloud.] child 26145 said into stderr: ""

Can that be caused by the outdated Ubuntu?

I would try it on my production server IF you can tell me how to ensure that the Updater does not actually run when the tests are passed...

VicDeo commented 8 years ago

@timreeves my fault. one closing square bracket is missing:

if ( !in_array($this->request->getRemoteAddress(), ['::1', '127.0.0.1', 'localhost', $this->request->server['SERVER_ADDR'] ]) ) {
timreeves commented 8 years ago

Heartfelt Thanks @VicDeo - SUCCESS, the corrected patch works, tested on Ubuntu 12.04 and 14.04.

timreeves commented 8 years ago

Don't know if this would be better as a new topic, but: I noticed that when I call the Updater and it runs through the initialisation - with success at the GUI - and before I click on "Start", then these entries appear in the OC log:

{"reqId":"...","remoteAddr":"176.28.16.171","app":"PHP","message":"Undefined index: argv at \/var\/www\/vhosts\/{domain}\/3rdparty\/symfony\/console\/Input\/ArgvInput.php#55","level":3,"time":"2016-09-06T07:30:42+00:00","method":"POST","url":"\/index.php\/occ\/config:list","user":"--"}

{"reqId":"...","remoteAddr":"176.28.16.171","app":"PHP","message":"array_shift() expects parameter 1 to be array, null given at \/var\/www\/vhosts\/{domain}\/3rdparty\/symfony\/console\/Input\/ArgvInput.php#59","level":3,"time":"2016-09-06T07:30:42+00:00","method":"POST","url":"\/index.php\/occ\/config:list","user":"--"}

{"reqId":"...","remoteAddr":"176.28.16.171","app":"PHP","message":"Invalid argument supplied for foreach() at \/var\/www\/vhosts\/{domain}\/3rdparty\/symfony\/console\/Input\/ArgvInput.php#262","level":3,"time":"2016-09-06T07:30:42+00:00","method":"POST","url":"\/index.php\/occ\/config:list","user":"--"}

{"reqId":"...","remoteAddr":"176.28.16.171","app":"PHP","message":"Undefined index: argv at \/var\/www\/vhosts\/{domain}\/3rdparty\/symfony\/console\/Input\/ArgvInput.php#55","level":3,"time":"2016-09-06T07:30:43+00:00","method":"POST","url":"\/index.php\/occ\/status","user":"--"}

{"reqId":"...","remoteAddr":"176.28.16.171","app":"PHP","message":"array_shift() expects parameter 1 to be array, null given at \/var\/www\/vhosts\/{domain}\/3rdparty\/symfony\/console\/Input\/ArgvInput.php#59","level":3,"time":"2016-09-06T07:30:43+00:00","method":"POST","url":"\/index.php\/occ\/status","user":"--"}

{"reqId":"...","remoteAddr":"176.28.16.171","app":"PHP","message":"Invalid argument supplied for foreach() at \/var\/www\/vhosts\/{domain}\/3rdparty\/symfony\/console\/Input\/ArgvInput.php#262","level":3,"time":"2016-09-06T07:30:43+00:00","method":"POST","url":"\/index.php\/occ\/status","user":"--"}

lock[bot] commented 5 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.