Closed galandilias closed 5 years ago
kesselb
commented
5 years ago https://github.com/nextcloud/server/issues/16860 https://github.com/nextcloud/server/issues/16054
As reference. Could you add some more details about the setup (see the linked issues above for examples).
galandilias
commented
5 years ago Sure - no problem:
galandilias
commented
5 years ago As for reference #12722 <- I found this one at first - and looks exactly the same to mine.
kesselb
commented
5 years ago As for reference #12722 <- I found this one at first - and looks exactly the same to mine.
Update to PHP 7.3 then? ;)
kesselb
commented
5 years ago Its interesting that most of these reports are from Gentoo users :thinking: Maybe they just do more monitoring :bar_chart:
galandilias
commented
5 years ago For my peace of mind :-) just upgraded php to 7.3 - let's see :-)
With gentoo typically you are low on hw ;-) so every core counts same as every bit of ram ;-)
galandilias
commented
5 years ago Okay - appeared again on php7.3. :( Same logs as before. Anything else needed?
kesselb
commented
5 years ago you are low on hw ;-)
Mind to share some details about your hw. CPU, Memory, Disk, etc.?
Same logs as before.
The notice is coming from searchdav.
Anything else needed?
Do you see higher load the mariadb? Could you enable query log and check what queries are related to the higher load?
galandilias
commented
5 years ago Okay - HW - basically Proliant N36L so:
Should not be that bad ;)
From perspective of top/htop apache is consuming one core, mariadb is not too active. I will enable the logging anyway. Before I will restart apache - anything else you want to collect? Issue is still present :)
kesselb
commented
5 years ago anything else you want to collect?
Is there anything in the apache2 logs?
apache is consuming one core
Hmm. Don't remember why but I suggested at #12722 to use php-fpm instead of mod_php. There should be no difference between php-fpm and mod_php but ...
Grokenberger
commented
5 years ago Maybe not related, but I've also a DAV related issue: When I use the Android client app, version 3.8.1, the CPU load goes high if I tap the "Shared" button Syslog writes immediately:
Nextcloud[216273]: {PHP} Undefined offset: 0 at /usr/share/webapps/nextcloud/3rdparty/icewind/searchdav/src/XML/Operator.php#71
Operating system: Arch Linux with kernel 4.19.78-1-lts
Web server: Apache 2.4.41-1
Database: mariadb 10.4.8-2
PHP version: php 7.3.10-1
Nextcloud version: 16.0.5
kesselb
commented
5 years ago Index: icewind/searchdav/src/XML/Operator.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- icewind/searchdav/src/XML/Operator.php (date 1568109763000)
+++ icewind/searchdav/src/XML/Operator.php (date 1570888761226)
@@ -68,6 +68,9 @@
if ($reader->nodeType === Reader::ELEMENT) {
$argument = $reader->parseCurrentElement();
if ($argument['name'] === '{DAV:}prop') {
+ if (!isset($argument['value'][0])) {
+ trigger_error('Element at Index 0 not present. Type: ' . $operator->type . ', Argument: ' . print_r($argument, true));
+ }
$operator->arguments[] = $argument['value'][0];
} else {
$operator->arguments[] = $argument['value'];Mind to patch your instance? Above code should add some more details about the element with the missing index.
Grokenberger
commented
5 years ago Here you go:
Nextcloud[272227]: {PHP} Element at Index 0 not present. Type: {DAV:}like, Argument: Array
(
[name] => {DAV:}prop
[value] => Array
(
)
[attributes] => Array
(
)
)
at /usr/share/webapps/nextcloud/3rdparty/icewind/searchdav/src/XML/Operator.php#72
Nextcloud[272227]: {PHP} Undefined offset: 0 at /usr/share/webapps/nextcloud/3rdparty/icewind/searchdav/src/XML/Operator.php#74Index: icewind/searchdav/src/XML/Operator.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- icewind/searchdav/src/XML/Operator.php (revision 9a5103deb1bfb8b0b809d86cd400b429fee06106)
+++ icewind/searchdav/src/XML/Operator.php (date 1570962291211)
@@ -68,6 +68,9 @@
if ($reader->nodeType === Reader::ELEMENT) {
$argument = $reader->parseCurrentElement();
if ($argument['name'] === '{DAV:}prop') {
+ if (!isset($argument['value'][0])) {
+ throw new \InvalidArgumentException('Element at Index 0 not present. Type: ' . $operator->type . ', Argument: ' . print_r($argument, true));
+ }
$operator->arguments[] = $argument['value'][0];
} else {
$operator->arguments[] = $argument['value'];Could you temporary apply this patch? It will probably take your instance down :see_no_evil: I throw a exception instead of trigger_error to generate a stack trace. The stack trace should show the way of the request.
Grokenberger
commented
5 years ago Yes. Done:
Nextcloud[328906]: {webdav} {"Exception":"InvalidArgumentException","Message":"Element at Index 0 not present. Type: {DAV:}like, Argument: Array\n(\n [name] => {DAV:}prop\n [value] => Array\n (\n )\n\n [attributes] => Array\n (\n )\n\n)\n","Code":0,"Trace":[{"function":"xmlDeserialize","class":"SearchDAV\\XML\\Operator","type":"::","args":["*** sensitive parameter replaced ***"]},{"file":"\/usr\/share\/webapps\/nextcloud\/3rdparty\/sabre\/xml\/lib\/Reader.php","line":246,"function":"call_user_func","args":[["SearchDAV\\XML\\Operator","xmlDeserialize"],"*** sensitive parameter replaced ***"]},{"file":"\/usr\/share\/webapps\/nextcloud\/3rdparty\/sabre\/xml\/lib\/Reader.php","line":174,"function":"parseCurrentElement","class":"Sabre\\Xml\\Reader","type":"->","args":[]},{"file":"\/usr\/share\/webapps\/nextcloud\/3rdparty\/sabre\/xml\/lib\/Reader.php","line":107,"function":"parseInnerTree","class":"Sabre\\Xml\\Reader","type":"->","args":[null]},{"file":"\/usr\/share\/webapps\/nextcloud\/3rdparty\/icewind\/searchdav\/src\/DAV\/QueryParser.php","line":61,"function":"parseGetElements","class":"Sabre\\Xml\\Reader","type":"->","args":[]},{"function":"SearchDAV\\DAV\\{closure}","class":"SearchDAV\\DAV\\QueryParser","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/usr\/share\/webapps\/nextcloud\/3rdparty\/sabre\/xml\/lib\/Reader.php","line":246,"function":"call_user_func","args":[{"__class__":"Closure"},"*** sensitive parameter replaced ***"]},{"file":"\/usr\/share\/webapps\/nextcloud\/3rdparty\/sabre\/xml\/lib\/Deserializer\/functions.php","line":79,"function":"parseCurrentElement","class":"Sabre\\Xml\\Reader","type":"->","args":[]},{"file":"\/usr\/share\/webapps\/nextcloud\/3rdparty\/icewind\/searchdav\/src\/XML\/BasicSearch.php","line":76,"function":"Sabre\\Xml\\Deserializer\\keyValue","args":["*** sensitive parameter replaced ***"]},{"function":"xmlDeserialize","class":"SearchDAV\\XML\\BasicSearch","type":"::","args":["*** sensitive parameter replaced ***"]},{"file":"\/usr\/share\/webapps\/nextcloud\/3rdparty\/sabre\/xml\/lib\/Reader.php","line":246,"function":"call_user_func","args":[["SearchDAV\\XML\\BasicSearch","xmlDeserialize"],"*** sensitive parameter replaced ***"]},{"file":"\/usr\/share\/webapps\/nextcloud\/3rdparty\/sabre\/xml\/lib\/Deserializer\/functions.php","line":79,"function":"parseCurrentElement","class":"Sabre\\Xml\\Reader","type":"->","args":[]},{"file":"\/usr\/share\/webapps\/nextcloud\/3rdparty\/sabre\/xml\/lib\/Element\/KeyValue.php","line":104,"function":"Sabre\\Xml\\Deserializer\\keyValue","args":["*** sensitive parameter replaced ***"]},{"function":"xmlDeserialize","class":"Sabre\\Xml\\Element\\KeyValue","type":"::","args":["*** sensitive parameter replaced ***"]},{"file":"\/usr\/share\/webapps\/nextcloud\/3rdparty\/sabre\/xml\/lib\/Reader.php","line":246,"function":"call_user_func","args":[["Sabre\\Xml\\Element\\KeyValue","xmlDeserialize"],"*** sensitive parameter replaced ***"]},{"file":"\/usr\/share\/webapps\/nextcloud\/3rdparty\/sabre\/xml\/lib\/Reader.php","line":71,"function":"parseCurrentElement","class":"Sabre\\Xml\\Reader","type":"->","args":[]},{"file":"\/usr\/share\/webapps\/nextcloud\/3rdparty\/sabre\/xml\/lib\/Service.php","line":118,"function":"parse","class":"Sabre\\Xml\\Reader","type":"->","args":[]},{"file":"\/usr\/share\/webapps\/nextcloud\/3rdparty\/icewind\/searchdav\/src\/DAV\/SearchPlugin.php","line":111,"function":"parse","class":"Sabre\\Xml\\Service","type":"->","args":["<?xml version=\"1.0\" encoding=\"UTF-8\"?><d:searchrequest xmlns:d=\"DAV:\" xmlns:oc=\"http:\/\/nextcloud.com\/ns\"><d:basicsearch><d:select><d:prop><d:displayname\/><d:getcontenttype\/><d:resourcetype\/><d:getcontentlength\/><d:getlastmodified\/><d:creationdate\/><d:getetag\/><d:quota-used-bytes\/><d:quota-available-bytes\/><oc:permissions xmlns:oc=\"http:\/\/owncloud.org\/ns\"\/><oc:id xmlns:oc=\"http:\/\/owncloud.org\/ns\"\/><oc:size xmlns:oc=\"http:\/\/owncloud.org\/ns\"\/><oc:favorite xmlns:oc=\"http:\/\/owncloud.org\/ns\"\/><\/d:prop><\/d:select><d:from><d:scope><d:href>\/files\/holli<\/d:href><d:depth>infinity<\/d:depth><\/d:scope><\/d:from><d:where><d:like><d:prop\/><d:literal\/><\/d:like><\/d:where><d:orderby\/><\/d:basicsearch><\/d:searchrequest>","\/remote.php\/dav",null]},{"function":"searchHandler","class":"SearchDAV\\DAV\\SearchPlugin","type":"->","args":[{"absoluteUrl":"https:\/\/nc.XXX.net\/remote.php\/dav","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"\/usr\/share\/webapps\/nextcloud\/3rdparty\/sabre\/event\/lib\/EventEmitterTrait.php","line":105,"function":"call_user_func_array","args":[[{"__class__":"SearchDAV\\DAV\\SearchPlugin"},"searchHandler"],[{"absoluteUrl":"https:\/\/nc.XXX.net\/remote.php\/dav","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"\/usr\/share\/webapps\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php","line":479,"function":"emit","class":"Sabre\\Event\\EventEmitter","type":"->","args":["method:SEARCH",[{"absoluteUrl":"https:\/\/nc.XXX.net\/remote.php\/dav","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"\/usr\/share\/webapps\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php","line":254,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[{"absoluteUrl":"https:\/\/nc.XXX.net\/remote.php\/dav","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"\/usr\/share\/webapps\/nextcloud\/apps\/dav\/lib\/Server.php","line":316,"function":"exec","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"\/usr\/share\/webapps\/nextcloud\/apps\/dav\/appinfo\/v2\/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server","type":"->","args":[]},{"file":"\/usr\/share\/webapps\/nextcloud\/remote.php","line":163,"args":["\/usr\/share\/webapps\/nextcloud\/apps\/dav\/appinfo\/v2\/remote.php"],"function":"require_once"}],"File":"\/usr\/share\/webapps\/nextcloud\/3rdparty\/icewind\/searchdav\/src\/XML\/Operator.php","Line":72,"CustomMessage":"--"}Thanks @Grokenberger
<?xml version=\"1.0\" encoding=\"UTF-8\"?><d:searchrequest xmlns:d=\"DAV:\" xmlns:oc=\"http:\/\/nextcloud.com\/ns\"><d:basicsearch><d:select><d:prop><d:displayname\/><d:getcontenttype\/><d:resourcetype\/><d:getcontentlength\/><d:getlastmodified\/><d:creationdate\/><d:getetag\/><d:quota-used-bytes\/><d:quota-available-bytes\/><oc:permissions xmlns:oc=\"http:\/\/owncloud.org\/ns\"\/><oc:id xmlns:oc=\"http:\/\/owncloud.org\/ns\"\/><oc:size xmlns:oc=\"http:\/\/owncloud.org\/ns\"\/><oc:favorite xmlns:oc=\"http:\/\/owncloud.org\/ns\"\/><\/d:prop><\/d:select><d:from><d:scope><d:href>\/files\/holli<\/d:href><d:depth>infinity<\/d:depth><\/d:scope><\/d:from><d:where><d:like><d:prop\/><d:literal\/><\/d:like><\/d:where><d:orderby\/><\/d:basicsearch><\/d:searchrequest>
@tobiasKaminsky do you know which action is behind this request?
galandilias
commented
5 years ago ok - applied patch on mine too - lets see :)
tobiasKaminsky
commented
5 years ago Thanks @Grokenberger
<?xml version=\"1.0\" encoding=\"UTF-8\"?><d:searchrequest xmlns:d=\"DAV:\" xmlns:oc=\"http:\/\/nextcloud.com\/ns\"><d:basicsearch><d:select><d:prop><d:displayname\/><d:getcontenttype\/><d:resourcetype\/><d:getcontentlength\/><d:getlastmodified\/><d:creationdate\/><d:getetag\/><d:quota-used-bytes\/><d:quota-available-bytes\/><oc:permissions xmlns:oc=\"http:\/\/owncloud.org\/ns\"\/><oc:id xmlns:oc=\"http:\/\/owncloud.org\/ns\"\/><oc:size xmlns:oc=\"http:\/\/owncloud.org\/ns\"\/><oc:favorite xmlns:oc=\"http:\/\/owncloud.org\/ns\"\/><\/d:prop><\/d:select><d:from><d:scope><d:href>\/files\/holli<\/d:href><d:depth>infinity<\/d:depth><\/d:scope><\/d:from><d:where><d:like><d:prop\/><d:literal\/><\/d:like><\/d:where><d:orderby\/><\/d:basicsearch><\/d:searchrequest>@tobiasKaminsky do you know which action is behind this request?
@kesselb this seems to be the favorite search: get all favorites in all directories.
mjanssens
commented
5 years ago @tobiasKaminsky do you know which action is behind this request?
@kesselb this seems to be the favorite search: get all favorites in all directories.
@kesselb also triggered in the android client by the shared search in the menu.
galandilias
commented
5 years ago More logs from my side - and I can confirm that such queries are generated from Android app for sure.
NaofumiHonda
commented
5 years ago I also encounter the same situation by putting the "Shared button" of the Android client app, version 3.8.1. By using gdb, I found that the server process enters into infinite loops around
do {
.........
} else {
$reader->read(); <=== HERE
}
} while ($reader->nodeType !== Reader::END_ELEMENT);of icewind/searchdav/src/XML/Operator.php, where the $reader->read() always returns false! If we replace $reader->read() with the code
if (!$reader->read()) {
$reader->next();
return $operator;
}then the problem goes away.
I do not know the essential reason of these strange behaviors, but at lease, to check the return value of read() is a good practice in case.
kesselb
commented
5 years ago By using gdb,
Nice! I will add this to my tools list :+1: Could be handy if xdebug is not available.
I found that the server process enters into infinite loops around
Thanks for debugging :tada: Looks similar to https://github.com/sabre-io/xml/pull/158.
if (!$reader->read()) {
break;
}I would prefer this version because it does not duplicate the return statment. This should also work.
I do not know the essential reason of these strange behaviors, but at lease, to check the return value of read() is a good practice in case.
Would you mind to open a pull request at https://github.com/icewind1991/SearchDAV with your changes? The process is: Fix it upstream (1), patch nextcloud/3rdparty (2) and update nextcloud/server (3). Don't worry I can take over 2 and/or 3.
cc @icewind1991 @rullzer
kesselb
commented
5 years ago <?xml version="1.0" encoding="UTF-8"?>
<d:searchrequest xmlns:d="DAV:" xmlns:oc="http://nextcloud.com/ns">
<d:basicsearch>
<d:select>
<d:prop>
<d:displayname />
<d:getcontenttype />
<d:resourcetype />
<d:getcontentlength />
<d:getlastmodified />
<d:creationdate />
<d:getetag />
<d:quota-used-bytes />
<d:quota-available-bytes />
<oc:permissions xmlns:oc="http://owncloud.org/ns" />
<oc:id xmlns:oc="http://owncloud.org/ns" />
<oc:size xmlns:oc="http://owncloud.org/ns" />
<oc:favorite xmlns:oc="http://owncloud.org/ns" />
</d:prop>
</d:select>
<d:from>
<d:scope>
<d:href>/files/holli</d:href>
<d:depth>infinity</d:depth>
</d:scope>
</d:from>
<d:where>
<d:like>
<d:prop />
<d:literal />
</d:like>
</d:where>
<d:orderby />
</d:basicsearch>
</d:searchrequest>@tobiasKaminsky I'm not an expert about these searchdav requests but sending a empty where looks strange. Is this intended?
kesselb
commented
5 years ago @NaofumiHonda thanks for opening the pull request over at icewind1991/SearchDAV :+1: I'm having troubles to reproduce the loop locally. Could you try https://github.com/nextcloud/server/issues/17447#issuecomment-541406151 and share the xml request?
NaofumiHonda
commented
5 years ago Here is a log (both the format and unformated ones).
kesselb
commented
5 years ago Thanks :+1: It's the same request as above :thinking:
NaofumiHonda
commented
5 years ago OK, I put debug prints in xmlDeserialize() of Operator.php as follows:
---------------------------
HERE $debugv = 0;
---------------------------
$reader->read();
do {
if ($reader->nodeType === Reader::ELEMENT) {
$argument = $reader->parseCurrentElement();
if ($argument['name'] === '{DAV:}prop') {
-----------------------------------
if (!isset($argument['value'][0])) {
HERE $debugv = 1;
}
-----------------------------------
$operator->arguments[] = $argument['value'][0];
} else {
$operator->arguments[] = $argument['value'];
}
------------------------------
if ($debugv === 1) {
HERE trigger_error('Element. Type: ' . $operator->type . ', Argument: ' . print_r($argument, true));
}
------------------------------
} else {
$reader->read();
}
-----------------------------
if ($debugv === 1) {
trigger_error('Next nodeType = ' . $reader->nodeType);
HERE if ($reader->nodeType === Reader::NONE) {
throw new \InvalidArgumentException('Reader returns NONE(1)!!!');
}
----------------------------
}
} while ($reader->nodeType !== Reader::END_ELEMENT);
----------------------------
if ($debugv === 1) {
HERE trigger_error('Exit loops');
}
----------------------------So, after the "Undefined offset: 0" error is triggered, all the reads and parse results are monitored. Now try to push the shared button in Andoroid client, we got the error log below with an exception caused by nodeType === NONE.
Fatal InvalidArgumentException: Reader returns NONE(1)!!!
Error Next nodeType = 0
at 3rdparty/icewind/searchdav/src/XML/Operator.php#86
Error Element.
Type: {DAV:}like,
Argument: Array ( [name] => {DAV:}literal
[value] => SearchDAV\XML\Literal Object ( [value] => )
[attributes] => Array ( )
)
at 3rdparty/icewind/searchdav/src/XML/Operator.php#80
Error Next nodeType = 1
at 3rdparty/icewind/searchdav/src/XML/Operator.php#86
Error Element.
Type: {DAV:}like,
Argument: Array ( [name] => {DAV:}prop
[value] => Array ( )
[attributes] => Array ( )
)
at 3rdparty/icewind/searchdav/src/XML/Operator.php#80
Error Undefined offset: 0
at 3rdparty/icewind/searchdav/src/XML/Operator.php#75Clearly, in deserialization of
NaofumiHonda
commented
5 years ago PS. My server is php 7.3.11, Apache httpd 2.4.41 with prefork.
tobiasKaminsky
commented
5 years ago @tobiasKaminsky I'm not an expert about these searchdav requests but sending a empty where looks strange. Is this intended?
Me neither ;-), but I assume that this boils down to something like "where 1=1" (on sql). A quick look into rfc does not reveal if it is allowed or not.
But I guess that this is not the problem, but the other one @NaofumiHonda pointed out.
(all in all: a valid server api endpoint should not lead to 100%. And yeah, we can discuss if there is a better approach for Android Files app to get all shares ;-))
NaofumiHonda
commented
5 years ago The following patch also resolves an infinite loop, that is, the loop terminates with $reader->nodeType === Reader::END_ELEMENT. Of course, "where" arguments in the XML request seems invalid (they are null), and thus, we have finally
InvalidArgumentException: Invalid argument 1 for like operation, expected property
/opt/htdocs/nextcloud/apps/dav/lib/Files/FileSearchBackend.php - line 274:
OCA\DAV\Files\FileSearchBackend->transformSearchOperation(SearchDAV\Qu ... ]})But this is another story. Here is a patch:
*** 3rdparty/icewind/searchdav/src/XML/Literal.php.org 2019-11-04 15:53:53.111854119 +0900
--- 3rdparty/icewind/searchdav/src/XML/Literal.php 2019-11-04 17:34:15.439518618 +0900
***************
*** 29,35 ****
static function xmlDeserialize(Reader $reader) {
$literal = new self();
! $literal->value = $reader->readText();
$reader->read();
return $literal;
--- 29,39 ----
static function xmlDeserialize(Reader $reader) {
$literal = new self();
! if ($reader->isEmptyElement) {
! $literal->value = '';
! } else {
! $literal->value = $reader->readText();
! }
$reader->read();
return $literal;InvalidArgumentException: Reader returns NONE(1)!!!
@NaofumiHonda could you share the stack trace for this exception? I would like to check the xml request and it's way.
Me neither ;-), but I assume that this boils down to something like "where 1=1" (on sql). A quick look into rfc does not reveal if it is allowed or not.
Right now requests without where are not allowed code wise.
a valid server api endpoint should not lead to 100%
Of course ;)
InvalidArgumentException: Invalid argument 1 for like operation, expected property
This looks ok.
But this is another story. Here is a patch:
Looks good. I'm tested it locally and could not see any difference but code makes sense ;) $reader->readText(); will return '' if the element is empty.
I send https://github.com/nextcloud/server/issues/17447#issuecomment-548385788 to my production instance (Nextcloud 16.0.x, PHP 7.2.x via FPM and Apache 2.4) and got "InvalidArgumentException: Invalid argument 1 for like operation, expected property" in return.
curl -X SEARCH \
https://yournextcloudinstance.com/remote.php/dav/ \
-H 'Authorization: Basic base64encodedstring username:password' \
-H 'Content-Type: application/xml' \
-d '<?xml version="1.0" encoding="UTF-8"?>
<d:searchrequest xmlns:d="DAV:" xmlns:oc="http://nextcloud.com/ns">
<d:basicsearch>
<d:select>
<d:prop>
<d:displayname/>
<d:getcontenttype/>
<d:resourcetype/>
<d:getcontentlength/>
<d:getlastmodified/>
<d:creationdate/>
<d:getetag/>
<d:quota-used-bytes/>
<d:quota-available-bytes/>
<oc:permissions xmlns:oc="http://owncloud.org/ns"/>
<oc:id xmlns:oc="http://owncloud.org/ns"/>
<oc:size xmlns:oc="http://owncloud.org/ns"/>
<oc:favorite xmlns:oc="http://owncloud.org/ns"/>
</d:prop>
</d:select>
<d:from>
<d:scope>
<d:href>/files/username</d:href>
<d:depth>infinity</d:depth>
</d:scope>
</d:from>
<d:where>
<d:like>
<d:prop />
<d:literal />
</d:like>
</d:where>
<d:orderby/>
</d:basicsearch>
</d:searchrequest>'Could you send the above request to your instance?
1) https://yournextcloudinstance.com/remote.php/dav/ replace it with your domain.
2) Authorization: Basic base64encodedusername:password "username:password" and encode it with base64. php -r "echo base64_encode('daniel:secret');"
3) <d:href>/files/username</d:href> insert your username
Thanks :+1: I'm sorry for all those questions. Still looking for a way how to reproduce this.
NaofumiHonda
commented
5 years ago OK! I understood everything completely now.
For your suggested SEARCH request, neither I have any infinite loop. However, if you remove all the tabs and spaces between tags in your data argument like below, then SURELY you have had the inifite loop!
curl -X SEARCH \
https://yournextcloudinstance.com/remote.php/dav/ \
-H 'Authorization: Basic base64encodedstring username:password' \
-H 'Content-Type: application/xml' \
-d '<?xml version="1.0" encoding="UTF-8"?>
<d:searchrequest xmlns:d="DAV:" xmlns:oc="http://nextcloud.com/ns">
<d:basicsearch>
<d:select>
<d:prop>
<d:displayname/>
<d:getcontenttype/>
<d:resourcetype/>
<d:getcontentlength/>
<d:getlastmodified/>
<d:creationdate/>
<d:getetag/>
<d:quota-used-bytes/>
<d:quota-available-bytes/>
<oc:permissions xmlns:oc="http://owncloud.org/ns"/>
<oc:id xmlns:oc="http://owncloud.org/ns"/>
<oc:size xmlns:oc="http://owncloud.org/ns"/>
<oc:favorite xmlns:oc="http://owncloud.org/ns"/>
</d:prop>
</d:select>
<d:from>
<d:scope>
<d:href>/files/username</d:href>
<d:depth>infinity</d:depth>
</d:scope>
</d:from>
<d:where><d:like><d:prop /><d:literal /></d:like></d:where><d:orderby/></d:basicsearch</d:searchrequest>'All these strange behaviors come from arguably a bad implementation of readText() in 3rdparty/sabre/xml/lib/Reader.php as you see below. If you call readText() in an EMPTY element, then all the subsequent elements are exhausted by $this->read() in the while loop because of $this->depth != $previousDepth (it should be at least $this->depth > $previousDepth). The important lesson is that you never call readText() in an empty element!
function readText() {
$result = '';
$previousDepth = $this->depth;
while ($this->read() && $this->depth != $previousDepth) {
if (in_array($this->nodeType, [XMLReader::TEXT, XMLReader::CDATA, XMLReader::WHITESPACE]))
{
$result .= $this->value;
}
}
return $result;
}Sorry... </d:basicsearch</d:searchrequest> should be </d:basicsearch></d:searchrequest> in the last line of the argument.
kesselb
commented
5 years ago For your suggested SEARCH request, neither I have any infinite loop. However, if you remove all the tabs and spaces between tags in your data argument like below, then SURELY you have had the inifite loop!
Unbelievable!
Index: tests/infiniteloopemptyliteral.xml
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- tests/infiniteloopemptyliteral.xml (date 1572958629271)
+++ tests/infiniteloopemptyliteral.xml (date 1572958629271)
@@ -0,0 +1,30 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<d:searchrequest xmlns:d="DAV:" xmlns:oc="http://nextcloud.com/ns">
+ <d:basicsearch>
+ <d:select>
+ <d:prop>
+ <d:displayname/>
+ <d:getcontenttype/>
+ <d:resourcetype/>
+ <d:getcontentlength/>
+ <d:getlastmodified/>
+ <d:creationdate/>
+ <d:getetag/>
+ <d:quota-used-bytes/>
+ <d:quota-available-bytes/>
+ <oc:permissions xmlns:oc="http://owncloud.org/ns"/>
+ <oc:id xmlns:oc="http://owncloud.org/ns"/>
+ <oc:size xmlns:oc="http://owncloud.org/ns"/>
+ <oc:favorite xmlns:oc="http://owncloud.org/ns"/>
+ </d:prop>
+ </d:select>
+ <d:from>
+ <d:scope>
+ <d:href>/files/naofumi</d:href>
+ <d:depth>infinity</d:depth>
+ </d:scope>
+ </d:from>
+ <d:where><d:like><d:prop/><d:literal/></d:like></d:where>
+ <d:orderby/>
+ </d:basicsearch>
+</d:searchrequest>
\ No newline at end of file
Index: src/XML/Literal.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- src/XML/Literal.php (date 1572862197000)
+++ src/XML/Literal.php (date 1572959172047)
@@ -29,7 +29,12 @@
static function xmlDeserialize(Reader $reader): Literal {
$literal = new self();
- $literal->value = $reader->readText();
+ if ($reader->isEmptyElement) {
+ $literal->value = '';
+ } else {
+ $literal->value = $reader->readText();
+ }
+
$reader->read();
return $literal;
Index: tests/SearchPluginTest.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- tests/SearchPluginTest.php (date 1572862197000)
+++ tests/SearchPluginTest.php (date 1572959138503)
@@ -526,4 +526,39 @@
$this->assertEquals(400, $response->getStatus());
}
+ public function testSearchQueryInfiniteLoopEmptyLiteral() {
+ $this->searchBackend->expects($this->any())
+ ->method('getArbiterPath')
+ ->willReturn('foo');
+
+ $plugin = new SearchPlugin($this->searchBackend);
+ $server = new Server();
+ $plugin->initialize($server);
+
+ $request = new Request('SEARCH', '/index.php/foo', [
+ 'Content-Type' => 'text/xml'
+ ]);
+ $request->setBaseUrl('/index.php');
+ $request->setBody(fopen(__DIR__ . '/infiniteloopemptyliteral.xml', 'r'));
+ $response = new Response();
+
+ $this->searchBackend->expects($this->any())
+ ->method('isValidScope')
+ ->willReturn(true);
+
+ $this->searchBackend->expects($this->never())
+ ->method('search');
+
+ $this->searchBackend->expects($this->any())
+ ->method('getPropertyDefinitionsForScope')
+ ->willReturn([
+ new SearchPropertyDefinition('{http://ns.nextcloud.com:}fileid', false, true, true, SearchPropertyDefinition::DATATYPE_NONNEGATIVE_INTEGER),
+ new SearchPropertyDefinition('{DAV:}getcontentlength', true, true, true, SearchPropertyDefinition::DATATYPE_NONNEGATIVE_INTEGER),
+ ]);
+
+ $plugin->searchHandler($request, $response);
+
+ $this->assertEquals(400, $response->getStatus());
+ }
+
}Would you mind to update your patch at icewind1991/SearchDAV like above? It's basically your patch and a test to reproduce the situation. We probably should report this upstream to sabre/xml.
Thanks again for your help :+1:
NaofumiHonda
commented
5 years ago @kesselb the above patch and its related test have been merged into icewind1991/SearchDAV thanks to your kind help. If possible, could you complete the rest, that is, patch nextcloud/3rdparty (2) and update nextcloud/server (3)?
Jaxom99
commented
4 years ago I recently encountered this issue on my production server (NC 15, nginx, debian 9) so i'll look into your proposals...
mjanssens
commented
4 years ago NC 15 is end of life: https://github.com/nextcloud/server/wiki/Maintenance-and-Release-Schedule
And the fix was merged into 18, backported to 17 and 16 https://github.com/nextcloud/server/pull/17905
So I think the best way to get around it would be upgrading Nextcloud and perhaps Debian too.
Jaxom99
commented
4 years ago Okay so it has indeed been backported 😌 🙏 Thanks for your quick reply! I'm lagging behind in terms of version, hopeful an upgrade is on its way.
mjanssens
commented
4 years ago No problem, however, normally Nextcloud updates don't install themselves automatically, so an action on your part would be needed. But that's not something to discuss in a bug report. If you need further help the forums would be a better place I think, you can find them here: https://help.nextcloud.com
fcimeson
commented
4 years ago Hello, I seem to be experiencing this issue on NextCloud 19 (I'm using duplicacy via WebDav). Is it possible there is a still a bug or is this something else?
PS. I've resorted to SFTP for my transfer protocol but I would prefer to open up less services to the wild.
Steps to reproduce
Expected behaviour
Noting :)
Actual behaviour
Server is going on high load on one of the cores - if waiting long enough it can spread to other cores as well - I assume that each error leads to high load on single CPU
Server configuration
Operating system: Gentoo Linux with kernel 4.19.72
Web server: Apache 2.4.4.1
Database: mysql Ver 15.1 Distrib 10.2.22-MariaDB
PHP version: 7.2.22
Nextcloud version: (see Nextcloud admin page) 16.0.5
Updated from an older Nextcloud/ownCloud or fresh install: Updated from 16.0.4
Where did you install Nextcloud from: Initial install was from gentoo portage but later was updated using built-in webupdate
Signing status:
Signing status
No errors have been found.List of activated apps:
App list
Enabled: - accessibility: 1.2.0 - activity: 2.9.1 - bookmarks: 2.2.0 - bruteforcesettings: 1.4.0 - calendar: 1.7.1 - camerarawpreviews: 0.7.0 - cloud_federation_api: 0.2.0 - comments: 1.6.0 - contacts: 3.1.6 - dav: 1.9.2 - deck: 0.6.6 - federatedfilesharing: 1.6.0 - federation: 1.6.0 - files: 1.11.0 - files_mindmap: 0.0.15 - files_pdfviewer: 1.5.0 - files_rightclick: 0.15.1 - files_sharing: 1.8.0 - files_texteditor: 2.8.0 - files_trashbin: 1.6.0 - files_versions: 1.9.0 - files_videoplayer: 1.5.0 - firstrunwizard: 2.5.0 - gallery: 18.3.0 - logreader: 2.1.0 - lookup_server_connector: 1.4.0 - metadata: 0.10.0 - nextcloud_announcements: 1.5.0 - notifications: 2.4.1 - oauth2: 1.4.2 - password_policy: 1.6.0 - polls: 0.10.2 - previewgenerator: 2.1.0 - privacy: 1.0.0 - provisioning_api: 1.6.0 - recommendations: 0.4.0 - richdocuments: 3.4.2 - serverinfo: 1.6.0 - sharebymail: 1.6.0 - spreed: 6.0.4 - support: 1.0.0 - survey_client: 1.4.0 - systemtags: 1.6.0 - theming: 1.7.0 - twofactor_backupcodes: 1.5.0 - updatenotification: 1.6.0 - viewer: 1.1.0 - workflowengine: 1.6.0 Disabled: - admin_audit - encryption - files_external - user_ldapNextcloud configuration:
Config report
{ "system": { "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "cloud.somedomain.com" ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "dbtype": "mysql", "version": "16.0.5.1", "overwrite.cli.url": "https:\/\/cloud.comedomain.com", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "mysql.utf8mb4": true, "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "maintenance": false, "memcache.local": "\\OC\\Memcache\\APCu", "redis": { "host": "***REMOVED SENSITIVE VALUE***", "port": 6379 }, "memcache.locking": "\\OC\\Memcache\\Redis", "mail_smtpmode": "smtp", "mail_sendmailmode": "smtp", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_domain": "***REMOVED SENSITIVE VALUE***", "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "mail_smtpport": "25", "theme": "", "loglevel": 2, "preview_libreoffice_path": "\/usr\/bin\/libreoffice", "enable_previews": true, "enabledPreviewProviders": [ "OC\\Preview\\TXT", "OC\\Preview\\MarkDown", "OC\\Preview\\PDF", "OC\\Preview\\Image", "OC\\Preview\\SVG", "OC\\Preview\\MP3", "OC\\Preview\\Movie", "OC\\Preview\\BMP", "OC\\Preview\\GIF", "OC\\Preview\\HEIC", "OC\\Preview\\JPEG", "OC\\Preview\\PNG", "OC\\Preview\\XBitmap" ] } }Are you using external storage, if yes which one: local/smb/sftp/... No
Are you using encryption: yes/no No
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/... No
Client configuration
Browser: Chrome 77.0.3865.90
Operating system: Windows 10
Logs
Web server error log
Web server error log
Nextcloud log (data/nextcloud.log)
Nextcloud log
{"reqId":"XZlvJQhoUxX8xwGWHu9tQgAAAEY","level":3,"time":"2019-10-06T04:35:50+00:00","remoteAddr":"someIP","user":"user_x","app":"PHP","method":"SEARCH","url":"\/remote.php\/dav","message":"Undefined offset: 0 at \/var\/www\/somedomain.com\/htdocs\/nextcloud\/3rdparty\/icewind\/searchdav\/src\/XML\/Operator.php#71","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.8.0","version":"16.0.5.1"} {"reqId":"XZlvJS7ZfjNWPg5BWl6jYAAAAA0","level":3,"time":"2019-10-06T04:35:50+00:00","remoteAddr":"someIP","user":"user_x","app":"PHP","method":"SEARCH","url":"\/remote.php\/dav","message":"Undefined offset: 0 at \/var\/www\/somedomain.com\/htdocs\/nextcloud\/3rdparty\/icewind\/searchdav\/src\/XML\/Operator.php#71","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.8.0","version":"16.0.5.1"} {"reqId":"XZlvTghoUxX8xwGWHu9tQwAAAEQ","level":3,"time":"2019-10-06T04:36:30+00:00","remoteAddr":"someIP","user":"user_x","app":"PHP","method":"SEARCH","url":"\/remote.php\/dav","message":"Undefined offset: 0 at \/var\/www\/somedomain.com\/htdocs\/nextcloud\/3rdparty\/icewind\/searchdav\/src\/XML\/Operator.php#71","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.8.0","version":"16.0.5.1"} {"reqId":"XZlvUy7ZfjNWPg5BWl6jaQAAAAY","level":3,"time":"2019-10-06T04:36:35+00:00","remoteAddr":"someIP","user":"user_x","app":"PHP","method":"SEARCH","url":"\/remote.php\/dav","message":"Undefined offset: 0 at \/var\/www\/somedomain.com\/htdocs\/nextcloud\/3rdparty\/icewind\/searchdav\/src\/XML\/Operator.php#71","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.8.0","version":"16.0.5.1"} {"reqId":"XZlvdi7ZfjNWPg5BWl6jbQAAABY","level":3,"time":"2019-10-06T04:37:11+00:00","remoteAddr":"someIP","user":"user_x","app":"PHP","method":"SEARCH","url":"\/remote.php\/dav","message":"Undefined offset: 0 at \/var\/www\/somedomain.com\/htdocs\/nextcloud\/3rdparty\/icewind\/searchdav\/src\/XML\/Operator.php#71","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.8.0","version":"16.0.5.1"}Browser log
Browser log
Not applicable