search

owncloud / core

:cloud: ownCloud web server core (Files, DAV, etc.)
https://owncloud.com
GNU Affero General Public License v3.0
8.38k stars 2.05k forks source link

Unit tests failing on Ubuntu 20.04 with PHP 7.4 #38348

Closed phil-davis closed 2 years ago

phil-davis commented 3 years ago

Yesterday the owncloudci/php:7.4 docker images was updated to use Ubuntu 20.04 as the base image (was previously Ubuntu 18.04). See PR https://github.com/owncloud-ci/php/pull/128

Now some unit tests are failing. See demonstration in PR #38346

Maybe Ubuntu 20.04 is missing some things that were installed by default in Ubuntu 18.04, or...?

Investigate and fix (or revert the docker image change for now)

phil-davis commented 3 years ago

An example in the nightly CI - files_primary_s3 running the core unit tests with PHP 7.4 https://drone.owncloud.com/owncloud/files_primary_s3/2077/8/7

There were 8 failures:

1) Test\LargeFileHelperGetFileSizeTest::testGetFileSizeViaCurl with data set #0 ('/var/www/owncloud/server/test...em.txt', 446)
Failed asserting that null is identical to 446.

/var/www/owncloud/server/tests/lib/LargeFileHelperGetFileSizeTest.php:53

2) Test\LargeFileHelperGetFileSizeTest::testGetFileSizeViaCurl with data set #1 ('/var/www/owncloud/server/test...2).txt', 446)
Failed asserting that null is identical to 446.

/var/www/owncloud/server/tests/lib/LargeFileHelperGetFileSizeTest.php:53

3) Test\Preview\BitmapTest::testGetThumbnail with data set #0 (-68, -77)
Failed asserting that 0.77 matches expected 1.41.

/var/www/owncloud/server/tests/lib/Preview/Provider.php:155
/var/www/owncloud/server/tests/lib/Preview/Provider.php:104

4) Test\Preview\BitmapTest::testGetThumbnail with data set #1 (48, 24)
Failed asserting that 0.77 matches expected 1.41.

/var/www/owncloud/server/tests/lib/Preview/Provider.php:155
/var/www/owncloud/server/tests/lib/Preview/Provider.php:104

5) Test\Preview\BitmapTest::testGetThumbnail with data set #2 (-95, 73)
Failed asserting that 0.77 matches expected 1.41.

/var/www/owncloud/server/tests/lib/Preview/Provider.php:155
/var/www/owncloud/server/tests/lib/Preview/Provider.php:104

6) Test\Preview\BitmapTest::testGetThumbnail with data set #3 (49, -34)
Failed asserting that 0.77 matches expected 1.41.

/var/www/owncloud/server/tests/lib/Preview/Provider.php:155
/var/www/owncloud/server/tests/lib/Preview/Provider.php:104

7) Tests\Core\Templates\TemplatesTest::test403
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
 '<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">\n
-<html><body><ul><li class="error">\n
-       Access forbidden<br><p class="hint"></p>\n
-</li></ul></body></html>\n
+<html><body><ul>\n
+<li class="error">\n
+       Access forbidden<br>\n
+       <p class="hint"></p>\n
+   </li>\n
+</ul></body></html>\n
 '

/var/www/owncloud/server/tests/lib/TestCase.php:506
/var/www/owncloud/server/tests/lib/TestCase.php:484
/var/www/owncloud/server/tests/Core/Templates/TemplatesTest.php:9

8) Tests\Core\Templates\TemplatesTest::test404
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
 '<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">\n
-<html><body><ul><li class="error">\n
-           File not found<br><p class="hint">The specified document has not been found on the server.</p>\n
-<p class="hint"><a href="/owncloud/">You can click here to return to ownCloud.</a></p>\n
-       </li></ul></body></html>\n
+<html><body><ul>\n
+<li class="error">\n
+           File not found<br>\n
+           <p class="hint">The specified document has not been found on the server.</p>\n
+           <p class="hint"><a href="/owncloud/">You can click here to return to ownCloud.</a></p>\n
+       </li>\n
+   </ul></body></html>\n
 '

/var/www/owncloud/server/tests/lib/TestCase.php:506
/var/www/owncloud/server/tests/lib/TestCase.php:484
/var/www/owncloud/server/tests/Core/Templates/TemplatesTest.php:16

--
phil-davis commented 3 years ago

I have reverted the changes to owncloudci/php:7.4 so that existing CI can be "great again" - PR https://github.com/owncloud-ci/php/pull/129

phil-davis commented 3 years ago

PR https://github.com/owncloud-ci/php/pull/130 has created owncloudci/php:7.4-ubuntu20.04 docker image. I will use that to test the phpunit, Behat, etc tests and get it passing properly before putting Ubuntu 20.04 into the "real" owncloudci/php:7.4 docker image.

VicDeo commented 3 years ago

Test\LargeFileHelperGetFileSizeTest::testGetFileSizeViaCurl seems to be a php_curl regression:

# curl -I file:///home/deo/public_html/oc-tmp/tests/data/lorem.txt
Content-Length: 446
Accept-ranges: bytes
Last-Modified: Mon, 17 Jun 2013 07:25:11 GMT

Headers are here. But the script 

<?php
$ch = \curl_init("file:///home/deo/public_html/oc-tmp/tests/data/lorem.txt");
\curl_setopt_array(
        $ch,
        [
                CURLOPT_NOBODY => true,
                CURLOPT_RETURNTRANSFER => true,
                CURLOPT_HEADER => true
        ]
);
$data = \curl_exec($ch);
\curl_close($ch);
var_dump($data);

returns no headers: string(0) ""

I'm pretty sure file protocol is enabled - after removing CURLOPT_NOBODY => true, the file content is dumped.

and it does return headers when the resource is changed from file:///home/deo/public_html/oc-tmp/tests/data/lorem.txt into e.g. http://localhost/oc/core/js/login.js?v=988646cb965e528fd57397e8b9b4468f

So CLI curl returns headers for the file protocol while php curl extendsion doesn't.

dpakach commented 3 years ago

@phil-davis this one fails for me even with php 7.3 when running locally, so seems like the problem is with ubuntu 20.04.

phil-davis commented 2 years ago

Note: this needs doing in the next month or two, because it will be good to run all CI with Ubuntu 20.04 before PHP 8.* gets progress.

phil-davis commented 2 years ago

So CLI curl returns headers for the file protocol while php curl extension doesn't.

I found this which reports the same problem:

https://stackoverflow.com/questions/59709625/curlfile-scheme-not-returns-headers-in-php-7-3

I am starting an Ubuntu 18.04 VM so that I can try it with PHP 7.4 etc.

Update:

This cURL code to get the headers works on Ubuntu 18.04 with any of the PHP that I tried - 7.2 7.3 and 7.4

Exactly the same code fails to return any header information on Ubuntu 20.04 with any version of PHP - I tried 5.6 7.0 7.1 7.2 7.3 7.4 8.0 8.1. So I guess that there is something different about libcurl.

On my systems: Ubuntu 18.04

$ curl-config --version
libcurl 7.58.0

Ubuntu 20.04

$ curl-config --version
libcurl 7.68.0

This issue looks like where the regressions was reported: https://github.com/curl/curl/issues/3083 And fixed in commit https://github.com/curl/curl/commit/e50a2002bd450a4800a165d2874ed79c95b33a07 But that commit seems to be included in curl 7.62 - so 7.68 should have the fix - what is going on???

phil-davis commented 2 years ago

I have confirmed that CURLOPT_HEADER for file:// works on Ubuntu 18.04 (in a VM that I installed) It does not work on Ubuntu 20.04 or Ubuntu 22.04.

I found a way to use CURLOPT_HEADERFUNCTION and that works on all Ubuntu 18.04 20.04 and 22.04 - see PR #40065

phil-davis commented 2 years ago

PRs #40054 #40059 and #40065 fix all the issues so that the code, tests and CI all pass on both Ubuntu 18.04 and Ubuntu 20.04.

After those are reviewed and merged, then I will make another PR to run PHP tests with 7.4-ubuntu20.04, and it should pass.

phil-davis commented 2 years ago

Currently blocked waiting for core 10.10.0 final release. Then I can merge the waiting PRs and continue...

kiranparajuli589 commented 2 years ago

@phil-davis owncloud core 10.10.0 has been released. can we move forward on this?

phil-davis commented 2 years ago

Now blocked by #40054 - carddav and caldav tests need python2, but Ubuntu 22.04 has python3 by default.

@xoxys has said that he will update those test suites so that they run with python3: https://github.com/owncloud/core/pull/40054#issuecomment-1122167023

The PHP unit tests in master now pass on Ubuntu 22.04

phil-davis commented 2 years ago

PR #40054 was merged. It explicitly uses python2 in the caldav/carddav test scripts. We will stay with that for now.

The next step is testing with Ubuntu 22.04 - see PR #40123

After that is passing, I will think about what to do for regular CI. Maybe we will wait until a few other infrastructure things have happened - for example, the guzzle v7 update and dropping PHP 7.3 support. Then we can decide what to do about the test matrix for PHP 7.4.

kiranparajuli589 commented 2 years ago

closing. see https://github.com/owncloud/files_primary_s3/issues/613#issuecomment-1278846592