search

nextcloud / richdocuments

📑 Collabora Online for Nextcloud
https://nextcloud.com/collaboraonline
355 stars 116 forks source link

RichDocuments built-in CODE causes significant slowdown/timeout issues #1282

Closed eganonoa closed 4 months ago

eganonoa commented 3 years ago

Over the last couple of weeks Snap and Docker versions of Nextcloud have updated to version 2.0.2 (snap on Friday). After the upgrade the connection to Nextcloud, particularly via the web-interface, slows to a crawl and/or times out. It appears to be caused by richdocuments/collabora.

What is strange about this is that you cannot see easily identify the problem: basic analytics show no major increase in load on the CPU, Ram or I/O, the logs don't immediately show anything out of the ordinary. If you open top/htop on the server, however, and observe for a while you can see big, temporary spikes where available CPU's are maxed out by richdocuments/the Appimage related to Collabora and the Code server. When you go to the admin settings under collabora you can see an error that says that it cannot establish a connection to the in-built CODE server.

The performance problem can be fixed by disabling both the collabora and code server apps: either in the web interface if you can get there through the performance issues, or in the server terminal (sudo snap run nextcloud.occ app:disable richdocuments ; sudo snap run nextcloud.occ app:disable richdocumentscore). After doing that normal service is resumed entirely across the Nextcloud server.

This indicates that the failure to establish a connection to the inbuilt CODE server is the underlying problem and that there is something that happens when upgrading from Nextcloud v. 2.0.1 to 2.0.2 that causes it to break. Strangely, however, this is an issue that only appears to be present in Nextcloud Snap (where slowdown occurs) and Docker (where timeouts occur) and doesn't appear to have been reported by users of the regular Nextcloud: see threads on Nextcloud Community Forum (https://help.nextcloud.com/t/nextcloud-20-0-2snap2-interface-incredibly-slow/100304), Nextcloud Snap Github (https://github.com/nextcloud/nextcloud-snap/issues/1569), and Nextcloud Reddit (https://www.reddit.com/r/NextCloud/comments/k7b1of/extremely_slow_interface_nextcloud_snap/).

calbav commented 3 years ago

Hello,

my Nextcloud instance is experiencing the same issue after version upgrade, although in my case it is enough to disable the Collabora Online - Built-in CODE Server app to solve the performance problem.

Environment: Ubuntu 18.04.5 LTS snap 20.0.2snap2

Thank you for your time and effort in looking into this matter. Should you need additional information please let me know.

kyrofa commented 3 years ago

Confirmed, this makes the snap unusable. Nothing interesting in the logs during usage that doesn't relate to collabora. When trying to open a document I see this though. Not sure if it's helpful:

{"reqId":"FX5iHBU58fJlpkNjGQsY","level":3,"time":"2020-12-07T17:38:50+00:00","remoteAddr":"127.0.0.1","user":"admin","app":"richdocuments","method":"GET","url":"/index.php/apps/richdocuments/index?fileId=11&requesttoken=AEyf86%2BIbFplFbZnaNJpmAjUL4WrtY98go6EYe6v%2BNM%3D%3ANDjKqtXyKw8dW9QFX7gBt2nlH%2FTYhd447v7HL6jXteE%3D","message":{"Exception":"Exception","Message":"Could not find urlsrc in WOPI","Code":0,"Trace":[{"file":"/var/snap/nextcloud/24828/nextcloud/extra-apps/richdocuments/lib/TokenManager.php","line":210,"function":"getUrlSrc","class":"OCA\\Richdocuments\\WOPI\\Parser","type":"->","args":["application/vnd.openxmlformats-officedocument.wordprocessingml.document"]},{"file":"/var/snap/nextcloud/24828/nextcloud/extra-apps/richdocuments/lib/Controller/DocumentController.php","line":256,"function":"getToken","class":"OCA\\Richdocuments\\TokenManager","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/snap/nextcloud/24828/htdocs/lib/private/AppFramework/Http/Dispatcher.php","line":169,"function":"index","class":"OCA\\Richdocuments\\Controller\\DocumentController","type":"->","args":["*** sensitive parameter replaced ***",null]},{"file":"/snap/nextcloud/24828/htdocs/lib/private/AppFramework/Http/Dispatcher.php","line":100,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OCA\\Richdocuments\\Controller\\DocumentController"},"index"]},{"file":"/snap/nextcloud/24828/htdocs/lib/private/AppFramework/App.php","line":152,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OCA\\Richdocuments\\Controller\\DocumentController"},"index"]},{"file":"/snap/nextcloud/24828/htdocs/lib/private/Route/Router.php","line":308,"function":"main","class":"OC\\AppFramework\\App","type":"::","args":["OCA\\Richdocuments\\Controller\\DocumentController","index",{"__class__":"OC\\AppFramework\\DependencyInjection\\DIContainer"},{"action":null,"_route":"richdocuments.document.index"}]},{"file":"/snap/nextcloud/24828/htdocs/lib/base.php","line":1008,"function":"match","class":"OC\\Route\\Router","type":"->","args":["/apps/richdocuments/index"]},{"file":"/snap/nextcloud/24828/htdocs/index.php","line":37,"function":"handleRequest","class":"OC","type":"::","args":[]}],"File":"/var/snap/nextcloud/24828/nextcloud/extra-apps/richdocuments/lib/WOPI/Parser.php","Line":54,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0","version":"20.0.2.2"}

Note that I've been trying to open that document for about 5 minutes now, it just keeps spinning.

kyrofa commented 3 years ago

Got a lot of these in my PHP error log:

[07-Dec-2020 17:38:41 UTC] PHP Warning:  file_get_contents(http://localhost:9982/hosting/capabilities): failed to open stream: HTTP request failed! in /var/snap/nextcloud/24828/nextcloud/extra-apps/richdocumentscode/proxy.php on line 208
[07-Dec-2020 17:38:43 UTC] PHP Warning:  file_get_contents(http://localhost:9982/hosting/capabilities): failed to open stream: HTTP request failed! in /var/snap/nextcloud/24828/nextcloud/extra-apps/richdocumentscode/proxy.php on line 208
[07-Dec-2020 17:38:45 UTC] PHP Warning:  file_get_contents(http://localhost:9982/hosting/capabilities): failed to open stream: HTTP request failed! in /var/snap/nextcloud/24828/nextcloud/extra-apps/richdocumentscode/proxy.php on line 208
[07-Dec-2020 17:38:47 UTC] PHP Warning:  file_get_contents(http://localhost:9982/hosting/capabilities): failed to open stream: Connection timed out in /var/snap/nextcloud/24828/nextcloud/extra-apps/richdocumentscode/proxy.php on line 208

And this type of thing in PHP-FPM error log:

[07-Dec-2020 09:40:44] WARNING: [pool www] child 17933 said into stderr: "File exists and file size matches, skipping"
[07-Dec-2020 09:40:44] WARNING: [pool www] child 17933 said into stderr: "Logging at warning level to file: /tmp/loolwsd.wQQ51O50TX/loolwsd.log"
[07-Dec-2020 09:40:44] WARNING: [pool www] child 15589 said into stderr: "Logging at warning level to file: /tmp/loolwsd.4qc5ViOkqZ/loolwsd.log"
[07-Dec-2020 09:40:44] WARNING: [pool www] child 17933 said into stderr: "Failed to listen on Server port(s) (9982-9982). Exiting."
[07-Dec-2020 09:40:45] WARNING: [pool www] child 15589 said into stderr: "Failed to listen on Server port(s) (9982-9982). Exiting."
[07-Dec-2020 09:40:48] WARNING: [pool www] child 17933 said into stderr: "/tmp/appimage_extracted_9bc52a5da15b4853d238ef00019d6a4d/opt/collaboraoffice6.4/share/autotext/es/standard.bau: ftw error: No such file or directory"
[07-Dec-2020 09:40:48] WARNING: [pool www] child 17933 said into stderr: "Failed to clean up cache directory"
juliusknorr commented 3 years ago

cc @timar @eszkadev Seems caused or related to richdocumentscode. Any idea?

cedartop commented 3 years ago

I found something on the nextcloud forum:

hpilkington gave the tip:

Confirmed sudo snap run nextcloud.occ app:disable richdocuments && sudo snap run nextcloud.occ app:disable richdocumentscore resolved my issues.

After having done that, the web interface turns fast again. I hope there will be a fix soon for collabora...

eszkadev commented 3 years ago

Confirmed, this makes the snap unusable. Nothing interesting in the logs during usage that doesn't relate to collabora. When trying to open a document I see this though. Not sure if it's helpful:

{"reqId":"FX5iHBU58fJlpkNjGQsY","level":3,"time":"2020-12-07T17:38:50+00:00","remoteAddr":"127.0.0.1","user":"admin","app":"richdocuments","method":"GET","url":"/index.php/apps/richdocuments/index?fileId=11&requesttoken=AEyf86%2BIbFplFbZnaNJpmAjUL4WrtY98go6EYe6v%2BNM%3D%3ANDjKqtXyKw8dW9QFX7gBt2nlH%2FTYhd447v7HL6jXteE%3D","message":{"Exception":"Exception","Message":"Could not find urlsrc in WOPI","Code":0,"Trace":[{"file":"/var/snap/nextcloud/24828/nextcloud/extra-apps/richdocuments/lib/TokenManager.php","line":210,"function":"getUrlSrc","class":"OCA\\Richdocuments\\WOPI\\Parser","type":"->","args":["application/vnd.openxmlformats-officedocument.wordprocessingml.document"]},{"file":"/var/snap/nextcloud/24828/nextcloud/extra-apps/richdocuments/lib/Controller/DocumentController.php","line":256,"function":"getToken","class":"OCA\\Richdocuments\\TokenManager","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/snap/nextcloud/24828/htdocs/lib/private/AppFramework/Http/Dispatcher.php","line":169,"function":"index","class":"OCA\\Richdocuments\\Controller\\DocumentController","type":"->","args":["*** sensitive parameter replaced ***",null]},{"file":"/snap/nextcloud/24828/htdocs/lib/private/AppFramework/Http/Dispatcher.php","line":100,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OCA\\Richdocuments\\Controller\\DocumentController"},"index"]},{"file":"/snap/nextcloud/24828/htdocs/lib/private/AppFramework/App.php","line":152,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OCA\\Richdocuments\\Controller\\DocumentController"},"index"]},{"file":"/snap/nextcloud/24828/htdocs/lib/private/Route/Router.php","line":308,"function":"main","class":"OC\\AppFramework\\App","type":"::","args":["OCA\\Richdocuments\\Controller\\DocumentController","index",{"__class__":"OC\\AppFramework\\DependencyInjection\\DIContainer"},{"action":null,"_route":"richdocuments.document.index"}]},{"file":"/snap/nextcloud/24828/htdocs/lib/base.php","line":1008,"function":"match","class":"OC\\Route\\Router","type":"->","args":["/apps/richdocuments/index"]},{"file":"/snap/nextcloud/24828/htdocs/index.php","line":37,"function":"handleRequest","class":"OC","type":"::","args":[]}],"File":"/var/snap/nextcloud/24828/nextcloud/extra-apps/richdocuments/lib/WOPI/Parser.php","Line":54,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0","version":"20.0.2.2"}

Note that I've been trying to open that document for about 5 minutes now, it just keeps spinning.

@juliushaertl I noticed similar issue probably introduced by: https://github.com/nextcloud/richdocuments/commit/6b3d3f1 Request for "/index.php/apps/richdocuments/index" hangs in DocumentController.php because we need discovery while getting token and after mentioned patch there is no local cache which may be used so we are trying to connect to richdocumentscode which may be not started yet. But it should be fixed now in my last commits by increasing timeout when it is starting.

thebearon commented 3 years ago

Looked into the snap case with @timar. The findings so far (spoiler: the interesting part is at the end): /var/snap/nextcloud/24828/log/php-fpm_errors.log has relevant content.

[08-Dec-2020 19:56:50] NOTICE: fpm is running, pid 3042
[08-Dec-2020 19:56:50] NOTICE: ready to handle connections

So far so good.

[08-Dec-2020 22:08:46] WARNING: [pool www] child 20232 said into stderr: "sh: 1: pidof: Permission denied"
[08-Dec-2020 22:08:46] WARNING: [pool www] child 20232 said into stderr: "sh: 1: disown: not found"

These aren't ideal, and might be worth investigating some other time (service may fail to kill the old version, and start the new one after an update), but isn't relevant now.

[08-Dec-2020 22:08:46] WARNING: [pool www] child 20232 said into stderr: "dlopen(): error loading libfuse.so.2"
[08-Dec-2020 22:08:46] WARNING: [pool www] child 20232 said into stderr: "AppImages require FUSE to run. "
[08-Dec-2020 22:08:46] WARNING: [pool www] child 20232 said into stderr: "You might still be able to extract the contents of this AppImage "
[08-Dec-2020 22:08:46] WARNING: [pool www] child 20232 said into stderr: "if you run it with the --appimage-extract option. "
[08-Dec-2020 22:08:46] WARNING: [pool www] child 20232 said into stderr: "See https://github.com/AppImage/AppImageKit/wiki/FUSE "
[08-Dec-2020 22:08:46] WARNING: [pool www] child 20232 said into stderr: "for more information"

This is expected, there's no libfuse2 in snap, content of the AppImage is extracted to /tmp/snap.nextcloud/tmp/.

[08-Dec-2020 22:08:51] WARNING: [pool www] child 20232 said into stderr: "Logging at warning level to file: /tmp/loolwsd.2y1F0DbTKo/loolwsd.log"

loolwsd has started.

[08-Dec-2020 22:08:51] WARNING: [pool www] child 20232 said into stderr: "Error: loolforkit incorrect user-name: root - aborting"
[08-Dec-2020 22:08:51] WARNING: [pool www] child 20232 said into stderr: "Security: --disable-lool-user-checking failed, loolforkit has some capabilities set."

^ This indicates the problem. Another indicator: http://<snap host>/extra-apps/richdocumentscode/proxy.php?status stays at {"status":"starting"} instead of OK.

Only happens with version 6.4.202, while the previous 6.4.14 version is okay. https://github.com/CollaboraOnline/online/commit/264008618fd45c41e8b518d30cd56ba49dccf9e2 altered a check so !hasCorrectUID("loolforkit") is always called, and emits the first error message, and added another check that causes the app to exit with error after the second error message. I don't know if that's all there is to this problem, but looks like a good starting point. @kendy, @Ashod, what are your thoughts on that?

kendy commented 3 years ago

@thebearon: I'd suggest removing the capabilities from the binaries when packaging this, that should make it working again.

kendy commented 3 years ago

@juliushaertl: So we have investigated this more, and actually the problem is that the AppImage is run as root - who of course by definition has all the capabilities, so we refuse to start... Is it really necessary to run us as root?

timar commented 3 years ago

@juliushaertl: So we have investigated this more, and actually the problem is that the AppImage is run as root - who of course by definition has all the capabilities, so we refuse to start... Is it really necessary to run us as root?

Actually, probably @kyrofa can answer this question.

juliusknorr commented 3 years ago

The loolwsd command should be ran as the same user that the webserver is using since richdocuments code is just calling PHP exec(), so I assume that this is rather snap specific.

kyrofa commented 3 years ago

Is it really necessary to run us as root?

Yes, as we've discussed before, daemons in snaps run as root, although note that root in snaps is still confined so there are no special capabilities granted.

eganonoa commented 3 years ago

@juliushaertl: So we have investigated this more, and actually the problem is that the AppImage is run as root - who of course by definition has all the capabilities, so we refuse to start... Is it really necessary to run us as root?

Important to note that this issue isn't limited to snap, but has been reported by Docker image users also (though perhaps only those that installed everything as root?). Also note that one user on the Community Forum has said that upgrading to the edge release fixes the issue: https://help.nextcloud.com/t/nextcloud-20-0-2snap2-interface-incredibly-slow/100304/27. It's just one user, of course.

LouisVallat commented 3 years ago

@juliushaertl: So we have investigated this more, and actually the problem is that the AppImage is run as root - who of course by definition has all the capabilities, so we refuse to start... Is it really necessary to run us as root?

Important to note that this issue isn't limited to snap, but has been reported by Docker image users also (though perhaps only those that installed everything as root?). Also note that one user on the Community Forum has said that upgrading to the edge release fixes the issue: https://help.nextcloud.com/t/nextcloud-20-0-2snap2-interface-incredibly-slow/100304/27. It's just one user, of course.

I use Nextcloud, latest version & fresh install, in Docker with the latest image, which corresponds to the latest stable release. I can't use richdocuments & richdocumentscode because of how slow and how sluggish my instance become when these apps are installed.

SimplyCorbett commented 3 years ago

This also happens if nextcloud is installed on bare metal. It's not dependent on docker or snap.

I have a collabora container on another VM, but nextcloud hasn't connected to it yet.

Unfortunately in my case with nextcloud 20.04 the entire install errors out with richdocuments enabled so I can't configure it. With 20.02 I could at least load into the app after 45-60 seconds and once configured the issue went away.

That's not possible on 20.04.

aaylnx commented 3 years ago

I really hope a fix for this is forthcoming soon. I definitely is a big showstopper for me.

timar commented 3 years ago

richdocumentcode 6.4.203 has been released. It fixes the issue, i.e. now we let loolforkit run as root.

eganonoa commented 3 years ago

richdocumentcode 6.4.203 has been released. It fixes the issue, i.e. now we let loolforkit run as root.

I'm afraid I cannot confirm the fix. I upgraded to 6.4.203 and re-enabled the CODE server and Collabora Online and it has, unfortunately, knocked out my Nextcloud server (latest snap version) again. Disabling them brings performance right back.

szjozsef commented 3 years ago

I can confirm, it still not working, logs:

xx.xx.xx.xx - - [23/Dec/2020:07:37:57 +0000] "GET /custom_apps/richdocumentscode/proxy.php?req=/hosting/capabilities HTTP/1.1" 200 628 "-" "Nextcloud Server Crawler" dlopen(): error loading libfuse.so.2

AppImages require FUSE to run. You might still be able to extract the contents of this AppImage if you run it with the --appimage-extract option. See https://github.com/AppImage/AppImageKit/wiki/FUSE for more information

Logging at warning level to file: /tmp/loolwsd.k6vtOuTaTH/loolwsd.log Error: loolforkit incorrect user-name, other than 'lool' - aborting Security: --disable-lool-user-checking failed, loolforkit has some capabilities set.

nextcloud 20.0.4 (docker) Collabora Online 3.7.11 Collabora Online - Built-in CODE Server 6.4.203

calbav commented 3 years ago

The fix seems to work on snap.

Although everything appears to be working a new notification appears (maybe unrelated?):

"Invalid private key for encryption app. Please update your private key password in your personal settings to recover access to your encrypted files."

Also many new errors are logged, mainly:

[encryption] Warning: OC\ServerNotAvailableException: Legacy cipher is no longer supported! at <>

  1. /snap/nextcloud/24940/htdocs/apps/encryption/lib/Crypto/Crypt.php line 405 OCA\Encryption\Crypto\Crypt->getLegacyCipher()
  2. /snap/nextcloud/24940/htdocs/apps/encryption/lib/KeyManager.php line 404 OCA\Encryption\Crypto\Crypt->decryptPrivateKey(" sensitive parameters replaced ")
  3. /snap/nextcloud/24940/htdocs/apps/encryption/lib/Hooks/UserHooks.php line 183 OCA\Encryption\KeyManager->init(" sensitive parameter replaced ", " sensitive parameter replaced ")
  4. /snap/nextcloud/24940/htdocs/lib/private/legacy/OC_Hook.php line 110 OCA\Encryption\Hooks\UserHooks->login(" sensitive parameters replaced ")
  5. /snap/nextcloud/24940/htdocs/lib/private/Server.php line 581 OC_Hook::emit("OC_User", "post_login", " sensitive parameter replaced ")
  6. <> OC\Server->OC{closure}(" sensitive parameters replaced ")
  7. /snap/nextcloud/24940/htdocs/lib/private/Hooks/EmitterTrait.php line 101 call_user_func_array(Closure {}, ["*** sensitive ... "])
  8. /snap/nextcloud/24940/htdocs/lib/private/Hooks/PublicEmitter.php line 40 OC\Hooks\BasicEmitter->emit("\OC\User", "postRememberedLogin", ["*** sensitive ... "])
  9. /snap/nextcloud/24940/htdocs/lib/private/User/Session.php line 903 OC\Hooks\PublicEmitter->emit("\OC\User", "postRememberedLogin", ["*** sensitive ... "])
  10. /snap/nextcloud/24940/htdocs/lib/base.php line 1058 OC\User\Session->loginWithCookie(" sensitive parameters replaced ")
    1. /snap/nextcloud/24940/htdocs/lib/base.php line 997 OC::handleLogin(OC\AppFramework\Http\Request {})
    2. /snap/nextcloud/24940/htdocs/index.php line 37 OC::handleRequest()
LouisVallat commented 3 years ago

I tested this update this morning, using Docker, and it still acts the same : installing richdocuments & richdocumentscode result is veeerry slow instance, and it goes back to normal after uninstalling these apps.

eganonoa commented 3 years ago

The fix seems to work on snap.

Although everything appears to be working a new notification appears (maybe unrelated?):

"Invalid private key for encryption app. Please update your private key password in your personal settings to recover access to your encrypted files."

Also many new errors are logged, mainly:

[encryption] Warning: OC\ServerNotAvailableException: Legacy cipher is no longer supported! at <>

  1. /snap/nextcloud/24940/htdocs/apps/encryption/lib/Crypto/Crypt.php line 405 OCA\Encryption\Crypto\Crypt->getLegacyCipher()
  2. /snap/nextcloud/24940/htdocs/apps/encryption/lib/KeyManager.php line 404 OCA\Encryption\Crypto\Crypt->decryptPrivateKey(" sensitive parameters replaced ")
  3. /snap/nextcloud/24940/htdocs/apps/encryption/lib/Hooks/UserHooks.php line 183 OCA\Encryption\KeyManager->init(" sensitive parameter replaced ", " sensitive parameter replaced ")
  4. /snap/nextcloud/24940/htdocs/lib/private/legacy/OC_Hook.php line 110 OCA\Encryption\Hooks\UserHooks->login(" sensitive parameters replaced ")
  5. /snap/nextcloud/24940/htdocs/lib/private/Server.php line 581 OC_Hook::emit("OC_User", "post_login", " sensitive parameter replaced ")
  6. <> OC\Server->OC{closure}(" sensitive parameters replaced ")
  7. /snap/nextcloud/24940/htdocs/lib/private/Hooks/EmitterTrait.php line 101 call_user_func_array(Closure {}, ["*** sensitive ... "])
  8. /snap/nextcloud/24940/htdocs/lib/private/Hooks/PublicEmitter.php line 40 OC\Hooks\BasicEmitter->emit("\OC\User", "postRememberedLogin", ["*** sensitive ... "])
  9. /snap/nextcloud/24940/htdocs/lib/private/User/Session.php line 903 OC\Hooks\PublicEmitter->emit("\OC\User", "postRememberedLogin", ["*** sensitive ... "])
  10. /snap/nextcloud/24940/htdocs/lib/base.php line 1058 OC\User\Session->loginWithCookie(" sensitive parameters replaced ")
  11. /snap/nextcloud/24940/htdocs/lib/base.php line 997 OC::handleLogin(OC\AppFramework\Http\Request {})
  12. /snap/nextcloud/24940/htdocs/index.php line 37 OC::handleRequest()

Can I ask, which version of snap are you on. I'm on the latest stable version and re-enabling the CODE server and Collabora Online after updating the app had no positive impact at all, but just brought back the same problem.

calbav commented 3 years ago

Can I ask, which version of snap are you on. I'm on the latest stable version and re-enabling the CODE server and Collabora Online after updating the app had no positive impact at all, but just brought back the same problem.

Hi, sure. Here is the output from snap list:

Name Version Rev Tracking Publisher Notes nextcloud 20.0.3snap1 24940 latest/stable nextcloud* -

SimplyCorbett commented 3 years ago

Can you test on 20.0.4 instead of 20.0.3 @calbav

calbav commented 3 years ago

@SimplyCorbett gladly, can you tell me how?

Would that be:

$ sudo snap refresh nextcloud --channel=<what channel? latest/beta?>

kyrofa commented 3 years ago

@calbav sudo snap refresh --beta nextcloud will get you on 20.0.4. It will be released to stable shortly.

avimanyu786 commented 3 years ago

I can confirm, it still not working, logs:

xx.xx.xx.xx - - [23/Dec/2020:07:37:57 +0000] "GET /custom_apps/richdocumentscode/proxy.php?req=/hosting/capabilities HTTP/1.1" 200 628 "-" "Nextcloud Server Crawler" dlopen(): error loading libfuse.so.2

AppImages require FUSE to run. You might still be able to extract the contents of this AppImage if you run it with the --appimage-extract option. See https://github.com/AppImage/AppImageKit/wiki/FUSE for more information

Logging at warning level to file: /tmp/loolwsd.k6vtOuTaTH/loolwsd.log Error: loolforkit incorrect user-name, other than 'lool' - aborting Security: --disable-lool-user-checking failed, loolforkit has some capabilities set.

nextcloud 20.0.4 (docker) Collabora Online 3.7.11 Collabora Online - Built-in CODE Server 6.4.203

Just the same as all three! Gets fixed as soon as I disable Collabora Online 3.7.11.

calbav commented 3 years ago

Thanks @kyrofa,

I tested 20.0.4snap1 (25275 latest/beta) and the problem persists. Again it is enough to disable the Collabora Online - Built-in CODE Server app to regain performance.

Unfortunately upon reverting to 20.0.3snap1 (24940 latest/stable) the issue revived. This time it wasn't enough to disable the Collabora Online - Built-in CODE Server app to regain the performance, I also had to disable the Collabora Online app (as for everyone else using snap in the thread) to regain the performance.

vawaver commented 3 years ago

I tested NC 20.0.4 snap stable today and there is no progress. Collabora Online 3.7.11 - installed Collabora Online - Built-in CODE Server 6.4.203 installed Still must be disabled to have fast access to server files. If Collabora Online is enabled - still a problem persists.

sliwowitz commented 3 years ago

I have the same problem on a Gentoo installation: apache + php-fpm + postgres, Nextcloud 20.0.4. Nextcloud was installed via ebuild so the source lives in the server's main /var/www. Therefore, this problem is not limited to snap or docker installations.

I have to disable the built-in CODE server, otherwise Nextcloud is unbearably slow. This doesn't affect any other apps in the server (php or not), and there's no unusual CPU load, memory consumption, or disk activity. It's just that all Nextcloud pages take forever to load. I don't know when it started - I used to work with it without problems some time ago, but now I've had this problem for at least a month.

SimplyCorbett commented 3 years ago

otherwise Nextcloud is unbearably slow.

20.0.4 doesn't work at all with richdocuments enabled.

sliwowitz commented 3 years ago

otherwise Nextcloud is unbearably slow.

20.0.4 doesn't work at all with richdocuments enabled.

I mean Nextcloud itself gets slow here (everything, even stuff like notes or tasks). The loading time for any page goes up form 1s to over 40s which makes it barely usable. And it really depends on the CODE app - again at least for me; I have no other server to try against except for the built-in CODE. But you're rigt, I can't open any office documents with Collabora enabled. It's just that I didn't try before because the overall Nextcloud slowdown was too much to bear so I always went straight to disabling the app to keep the other stuff usable.

skydns commented 3 years ago

Still got the same issue. Ubuntu 20.04.1 Nextcloud 20.0.4 Collabora Online 3.7.11 richdocumentscode_arm64 6.4.203 Raspberry Pi4 4GB

performance problems are solved by stopping richdocumentscode_arm64 with nextcloud.occ app:disable richdocumentscode_arm64

i hope the next update will solve the problem

May be this helps:

found in log /var/snap/nextcloud/current/logs/php_errors.log

full of these lines, it seems the richdocumentscode isn't running file_get_contents(http://localhost:9982/hosting/capabilities): failed to open stream: HTTP request failed! in /var/snap/nextcloud/25284/nextcloud/extra-apps/richdocumentscode_arm64/proxy.php on line 208

pallebone commented 3 years ago

Same issue glad I found this post. Collabora breaks nextcloud :(

theoneandonly-vector commented 3 years ago

For me this doesn't happen on Rapberry Pi4 but on a x86vm only (same version).

moshemalawach commented 3 years ago

Happening on a latest version of docker images (x86_64), both apache and fpm... on a new install.

Was banging my head against walls for hours until I found that thread and disabled both apps (that are installed by default!) Now it works perfectly...

tyl0r commented 3 years ago

I just did a fresh NC install and noticed the same issue, which is what brought me here. I tried installing 20.0.1, 20.0.0, and 19.0.6 and they all had the same issue. I'm currently trying to get my company to use this software and really needed to evaluate NextCloud with Collabora. The issue appears to be isolated to the later versions of the apps (richdocuments and/or richdocumentscode). Here's what I did to get it working:

First, I installed the latest 20.0.4, again. I then downloaded older versions of richdocuments and richdocumentscode apps, extracted them to the nextcloud apps dir, and chown'd the dirs.

cd [your_nextcloud_dir]/apps
rm -r richdocuments/ richdocumentscode/
wget https://github.com/CollaboraOnline/richdocumentscode/releases/download/6.4.13/richdocumentscode.tar.gz
wget https://github.com/nextcloud/richdocuments/releases/download/v3.7.6/richdocuments.tar.gz
tar xzf richdocuments.tar.gz
tar xzf richdocumentscode.tar.gz 
chown -R www-data:www-data richdocuments*
rm -r *.tar.gz

Then I updated the database with the same versions of the apps:

update oc_appconfig set configvalue = '3.7.6' where appid = 'richdocuments' and configKey = 'installed_version';
update oc_appconfig set configvalue = '6.4.13' where appid = 'richdocumentscode' and configKey = 'installed_version';

Ultimately, I ended up disabling richdocumentscode and setup a standalone server for that.

update oc_appconfig set configvalue = 'no' where configkey = 'enabled' and appid = 'richdocumentscode';
select * from oc_appconfig where appid like 'richdocuments%';
+-------------------+----------------------------------+------------------------------------------+
| appid             | configkey                        | configvalue                              |
+-------------------+----------------------------------+------------------------------------------+
| richdocuments     | disable_certificate_verification |                                          |
| richdocuments     | enabled                          | yes                                      |
| richdocuments     | installed_version                | 3.7.6                                    |
| richdocuments     | public_wopi_url                  | https://collabora.clearhello.com         |
| richdocuments     | types                            | filesystem,dav,prevent_group_restriction |
| richdocuments     | wopi_url                         | https://collabora.clearhello.com         |
| richdocumentscode | enabled                          | no                                       |
| richdocumentscode | installed_version                | 6.4.13                                   |
| richdocumentscode | types                            |                                          |
+-------------------+----------------------------------+------------------------------------------+

Hope that helps someone!

juliusknorr commented 3 years ago

I can finally reproduce the issue on one of my tests systems. I'll look into that.

juliusknorr commented 3 years ago

There are basically two things we need to achieve here:

SimplyCorbett commented 3 years ago 
* however everyone should at least setup APCu caching for their Nextcloud instances as this helps a lot with the general performance.

I have no performance issues and no reason to setup APCu or redis.

If APCu will fix this issue I'll install it but I don't consider that a fix to the problem either.

skydns commented 3 years ago

I'm manually downloading richdocumentscode_arm64.tar.gz 6.4.301 released 5h ago, it's a big package an github is slow so the download will be ready in 8h :(

I installed php-apcu and changed in nextcloud config.php line 'memcache.local' => '\OC\Memcache\Redis', to 'memcache.local' => '\OC\Memcache\APCu', after snap restart nextcloud my changes rewrite to Redis

I'll try tomorrow, if it will solve the perfomance problems.

kyrofa commented 3 years ago

I installed php-apcu and changed in nextcloud config.php line 'memcache.local' => '\OC\Memcache\Redis', to 'memcache.local' => '\OC\Memcache\APCu',

I believe @juliushaertl was simply pointing out that there is an issue today if you don't have a memcache, not trying to say that you must use APCu. You have a memcache: it's redis.

skydns commented 3 years ago

Thanks kyrofa, you're right. Tried with richdocumentscode_arm64 6.4.203 same issue

pallebone commented 3 years ago

I did a "sudo apt-get instal php-redis" and then "sudo snap restart nextcloud" but it did not resolve the issue. However I believe differently to SimplyCorbett that if this worked it would not be a solution. If it improves performance and allows richdocumentscode to work as it expects, this solution would be quite elegant and make the entire process more quick and snappy. Im not sure if there are some extra steps I missed to make it work.

kyrofa commented 3 years ago

"sudo apt-get instal php-redis" and then "sudo snap restart nextcloud"

Snap users, please stop doing this, you already have a memcache and this will not help anything.

pallebone commented 3 years ago

Ok sorry Im still learning and doing my best :)

avimanyu786 commented 3 years ago

Gave up and disabled Collabora Online and Collabora Online - Built-in CODE Server. Installed OPENOFFICE and Community Document Server from the App store instead and then did the upgrade from 20.0.2 to 20.0.4. No idea why the community server for open office has a 1 star rating. Seems fine on my end.

gerd21 commented 3 years ago

Perhaps nor really necessary to state that I see this too. I use it in a VM with docker without root rights, I'm in the docker group. And I use 20.0.4, the latest docker image. The interesting thing and the reason for my post is, that everything works fine, if I connect to NC on the host machine with 'http://localhost:8080'. Everything runs fast and in a way I expect. BUT: this 'localhost' has has an IP as well. And if I try to enter that way 'http://192.168.178.34:8080' (for example) it is slow as described. Config.php is modified in trusted_hosts. And while the real IP is spinning, the parallel localhost tab works fine. The 'fix' mentioned above works for me.

Perhaps it helps.

vawaver commented 3 years ago

Gave up and disabled Collabora Online and Collabora Online - Built-in CODE Server. Installed OPENOFFICE and Community Document Server from the App store instead and then did the upgrade from 20.0.2 to 20.0.4. No idea why the community server for open office has a 1 star rating. Seems fine on my end.

I have Ubuntu 20.04 server + Nextcloud via SNAP. Is there any tutorial on how to install Openoffice and Community Document Server? I cannot wait until the Collabora issue will be fixed. Meantime I would like to use another service for online editing of files. Thank you.

juliusknorr commented 3 years ago

Please head over to the forum for discussing setup questions. I'll lock this conversation since we already spotted the actual issue and just need to dive into a solution as described in https://github.com/nextcloud/richdocuments/issues/1282#issuecomment-756816460