Nextcloud Office disabled after 4.9.0 update

[MariusQuabeck]

I've finally updated AIO to 4.9.0 and can now no longer access Nextcloud Office

the AIO page shows all containers running OK

but still...

nextcloud-aio-collabora log (this part keeps repeating itself):

wsd-00001-00001 2023-04-27 16:06:04.388887 +0000 [ coolwsd ] INF  Waiting for a new child for a max of 20000ms| wsd/COOLWSD.cpp:5594
wsd-00001-00033 2023-04-27 16:06:04.597754 +0000 [ prisoner_poll ] TRC  Poll completed with 0 live polls max (5000000us)(timedout)| net/Socket.cpp:358
wsd-00001-00033 2023-04-27 16:06:04.597843 +0000 [ prisoner_poll ] TRC  #20: Starting handling poll events of prisoner_poll at index 0 (of 1): 0x0| net/Socket.cpp:437
wsd-00001-00033 2023-04-27 16:06:04.597851 +0000 [ prisoner_poll ] TRC  #20: setupPollFds getPollEvents: 0x1| net/Socket.hpp:877
wsd-00001-00033 2023-04-27 16:06:04.597855 +0000 [ prisoner_poll ] TRC  ppoll start, timeoutMicroS: 5000000 size 1| net/Socket.cpp:339
wsd-00001-00033 2023-04-27 16:06:09.602333 +0000 [ prisoner_poll ] TRC  Poll completed with 0 live polls max (5000000us)(timedout)| net/Socket.cpp:358
wsd-00001-00033 2023-04-27 16:06:09.602428 +0000 [ prisoner_poll ] TRC  #20: Starting handling poll events of prisoner_poll at index 0 (of 1): 0x0| net/Socket.cpp:437
wsd-00001-00033 2023-04-27 16:06:09.602440 +0000 [ prisoner_poll ] TRC  #20: setupPollFds getPollEvents: 0x1| net/Socket.hpp:877
wsd-00001-00033 2023-04-27 16:06:09.602445 +0000 [ prisoner_poll ] TRC  ppoll start, timeoutMicroS: 5000000 size 1| net/Socket.cpp:339
wsd-00001-00033 2023-04-27 16:06:14.607603 +0000 [ prisoner_poll ] TRC  Poll completed with 0 live polls max (5000000us)(timedout)| net/Socket.cpp:358
wsd-00001-00033 2023-04-27 16:06:14.607723 +0000 [ prisoner_poll ] TRC  #20: Starting handling poll events of prisoner_poll at index 0 (of 1): 0x0| net/Socket.cpp:437
wsd-00001-00033 2023-04-27 16:06:14.607733 +0000 [ prisoner_poll ] TRC  #20: setupPollFds getPollEvents: 0x1| net/Socket.hpp:877
wsd-00001-00033 2023-04-27 16:06:14.607737 +0000 [ prisoner_poll ] TRC  ppoll start, timeoutMicroS: 5000000 size 1| net/Socket.cpp:339
wsd-00001-00033 2023-04-27 16:06:19.611646 +0000 [ prisoner_poll ] TRC  Poll completed with 0 live polls max (5000000us)(timedout)| net/Socket.cpp:358
wsd-00001-00033 2023-04-27 16:06:19.611753 +0000 [ prisoner_poll ] TRC  #20: Starting handling poll events of prisoner_poll at index 0 (of 1): 0x0| net/Socket.cpp:437
wsd-00001-00033 2023-04-27 16:06:19.611763 +0000 [ prisoner_poll ] TRC  #20: setupPollFds getPollEvents: 0x1| net/Socket.hpp:877
wsd-00001-00033 2023-04-27 16:06:19.611767 +0000 [ prisoner_poll ] TRC  ppoll start, timeoutMicroS: 5000000 size 1| net/Socket.cpp:339

nextcloud-aio-nextcloud log:

Config value base_endpoint for app notify_push set to https://[URLredacted]/push
Config value wopi_url for app richdocuments set to https://[URLredacted]/
System config value allow_local_remote_servers set to boolean true
Config value wopi_allowlist for app richdocuments set to [IPredacted],127.0.0.1/8,192.168.0.0/16,172.16.0.0/12,10.0.0.0/8,fd00::/8,::1,127.0.1.1,2a01:7e01::f03c:93ff:fe35:9ef2
[2023-04-27 16:02:46.653463 +00:00] ERROR [notify_push] src/main.rs:77: Self test failed: Error while communicating with nextcloud instance
[27-Apr-2023 16:02:46] NOTICE: fpm is running, pid 359
[27-Apr-2023 16:02:46] NOTICE: ready to handle connections
Activating collabora config...
Failed to activate any config changes
cURL error 28: Operation timed out after 45000 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://[URLredacted]/hosting/discovery
#0 /var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php(158): GuzzleHttp\Handler\CurlFactory::createRejection(Object(GuzzleHttp\Handler\EasyHandle), Array)
#1 /var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php(110): GuzzleHttp\Handler\CurlFactory::finishError(Object(GuzzleHttp\Handler\CurlHandler), Object(GuzzleHttp\Handler\EasyHandle), Object(GuzzleHttp\Handler\CurlFactory))
#2 /var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlHandler.php(47): GuzzleHttp\Handler\CurlFactory::finish(Object(GuzzleHttp\Handler\CurlHandler), Object(GuzzleHttp\Handler\EasyHandle), Object(GuzzleHttp\Handler\CurlFactory))
#3 /var/www/html/lib/private/Http/Client/DnsPinMiddleware.php(113): GuzzleHttp\Handler\CurlHandler->__invoke(Object(GuzzleHttp\Psr7\Request), Array)
#4 /var/www/html/3rdparty/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php(35): OC\Http\Client\DnsPinMiddleware->OC\Http\Client\{closure}(Object(GuzzleHttp\Psr7\Request), Array)
#5 /var/www/html/3rdparty/guzzlehttp/guzzle/src/Middleware.php(31): GuzzleHttp\PrepareBodyMiddleware->__invoke(Object(GuzzleHttp\Psr7\Request), Array)
#6 /var/www/html/3rdparty/guzzlehttp/guzzle/src/RedirectMiddleware.php(71): GuzzleHttp\Middleware::GuzzleHttp\{closure}(Object(GuzzleHttp\Psr7\Request), Array)
#7 /var/www/html/3rdparty/guzzlehttp/guzzle/src/Middleware.php(63): GuzzleHttp\RedirectMiddleware->__invoke(Object(GuzzleHttp\Psr7\Request), Array)
#8 /var/www/html/3rdparty/guzzlehttp/guzzle/src/HandlerStack.php(75): GuzzleHttp\Middleware::GuzzleHttp\{closure}(Object(GuzzleHttp\Psr7\Request), Array)
#9 /var/www/html/3rdparty/guzzlehttp/guzzle/src/Client.php(331): GuzzleHttp\HandlerStack->__invoke(Object(GuzzleHttp\Psr7\Request), Array)
#10 /var/www/html/3rdparty/guzzlehttp/guzzle/src/Client.php(168): GuzzleHttp\Client->transfer(Object(GuzzleHttp\Psr7\Request), Array)
#11 /var/www/html/3rdparty/guzzlehttp/guzzle/src/Client.php(187): GuzzleHttp\Client->requestAsync('get', Object(GuzzleHttp\Psr7\Uri), Array)
#12 /var/www/html/lib/private/Http/Client/Client.php(218): GuzzleHttp\Client->request('get', 'https://cloud.n...', Array)
#13 /var/www/html/custom_apps/richdocuments/lib/WOPI/DiscoveryManager.php(94): OC\Http\Client\Client->get('https://cloud.n...', Array)
#14 /var/www/html/custom_apps/richdocuments/lib/WOPI/DiscoveryManager.php(66): OCA\Richdocuments\WOPI\DiscoveryManager->fetchFromRemote()
#15 /var/www/html/custom_apps/richdocuments/lib/WOPI/Parser.php(41): OCA\Richdocuments\WOPI\DiscoveryManager->get()
#16 /var/www/html/custom_apps/richdocuments/lib/Command/ActivateConfig.php(68): OCA\Richdocuments\WOPI\Parser->getUrlSrc('Capabilities')
#17 /var/www/html/custom_apps/recognize/vendor/symfony/console/Command/Command.php(298): OCA\RichDocuments\Command\ActivateConfig->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 /var/www/html/custom_apps/recognize/vendor/symfony/console/Application.php(1040): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#19 /var/www/html/custom_apps/recognize/vendor/symfony/console/Application.php(301): Symfony\Component\Console\Application->doRunCommand(Object(OCA\RichDocuments\Command\ActivateConfig), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#20 /var/www/html/custom_apps/recognize/vendor/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#21 /var/www/html/lib/private/Console/Application.php(214): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#22 /var/www/html/console.php(100): OC\Console\Application->run()
#23 /var/www/html/occ(11): require_once('/var/www/html/c...')
#24 {main}

[szaimen]

Hm... can you follow https://github.com/nextcloud/all-in-one/discussions/1358

[MariusQuabeck]

will do and report back

[MariusQuabeck]

I did check the hosts file, it looks ok

# /etc/hosts
127.0.0.1       localhost

# The following lines are desirable for IPv6 capable hosts
::1             localhost ip6-localhost ip6-loopback
ff02::1         ip6-allnodes
ff02::2         ip6-allrouters

the allow list for WOPI requests input field does not show up on the page

output from curl -vvv https://$NC_DOMAIN:443/hosting/discovery

 TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=redactedURL
*  start date: Apr  2 04:16:39 2023 GMT
*  expire date: Jul  1 04:16:38 2023 GMT
*  subjectAltName: host "redactedURL" matched cert's "redactedURL"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: GET]
* h2h3 [:path: /hosting/discovery]
* h2h3 [:scheme: https]
* h2h3 [:authority: redactedURL]
* h2h3 [user-agent: curl/7.83.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x7fd0dc7b71b0)
> GET /hosting/discovery HTTP/2
> Host: redactedURL
> user-agent: curl/7.83.1
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!

[szaimen]

Nothing further gets returned by the command above?

If no, can you please post the full collabora logs here?

[MariusQuabeck]

that was all I could see from the linode console, there might be more but it won't let me scroll up nor does it correctly show anything if I pipe the output to a file, same via ssh

here is the full collabora log https://cloud.nerdzoom.media/s/7TRwcwxejn2H7ba

[szaimen]

Found your problem:

frk-00031-00031 2023-04-27 17:58:20.263938 ERR  Capability cap_sys_chroot is not set for the coolforkit program.| kit/ForKit.cpp:237
frk-00031-00031 2023-04-27 17:58:20.263938 +0000 [ coolforkit ] ERR  Capability cap_mknod is not set for the coolforkit program.| kit/ForKit.cpp:237
frk-00031-00031 2023-04-27 17:58:20.263943 +0000 [ coolforkit ] ERR  Capability cap_fowner is not set for the coolforkit program.| kit/ForKit.cpp:237
frk-00031-00031 2023-04-27 17:58:20.263948 +0000 [ coolforkit ] ERR  Capability cap_chown is not set for the coolforkit program.| kit/ForKit.cpp:237
Capabilities are not set for the coolforkit program.

[szaimen]

The question is now why this is happening for you...

[szaimen]

I found a really nice article which helps debugging this. https://earthly.dev/blog/intro-to-linux-capabilities/

Can you run sudo docker inspect nextcloud-aio-collabora | grep Pid

And use the Pid in the following command: sudo getpcaps <container-pid>

I need the output of the second command.

Please additionally post the output of sudo capsh --print

[MariusQuabeck]

root@cloud:~# sudo docker nextcloud-aio-collabora | grep Pid
docker: 'nextcloud-aio-collabora' is not a docker command.
See 'docker --help'

[szaimen]

Sorry, I updated the instrucrions. Please try again

[MariusQuabeck]

ah thank you, I was just trying to understand this blog post

output:

root@cloud:~# sudo docker inspect nextcloud-aio-collabora | grep Pid
            "Pid": 24090,
            "PidMode": "",
            "PidsLimit": null,
root@cloud:~# sudo getpcaps 24090
24090: =
root@cloud:~# sudo capsh --print
Current: =ep
Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read,cap_perfmon,cap_bpf,cap_checkpoint_restore
Ambient set =
Current IAB: 
Securebits: 00/0x0/1'b0
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
 secure-no-ambient-raise: no (unlocked)
uid=0(root) euid=0(root)
gid=0(root)
groups=0(root)
Guessed mode: UNCERTAIN (0)

I'm afraid this doesn't contain what you were hoping for :/

[szaimen]

It shows that indeed no capabilities are granted to the container so it confirms my assumption. Still the questions is why.

[szaimen]

I just upgraded docker on my test instance to 23.0.5 and collabora still works. So that also doesnt seem to be it...

Can you post the output of sudo docker info?

And on which hoster are you currently?

[szaimen]

For me docker info returns this:

Server:
 Containers: 11
  Running: 9
  Paused: 0
  Stopped: 2
 Images: 13
 Server Version: 23.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc version: v1.1.5-0-gf19387a
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-71-generic
 Operating System: Ubuntu 22.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.738GiB
 Name: aio-testing1
 ID: 5T4Z:Y64Q:ET2R:72X2:25DQ:4BU4:6NZ3:WGHF:6WTS:ON3V:YQPS:3UJZ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[MariusQuabeck]

sudo docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.4
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.17.3
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 9
  Running: 7
  Paused: 0
  Stopped: 2
 Images: 10
 Server Version: 23.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
Cgroup Version: 2
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
yslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc version: v1.1.5-0-gf19387a
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-71-generic
 Operating System: Ubuntu 22.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.832GiB
 Name: cloud.nerdzoom.media
 ID: WQZ5:U46M:FC5U:QGNY:5W3A:AES5:7DMR:QFHM:IIVT:ZZEC:LLYJ:GCN6
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[szaimen]

And it gets even more interesting: getpcaps also returns zero for the resulting collabora container on my end but the container still works on my end without throwing these errors...

[szaimen]

Apparently one of the children processes has the caps applied: Can you also try this? (get the children processes with pgrep -P and then run getpcaps for each?