Open Pilzinsel64 opened 1 year ago
I don't know if this is related but I also have troubles with loading Collabora Online - Built-in CODE Server after upgrading to v23.5.5 . No connection to CODE server and blank page when trying to open document files. No related errors in nextcloud log (even on debug level).
Only error I get in apache log is: coolwsd: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by coolwsd)
same problem for me. nothing in nextcloud log.
Collabora 22.5.1301 works without issues
Does anyone even read this issues? Some acknowledgement would be great. More and more people are reporting similar problems... I am aware this is free and opensource project but I think they at least should not be forcing this update if it is breaking things.
coolwsd: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by coolwsd)
This looks like a serious issue, right? We build the packages for the AppImage on Ubuntu 18.04, it has glibc-2,27, while CentOS 7 has glibc-2.17. For the 22.5.x series we used Ubuntu 16.04 to build the packages. When I changed the baseline, I did not think of this use case, i.e. someone would run the deb packages on CentOS7 (via several abstraction layers, snap, Appimage, etc.). The solution could be to build deb packages on Ubuntu 16.04 again. Of course it won't solve the problems on RPi. I think those problems are unrelated to yours.
Hmm, even Ubuntu 16.04 has glibc-2.23. The issue needs further investigation.
@timar If you need any more information or want me to test something, just tell me. :)
@timar If you need any more information or want me to test something, just tell me. :)
coolwsd from AppImage logs at weird places, can you please try to find it? In /tmp for example (find /tmp -name coolwsd.log). Also webserver's error log can tell if there is an issue with proxy.php. I tried arm64 build on AWS, it worked for me, but I know that there are many options, many different setups.
@timar Sure! Here are the webserver log files:
webserver-logs.zip
The last line(s) of the apache_error.log
and php_errors.log
file may be interesting, but the time seems to not really match. 🤔
The coolwsd.log don't get generated at all when using the new version. When using v22.5.1301 there is a log file.
This is what I did from enabling the app to picking the log files:
I hope this helps! If you need anything else, just tell me.
@timar Maybe the new AppImage needs a new dependency that is not installed on my system? 🤔
EDIT: I'm sorry, but I will not be able anymore to provide more details sadly. Aber long time of waiting I moved my server to a new hardware with more perfoemave and different architecture. The Built-In CODE app work here now.
unable to get Collabora to run on Rock5 since NC26 update. please see my issue here: https://github.com/nextcloud-snap/nextcloud-snap/issues/2477
I don't know if this is related but I also have troubles with loading Collabora Online - Built-in CODE Server after upgrading to v23.5.5 . No connection to CODE server and blank page when trying to open document files. No related errors in nextcloud log (even on debug level).
Only error I get in apache log is: coolwsd: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by coolwsd)
- OS: Centos 7 ( 3.10.0-1160.90.1.el7.x86_64 )
- PHP: 8.1.20
- Nextcloud version: 26.0.2
- APP: Collabora Online - Built-in CODE Server: v23.5.5
- APP: Nextcloud Office: v8.0.2
After migration from Centos7 to Alma9 (which has support for newer versions of GLIBC) everything works ok.
I'm on a shared host and have all the latest versions of NC and collabora installed. I'm on PHP 8.2. Since 23.5.102 all versions of collabora server are not loading. When downgrading to 23.5.5 it loads. I don't know what the issue is. Maybe it has something to do with fontconfig. Anyway I will stay on 23..5.5 until a newer version is loading again.
I think this might affect also other users on shared hosts. If any developer is interested in researching this issue, I can provide access to my test server. Just give me an email.
Everyone who still read this:
Did you try to install all required libraries? On Ubuntu Server 22.04 it is common that fontconfig
is missing, as it looks like. At list this one library is required to get the app working. But to ensure also glibc
should be installed.
See: https://github.com/CollaboraOnline/richdocumentscode#system-requirements
Just found that out while searching for a similar issue. Sadly, I can't test it myself anymore, I'm on amd64 now. :/ So please give it a try and let me know if it solved your problem! ;)
I am happy to test this on my arm (rock5) setup. can you let me know how I could install these two on armbian?
Try this: sudo apt-get update && sudo apt-get install fontconfig glibc-source -y
Ich denke beide sind schon installiert.
sudo apt list fontconfig
Auflistung… Fertig
fontconfig/jammy,now 2.13.1-4.2ubuntu5 arm64 [Installiert,automatisch]
fontconfig/jammy 2.13.1-4.2ubuntu5 armhf
sudo apt list glibc-source -a
Auflistung… Fertig
glibc-source/jammy-security,jammy-security,jammy-updates,jammy-updates 2.35-0ubuntu3.4 all
glibc-source/jammy,jammy 2.35-0ubuntu3 all
habe glib installiert - hat leider nicht verbessert. akutelle version vom collabora started nicht
rock@MinkaRock5:~$ sudo apt list glibc-source -a Auflistung… Fertig glibc-source/jammy-security,jammy-security,jammy-updates,jammy-updates,now 2.35-0ubuntu3.4 all [installiert] glibc-source/jammy,jammy 2.35-0ubuntu3 all
This worked for me. Edit proxy.php on lines 152 an 153:
151 exec('( /sbin/ldconfig -p || scanelf -l ) | grep fontconfig > /dev/null 2>&1', $output, $return); 152 // if ($return) 153 // return 'no_fontconfig'; 154 return '';
Fontconfig is installed and running on my server. However Collabora still thinks it's not.
@Jolopu Interesting. But we're close.
What is the output of this command on your system? ( /sbin/ldconfig -p || scanelf -l ) | grep fontconfig
?
Mine is:
libfontconfig.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libfontconfig.so.1
... while libfontconfig.so.1
is the requirement.
Also, what is the output of this address? https://yourcloud.com/index.php/apps/richdocuments/settings/fonts.json
{"kind":"fontconfiguration","server":"Myserver (https:\/\/mycloud.tv)","fonts":[]}
@Jolopu Interesting. But we're close.
What is the output of this command on your system?
( /sbin/ldconfig -p || scanelf -l ) | grep fontconfig
?Mine is:
libfontconfig.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libfontconfig.so.1
... while
libfontconfig.so.1
is the requirement.
I'm on a shared host. Sorry.
Ok, thank you anyway for tryit it out. Then, maybe another library is still missing. 🤔
@timar We wrote a guide how to install Collabora Online & Built-In CODE on a fresh installed Ubuntu Server 22.04 LTS on x86_64 devices. This should be the same for ARM64.
Please, follow my steps here and you will be able to reproduce and debug the issue, I'm 100% sure!
sudo snap install nextcloud
I hope you can take you the time, there are many users affected. Please try my guide on a ARM64 VM or device.
Thanks in advance! :)
Exact same issue here too - let me know if I can help test anything!
Some extra notes...
Arm64 Ubuntu 22.04.3 LTS VM (Parallels on Mac M1 host) - latest Nextcloud Snap.
$ ( /sbin/ldconfig -p || scanelf -l ) | grep fontconfig
returns
libfontconfig.so.1 (libc6,AArch64) => /lib/aarch64-linux-gnu/libfontconfig.so.1
I tried editing Proxy.php as mentioned in https://github.com/CollaboraOnline/richdocumentscode/issues/226#issuecomment-1750423699 but this did not work - although, I question my changes were loaded. After editing the file, how can I ask NextCloud to ensure it is using my modifications? @Jolopu
I also ensured all packages were installed... sudo apt-get update && sudo apt-get install fontconfig glibc-source -y
I see green ticks in admin confirming server is reachable, but creating and opening a new ODT file shows an infinite spinning wheel.
I am here for assistance if needed and would love to help. Thank you for the great project, but it is completely broken at the moment for every Arm64 user and has been for multiple months!... please let us help you fix it.
I don't think its fair users are wasting time assuming they or their devices are at fault when in reality, arm64 simply isn't supported for last few months. Even if issue is fixed someday, I believe updating the readme until the issue is fixed is the minimum fair thing to do, to avoid users wasting hours of time today.
https://github.com/CollaboraOnline/richdocumentscode/pull/243
@Jolopu
This worked for me. Edit proxy.php on lines 152 an 153:
151 exec('( /sbin/ldconfig -p || scanelf -l ) | grep fontconfig > /dev/null 2>&1', $output, $return); 152 // if ($return) 153 // return 'no_fontconfig'; 154 return '';
Fontconfig is installed and running on my server. However Collabora still thinks it's not.
Can you tell me more about how you tested this please? Maybe we can make a PR to fix the issue. I'd like to test your theory more please
@LMRW I understand your frustration, and I'm sorry for the problems. I tried the app with nexctcloud docker on arm64 host and it worked. So it's not entirely broken.
docker run -d -p 80:80 --name nextcloud nextcloud
Then I tried snap as suggested by @Pilzinsel64
sudo snap install nextcloud
root@cp-test-arm64:/var/snap/nextcloud/38464/nextcloud/extra-apps/richdocumentscode/collabora# file Collabora_Online.AppImage
Collabora_Online.AppImage: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=30e06184968532b6a9aa36f44ada39e4af0bda56, for GNU/Linux 2.6.32, stripped
WTF?! This is unexpected. I installed the arm64 version....
Hmm, that is also there, in richdocumentscode_arm64 folder. Manually starting the AppImage lead to success...
root@cp-test-arm64:/var/snap/nextcloud/38464/nextcloud/extra-apps/richdocumentscode_arm64/collabora# ./Collabora_Online.AppImage
Logging at warning level to file: /tmp/coolwsd.1p9A6Fu6f4/coolwsd.log
Security: coolforkit incorrect user-name, other than 'cool'
coolforkit version details: 23.05.5.3 - 5093121
Init vcl
preload: xsec_xmlsec merged ucpchelp1 wpftwriter wpftcalc wpftimpress wpftdraw writerfilter msforms ucppkg1 ucpcmis1 cached1 vbaswobj swd sw ucpdav1 smd sm pdfimport PresentationMinimizer sd scriptframe protocolhandler dlgprov date analysis vbaobj scfilt scd xmlsecurity sc expwrap oox LanguageTool ldapbe2 pcr storagefd log chartcore pdffilter migrationoo3 deploymentgui scn cui sdbt mozbootstrap bootstrap flat io animcore svgfilter embobj t602filter dbaxml stocservices chartcontroller namingservice cairocanvas invocadapt introspection migrationoo2 dbpool2 binaryurp mysql_jdbc pricing proxyfac calc invocation dba uuresolver solver reflection writer textconversiondlgs hwp msword graphicfilter emboleobj sdd slideshow dbase bib
Disabled: ucpftp1 rptxml rptui rpt dbp abp sdbc2 cmdmail PresenterScreen dbu odbc
Allowlisted languages: de_DE el en_GB en_US es_ES fr_FR hu it nl pt_BR pt_PT ru
Preloading dictionaries: de-DE en-US fr-FR it-IT nl-NL pt-BR ru-RU en-GB nl-BE pt-PT es-ES
Preloading thesauri: de-DE en-US fr-FR it-IT nl-NL pt-BR ru-RU en-GB nl-BE pt-PT es-ES
Preload icons
Preload short cut accelerators
Preload languages
Preload fonts
Preload config
office version details: { "ProductName": "Collabora Office", "ProductVersion": "23.05", "ProductExtension": ".5.3", "BuildId": "e7921fedb8b5615b0dc7cd3ec8b91c9b4e06d002", "BuildConfig": "'--enable-mpl-subset' '--with-vendor=Collabora' '--disable-community-flavor' '--with-branding=icon-themes/galaxy/brand_cp' '--with-system-dicts' '--with-myspell-dicts' '--with-system-zlib' '--disable-poppler' '--enable-cairo-rgba' '--without-system-cairo' '--without-system-fontconfig' '--without-system-freetype' '--without-system-graphite' '--without-system-harfbuzz' '--without-system-openssl' '--without-system-libpng' '--without-system-libxml' '--without-system-jpeg' '--without-system-expat' '--without-system-curl' '--without-system-icu' '--without-system-nss' '--without-system-jars' '--without-system-postgresql' '--without-java' '--without-junit' '--without-help' '--with-linker-hash-style=both' '--with-fonts' '--enable-noto-font' '--with-galleries=no' '--with-theme=colibre colibre_svg' '--with-external-thes-dir=/usr/share/mythes' '--with-external-hyph-dir=/usr/share/hyphen' '--with-external-dict-dir=/usr/share/hunspell' '--disable-dbus' '--enable-extension-integration' '--disable-odk' '--disable-kf5' '--disable-gtk3' '--disable-qt5' '--disable-gstreamer-1-0' '--disable-evolution2' '--disable-gio' '--disable-gui' '--disable-scripting-beanshell' '--disable-scripting-javascript' '--disable-ext-wiki-publisher' '--disable-report-builder' '--disable-ext-nlpsolver' '--disable-sdremote' '--disable-sdremote-bluetooth' '--disable-postgresql-sdbc' '--disable-firebird-sdbc' '--disable-randr' '--disable-ext-numbertext' '--enable-epm' '--enable-python=internal' '--disable-online-update' '--disable-dconf' '--enable-mergelibs' '--with-package-format=deb rpm' '--enable-release-build' '--with-lang=ar bg ca cs da de el en-US en-GB eo es eu fi fr gl he hr hu id is it ja ko lo nb nl oc pl pt pt-BR sq ru sk sl sv tr uk vi zh-CN zh-TW' '--disable-lotuswordpro' '--disable-lpsolve' '--enable-symbols' '--enable-sal-log' '--without-templates' '--with-external-tar=/home/collabora/jenkins/external' '--disable-symbols' '--with-package-format=deb' '--srcdir=/home/collabora/jenkins/workspace/core-co-22.05-for-arm64-PI' '--enable-option-checking=fatal'" }
Ready to accept connections on port 9983.
Something may be wrong with the nextcloud snap.
@timar Thank you very much for trying thisa out!
Well, the result is unexpected for me. 🤔 The only idea I have could be a missing or wrong dependency. What exactly has changed in-between this both version (23.5.5 and 22.05.1301) regarding dependencies?
Edit: I just see that it tries to start the x86_64 version and not the arm64 version.Do you have installed both? (Maybe "recommended apps" installed the non-arm64 version)
As a point of interest this appears to be an issue with NextCloud not running the code server rather than with the code server itself. Running the following locally on the server will start up the code server and allow a NextCloud snap to connect to it with no issues:
root@home-pi-base-server:/var/snap/nextcloud/38464/nextcloud/extra-apps/richdocumentscode_arm64/collabora# ./Collabora_Online.AppImage
I am able to create, edit, save, and reopen files without any issues. To note, I have run the following in an attempt to make things work but it does not seem to have made a difference:
sudo apt-get update && sudo apt-get install fontconfig glibc-source -y
Digging a bit deeper this appears to be where it is crashing on startup in the proxy.php file. It would appear that something is pssing 'ui_theme=light' when it should be passing a 'status' or 'req=...' parameter. Will update if I find when this issue was introduced:
[11-Dec-2023 08:59:33 UTC] richdocumentscode (proxy.php) error exit, PID: 23106, Message: The param should be 'status' or 'req=...', but is: 'ui_theme=light'
[11-Dec-2023 09:07:35 UTC] PHP Warning: file_get_contents(http://localhost:9983/hosting/capabilities): Failed to open stream: HTTP request failed! in /var/snap/nextcloud/38464/nextcloud/extra-apps/richdocumentscode_arm64/proxy.php on line 250
[11-Dec-2023 09:09:37 UTC] richdocumentscode (proxy.php) error exit, PID: 24110, Message: No content in reply from coolwsd. Is SSL enabled in error ?
Ok, I think I have the when and why it started breaking now. This https://github.com/nextcloud/richdocuments/pull/2964 pull request was just before everything started breaking. It included this commit https://github.com/nextcloud/richdocuments/pull/2964/commits/e192ccfc205c67995e6fe21582b92eb1a3e9e909 . This introduced a new uiTheme that they appear to render and extract ui_theme from which is why it doesn't show up in searches. They seem to be using some new system to pass this information to the code server.
I am not well versed enough in NextCloud development to attempt a fix currently but hopefully this at least helps get things started.
A few further updates, as a temporary measure commenting out this else clause will allow the program to continue at least. It then fails silently on the launch command:
// Extract the AppImage if FUSE is not available
$launchCmd = "bash -c \"( $appImage $remoteFontConfig --pidfile=$pidfile || $appImage --appimage-extract-and-run $remoteFontConfig --pidfile=$pidfile) >/dev/null & disown\"";
With some trial and error it turns out that both the FUSE and non-FUSE commands are failing in the snap environment. The FUSE version works when run as a normal linux user but fails in the snap environment. It functions even if running the exact command as a normal use. The non-FUSE version fails with a glibc error. My first thought is trying to fix the FUSE error first since it is probably just a configuration issue for the arm64 build. The non-FUSE issue is similar to what has been seen above and I am not sure how to go about fixing it.
Here is the FUSE only code error:
dlopen(): error loading libfuse.so.2
AppImages require FUSE to run.
You might still be able to extract the contents of this AppImage
if you run it with the --appimage-extract option.
See https://github.com/AppImage/AppImageKit/wiki/FUSE
for more information
Here is the non-Fuse information:
fontconfig is already the newest version (2.13.1-4.2).
glibc-source is already the newest version (2.31-13+rpt2+rpi1+deb11u7).
0 upgraded, 0 newly installed, 0 to remove and 40 not upgraded.
root@home-pi-base-server:/home/admin# tail /var/snap/nextcloud/common/nextcloud/tmp/testy_test.err
coolwsd: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by coolwsd)
coolwsd: /usr/lib/aarch64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by coolwsd)
coolwsd: /lib/aarch64-linux-gnu/libc.so.6: version
GLIBC_2.28' not found (required by coolwsd)`
Hm, this should be expected. Libraries outside of the snap environment (the snap must explicitely define them) can not be used normally. @kyrofa please correct me, if I'm false.
In Nextcloud snap libfuse.so.2
and glibc-source
are not installed within the snap itself and never was. So, this "can not" be the reason why it worked some versions before, on arm64 & amd64.
However, for me on my amd64 devices it worked to install that libraries manually via apt-get
as they weren't even installed on the system itself. Sadly, I can't longer test this on an arm64 device . :/
In Nextcloud snap
libfuse.so.2
andglibc-source
are not installed within the snap itself and never was.
Theoretically this is true, but snap does some weird stuff. It is very poorly documented but unless the no-system-libraries build attribute is used snapcraft will pull any libraries from the system that it can find that it has marked as needing. Here is the documentation as such and here is an example of someone having issues on different project due to it.
From my understanding the first issue that caused problems came from the change with ui_theme being passed, and my far less certain but not entirely unfounded guess is that in the meantime the build environment changed. Without knowing what the environment looked like for builds that did work I can't confidently say that snapcraft wasn't pulling in either the fuse or glibc libraries from the build environment.
One way or another one of the two libraries are likely required to get it working again along with handling the ui_information better than just discarding all errors.
P.S. no-system-libraries may have been removed at some point but the snapcraft documentation search is down for me at the moment so I can't verify it unfortunately. If for some reason it was removed that could possibly explain that set of issues.
I think I found where the other issue began, I haven't been able to confirm it or look into a fix yet but this commit was made just before v23.5.5 and moved the docker build system from Ubuntu 18.04 to DebianSlim. Given that the nextcloud snap runs on core18 it should have had full compatibility with the old docker builder. Why it only broke on arm64 and not x86_64 is yet to be determined but it seems pretty likely that it has something to do with this. They have a couple libraries that are prebuilt in their docker container if the documentation is up to date. I am not sure why snap seems to act different in this case on different architectures but it appears at the point poco and libc6 were built DebianSlim had slightly newer versions than core18.
Unfortunately this is still all kinda in the "why is this happening" rather than a fix, hopefully I can find something about path usage on different architectures but I don't know if that documentation will exist.
Edit: I did have one thought, it is probable that they built the amd64 libraries first. If they built the amd version before moving away from ubuntu but built the arm version afterwards it would explain why one works while the other doesn't. Obviously this is completely speculation but might explain the inconsistency.
Ok, think I have enough evidence now. There is very little doubt that the zlib and poco libraries were built in differently configured systems,. The arm64 version requires glibc to be at 2.28 as is reported by my own install. On the other hand the x86_64 build only requires 2.25. I checked by downloading the current appimage build and examining the coolwsd requirements after manually extracting it. Here is the report for the x86_64 version:
collabora_test/usr/bin$ objdump -T coolwsd | grep GLIBC |sed 's/.*GLIBC_\([.0-9]*\).*/\1/g' | sort -Vu
2.2.5
2.3
2.3.2
2.3.3
2.3.4
2.4
2.6
2.7
2.9
2.10
2.11
2.12
2.14
2.17
2.25
And here is the report for the arm64 version:
/var/snap/nextcloud/current/nextcloud/extra-apps/richdocumentscode_arm64/collabora/squashfs-root/usr/bin# objdump -T coolwsd | grep GLIBC |sed 's/.*GLIBC_\([.0-9]*\).*/\1/g' | sort -Vu
2.17
2.25
2.28
I used this guide to find the command to list glibc dependencies.
As you can see x86_64 only requires up to 2.25 which is met by the 2.27 library provided by core18. On the other hand the arm64 requires 2.28 which can not bet met. As stated above the FUSE command can't work without the snap interface for it but there is no plan to support that as per this.
This leaves two main options, either moving the nextcloud-snap to core20 or above or shipping a prebuilt glibc with this plugin for arm64 users. I understand that the bump to core20 is not likely in the near future and shipping glibc is not anywhere near optimal. Mostly leaving this here for documentation, I don't really have much expectation that this will be fixed in the near future.
This does bring up an interesting other point though, which is why I went ahead and confirmed my suspicions. If collabora ever has an issue with the zlib and poco libraries there is a strong chance that they will be rebuilt and it will stop working on all builds based around ubuntu 18.04. Depending on when this occurs it could even break on much newer bases since they are using Debian stable as the basis of their docker build environment. Given that it hasn't been updated in over half a year this probably isn't going to be a common issue but is probably worth considering if someone does decide to create a fix for this.
I am probably going to move to using containers for my nextcloud install since I really would like collabora support so I may or may not be able to test easily if a fix is introduced.
P.S. it is quite likely that https://github.com/CollaboraOnline/richdocumentscode/issues/246 is based on this issue as well. I don't have a CentOS 7 machine to test it on at the moment but it seems to be a very similar issue.
This has been broken for over one year for arm snap users, maybe consider updating the readme until it is resolved? I made a PR
Many users are wasting time assuming this should work when a simple readme note might save them hours
https://github.com/CollaboraOnline/richdocumentscode/pull/243
Issue
I'm using the app Collabora Online Built-in CODE (arm64). Since the last update (v23.5.5) the CODE server seems to have problems to start. When downgrading to the previous version (v22.5.1301) and trigger everything works fine again.
But I need to do the downgrade each time the server restarts (basically every night) now since there is no possibility to disbale auto-updates in the snap packages.
Log
There isn't any new entry in the Nextcloud log. If you want the log anyway, please tell me.
System report
Link: https://dragoncloud.ddnss.de/index.php/s/XFe5CdWLnyBC5nf Passwort:
cBKcRkWcNs
Environment infos