Edits and changes made in Onlyoffice are not synced into Nextcloud

[timo12357]

Issue

Edits and changes made in Onlyoffice are not saved into Nextcloud and the next sync from Nextcloud to Onlyoffice destroys all changes made in Onlyoffice. This results in loss of work.

Expected Behavior

Edits and changes written in Onlyoffice sync to the local desktop environment and the changes done in the desktop environment sync to OnlyOffice

Actual Behavior

Files stored in the local desktop environment are synced flawlessly to Nextcloud. Onlyoffice opens them flawlessly and lets the user edit the document. After saving the document and or logging out from nextcloud changes are kept in OnlyOffice cache and never written to the actual document. When user opens same document from the desktop the changes have disappeared. If user saves the document with a small change, this change is synced to OnlyOffice overwriting the changes in the cache.

Possible Fix

Write cache data to file regularly and frequently. Recognize when user leaves Onlyoffice or closes file and write cache to file. In short: Copy the functionality of Google Docs.

Steps to Reproduce

1. Use a desktop wordprocessor and create a document in a NextCloud synced folder in your desktop environment
2. Log onto the Nextcloud web UI and open the file with OnlyOffice.
3. Ask colleague to do the same
4. Make changes in document
5. Exit document
6. Open document with desktop wordprocessor from local NextCloud synced folder.
7. Notice that the changes never were saved to the file in NextCloud,.
8. Open file in OnlyOffice from the Nextcloud web UI to notice changes are still there
9. Close OnlyOffice
10. Make minimal change in the desktop wordprocessor and save file locally.
11. Let NextCloud sync
12. Open file in OnlyOffice from the Nextcloud web UI to notice all changes made in OnlyOffice have been lost and overwritten with those made in the desktop environment
13. Feel despair.

Context

We just lost 2 days worth of expensive work made by our consultant. This is becoming very expensive to us!

Your Environment

• Nextcloud 18.0.3
• OnlyOffice Community Server bundled with NextCloud 18
• Nextcloud desktop client 2.6.4 Linux
• Nextcloud desktop client 2.6.4 Mac

Edit: Tried also this cronjob proposed by @googol42 but it does NOT solve the issue

• • • • • php -f /var/www/yourdomain/occ documentserver:flush

[Ulch]

I can confirm this behavior in an environment with Nextcloud 18.0.3, OnlyOffice Community Server bundled with NextCloud 18 and Nextcloud desktop client 2.5.1git on Debian 10.

A fix would be highly desirable to allow the deployment of Nextcloud Hub in productive environments.

[timo12357]

The onlyoffice integration as it is now is not suitable for production. DO NOT USE IT, YOU WILL LOSE DATA. I do not know who to tag to get attention to this? This bug is causing economic losses at the monent.

[cju77]

Same here with v18.0.3 ...

[timo12357]

Just got a mail from our consultant. She never opened the file from the desktop, only using Onlyoffice from our server. Seems like the data in the onlyoffice cache is never written to Nextcloud. The "Save as" function saves to cache, but the cache never gets written to Nextcloud. User gets false secure feeling about having backed up data while in real life it is stored in cache and lost.

[fabiangebert]

See #78, #77, #12, #87 all reporting the same issue. I think we should try to get in touch with Nextcloud team directly. Any clue how to reach them?

[timo12357]

@skjnldsv could you lead us to the right contact to raise awareness?

[skjnldsv]

Of course! @icewind1991 @rullzer @nickvergessen @blizzz :)

[blizzz]

IIRC it requires the tab to be closed so OO writes back the changes. It is by design of the OO backend however. I am not really involved there though.

[fabiangebert]

I tried closing in various ways, could it be related to autosaving or fast collaboration? (See "Advanced settings..." in File tab.

[timo12357]

In case of multiple users, does the writeout happen after any user closing the tab or the last user closing the tab?

[p-jo]

My documents are saving on a normal cron-run. Not sure how often I can run cron though, every minute would be good for saving the documents (my current settings are for every 5 minutes):

*/5 * * * * php -f /var/www/nextcloud/cron.php

[timo12357]

Mine are NOT saving on normal cron run.

[timo12357]

IIRC it requires the tab to be closed so OO writes back the changes. It is by design of the OO backend however. I am not really involved there though.

Who would be involved? It would be good to get the attention of a person who can fix this before more damage is caused.

[fabiangebert]

None of the cron settings leads to a working setup. However, I also cannot view the logs. I'm on a hosted environment (Hetzner, Germany), not sure if that makes a difference?

[timo12357]

@icewind1991 @rullzer @nickvergessen @blizzz What should we do to get this tagged as a bug? Us discussing here does not contribute to much.

[fabiangebert]

When setting Cron to AJAX it works. So it seems, either my hoster has messed up the cron Configuration or cron doesn't use the correct user name? There should be an indication in the OnlyOffice community server settings about the last time the cron ran successfully.

[skjnldsv]

There should be an indication in the OnlyOffice community server settings about the last time the cron ran successfully.

There is in your cron setup page

[fabiangebert]

There should be an indication in the OnlyOffice community server settings about the last time the cron ran successfully.

There is in your cron setup page

I know, but it will show up regardless if the Community Server was able to write the files. But it could be that I am missing the logs. I'll reach out to my hoster now.

[flow10]

Can confirm this issue. Changing the cron settings does not resolve this for me. Unfortunately this makes OO community server unusable.

[icewind1991]

To clarify, the "intended behaviour" is that documents are saved when the cron runs after all users close the document.

If your documents are not saving properly even some time after closing, please verify that your cron is functional (see skjnldsv's comment above) and check your nextcloud log for any errors that might have been thrown during saving.

[FreemoX]

I'm experiencing the same issues ( #103 ) as mention by OP, although without anyone else editing my file at the same time (someone was working with me on a paper, but they started working after I exited the editor and saved my file). I don't have issues with cron not working (as far as I know), so I don't know if mine and OPs issues are the same, but it seems to be an issue with the newer NextCloud versions. Seems to me that the editor either times out or gets "confused" as soon as another person edits the same file, or there is a writing issue from the cache.

[timo12357]

Our cron is functional. Nevertheless, the changes are lost. This is worth to be tagged a bug.

[HFranz]

Our cron is functional. Nevertheless, the changes are lost. This is worth to be tagged a bug.

Same here. I had this behavior, but I can not reproduce anymore. I’m using system cron on an own server.

[Ulch]

Since I also haven't been able to solve the issue by adjusting the cron settings I contacted my hoster and asked him to address the problem. His response:

At the moment I don't have a solution for you and can only ask you to be patient until a new version with a correction of this error is being published.

I really think it would be great if this could be tagged as a bug and get the attention of the developers. As others in this thread also said, this is a serious problem which makes OO community server unusable and in part jeopardizes the whole idea of Nextcloud "Hub".

[OITDE]

We've got the same problem. Please fix it.

[dhapheto]

i have got same issue

[lefpiz]

Same problem here. It seems to be a problem of writing data from the database back to the files nextcloud. I have the following warning in the /var/log/nextcloud.log all the time. Do you have the same errors in the logfile?

{"reqId":"1RFumGuNoErgHMyNyJqi","level":3,"time":"2020-04-01T09:00:22+02:00","remoteAddr":"","user":"--","app":"documentserver_community","method":"","url":"--","message":{"Exception":"OCA\DocumentServer\Document\DocumentConversionException","Message":"namespace error : Namespace prefix wp14 on sizeRelH is not defined\nr 3\" hidden=\"false\"/><wp14:sizeRelH relativeFrom=\"margin\"\n ^\nnamespace error : Namespace prefix wp14 on pctWidth is not defined\nlse\"/><wp14:sizeRelH relativeFrom=\"margin\"><wp14:pctWidth\n ^\nnamespace error : Namespace prefix wp14 on sizeRelV is not defined\n4:pctWidth>0</wp14:pctWidth></wp14:sizeRelH><wp14:sizeRelV relativeFrom=\"margin\"\n ^\nnamespace error : Namespace prefix wp14 on pctHeight is not defined\n14:pctWidth></wp14:sizeRelH><wp14:sizeRelV relativeFrom=\"margin\"><wp14:pctHeight\n ^\n","Code":0,"Trace":[{"file":"/var/www/nextcloud/apps/documentserver_community/lib/DocumentConverter.php","line":123,"function":"run","class":"OCA\DocumentServer\Document\ConverterBinary","type":"->","args":["/upload_tmp/oc_tmp_mqVjBS-.xml",null]},{"file":"/var/www/nextcloud/apps/documentserver_community/lib/DocumentConverter.php","line":78,"function":"runCommand","class":"OCA\DocumentServer\DocumentConverter","type":"->","args":[{"class":"OCA\DocumentServer\Document\ConvertCommand"}]},{"file":"/var/www/nextcloud/apps/documentserver_community/lib/Document/DocumentStore.php","line":151,"function":"saveChanges","class":"OCA\DocumentServer\DocumentConverter","type":"->","args":[" sensitive parameter replaced ",["78;AgAAADEA//8BAEtMDRkkwgUAhwAAAAEAAAAAAAAAAAAAAAAAAAAAAAAA9v///xwAAAA1AC4ANAAuADIALgA0ADYALgBAAEAAUgBlAHYA","34;BgAAADYAMQAwAAIAHAABAAAABwAAAAIAAAAAAAAAAAAAAA=="],"/var/nc_data/appdata_ocrvn9u8e3bd/documentserver_community/doc_754877581/saved.docx","docx"]},{"file":"/var/www/nextcloud/apps/documentserver_community/lib/LocalAppData.php","line":100,"function":"OCA\DocumentServer\Document\{closure}","class":"OCA\DocumentServer\Document\DocumentStore","type":"->","args":[" sensitive parameters replaced "]},{"file":"/var/www/nextcloud/apps/documentserver_community/lib/Document/DocumentStore.php","line":164,"function":"getReadWriteLocalPath","class":"OCA\DocumentServer\LocalAppData","type":"->","args":[{"class":"OC\Files\SimpleFS\SimpleFolder"},{"class":"Closure"}]},{"file":"/var/www/nextcloud/apps/documentserver_community/lib/Document/SaveHandler.php","line":56,"function":"saveChanges","class":"OCA\DocumentServer\Document\DocumentStore","type":"->","args":[754877581,[{"class":"OCA\DocumentServer\Document\Change"},{"class":"OCA\DocumentServer\Document\Change"},{"class":"OCA\DocumentServer\Document\Change"},{"class":"OCA\DocumentServer\Document\Change"},{"class":"OCA\DocumentServer\Document\Change"},{"class":"OCA\DocumentServer\Document\Change"},{"class":"OCA\DocumentServer\Document\Change"},{"class":"OCA\DocumentServer\Document\Change"},{"class":"OCA\DocumentServer\Document\Change"},{"class":"OCA\DocumentServer\Document\Change"},{"class":"OCA\DocumentServer\Document\Change"},{"class":"OCA\DocumentServer\Document\Change"},{"class":"OCA\DocumentServer\Document\Change"},{"class":"OCA\DocumentServer\Document\Change"},{"class":"OCA\DocumentServer\Document\Change"},{"class":"OCA\DocumentServer\Document\Change"},{"class":"OCA\DocumentServer\Document\Change"},{"class":"OCA\DocumentServer\Document\Change"},{"class":"OCA\DocumentServer\Document\Change"},{"class":"OCA\DocumentServer\Document\Change"},{"class":"OCA\DocumentServer\Document\Change"},{"class":"OCA\DocumentServer\Document\Change"},{"class":"OCA\DocumentServer\Document\Change"}]]},{"file":"/var/www/nextcloud/apps/documentserver_community/lib/BackgroundJob/Cleanup.php","line":69,"function":"flushChanges","class":"OCA\DocumentServer\Document\SaveHandler","type":"->","args":[754877581]},{"file":"/var/www/nextcloud/lib/public/BackgroundJob/Job.php","line":79,"function":"run","class":"OCA\DocumentServer\BackgroundJob\Cleanup","type":"->","args":[null]},{"file":"/var/www/nextcloud/cron.php","line":125,"function":"execute","class":"OCP\BackgroundJob\Job","type":"->","args":[{"class":"OC\BackgroundJob\JobList"},{"class":"OC\Log"}]}],"File":"/var/www/nextcloud/apps/documentserver_community/lib/Document/ConverterBinary.php","Line":64,"CustomMessage":"Error while applying changes for document 754877581"},"userAgent":"--","version":"18.0.3.0"}

[biva]

Shouldn't we have a unique place to discuss this issue wich looks very important to a lot of people? The oldest one I found on the nextcloud repository is https://github.com/nextcloud/documentserver_community/issues/12 and the main one on ONLYOFFICE repository is https://github.com/ONLYOFFICE/onlyoffice-nextcloud/issues/21 (with a specific interesting comment in https://github.com/ONLYOFFICE/onlyoffice-nextcloud/issues/21#issuecomment-568745636)

[timo12357]

Shouldn't we have a unique place to discuss this issue wich looks very important to a lot of people? The oldest one I found on the nextcloud repository is #12 and the main one on ONLYOFFICE repository is ONLYOFFICE/onlyoffice-nextcloud#21 (with a specific interesting comment in ONLYOFFICE/onlyoffice-nextcloud#21 (comment))

Yes it should, but not one of these reports on the issue has been tagged as a bug. Apparently it is not a bug? We solved the problem by removing Onlyoffice from our Nextcloud Hub immediately when this, erm, "undocumented feature" was identified. No files have been lost since, and work continues.

[tjareson]

We solved the problem by removing Onlyoffice from our Nextcloud Hub immediately when this, erm, "undocumented feature" was identified. No files have been lost since, and work continues.

Same here. I've tried all kind of counter measures, also running /usr/bin/docker exec -t containername sudo -u www-data php occ documentserver:flush as cron job just produced "more sophisticated" data loss, like just the changes were left in the document but the rest was missing and things alike.

Our solution: use collabora as docker container. It is not as compatible with MS office docs as onlyoffice, but right now for us the only solution that works reliable with nextcloud. Not saving changes destroys the trust in cloud based work environments and brings future discussions for me ("...it just wasn't saved, I don't know...") I can not argue with the users anymore.

It's sad that the magnitude of this issue for user trust into nextcloud is somehow not noticed/ignored.

[audelorAd]

Our solution: use collabora as docker container. It is not as compatible with MS office docs as onlyoffice, but right now for us the only solution that works reliable with nextcloud.

Why did you choose collabora ? Just for me to understand, have you tried onlyoffice as docker container instead of Nextcloud's Community Document Server app ?

I read several times that onlyoffice docker installation resolved this "unsaved document" issue.

[tjareson]

Why did you choose collabora ?

The reason was that I needed a very quick solution, as our nextcloud is a production system. So the collabora setup was somehow super easy. I just installed the container + nginx for ssl certificate (collabora is running on a different machine) and configure the URL of the nextcloud server which is allowed to use it as environment variable when starting the collabora container. That's it. On nextcloud I only needed to install collabora app instead of onlyoffice app. So that whole thing was tackled within 20 minutes. Although I would probably prefer onlyoffice container I didn't find an equally simple setup method. Any hints?

[audelorAd]

So that whole thing was tackled within 20 minutes. Although I would probably prefer onlyoffice container I didn't find an equally simple setup method. Any hints?

Not really hints sorry, cause i don't install it myself. Just seems that docker's install of onlyoffice document server seems well documented too : https://github.com/ONLYOFFICE/Docker-DocumentServer. I also prefer OnlyOffice because of MS Office better compatibility.

[Joe7new]

I don't understand that none of nextcloud team mentioned that bug so far, not even set the bug level after more than 2 months (issue #77). All the issues refer to the same problem. And contrary to the opinion of icewind1991 (above) the problem is still there, even when cron works correct .

Nextcloud is a great product and thanks to all developers etc. for great work so far !!

But sorry I could not hide a comment : _sometimes it would be better to have less features that work reliable than a lot of features that causes of great amount of requests that can't be processed ( wasn't that the origin problem of owncloud ?) .

I opened an issue (#77 ) in January (!) concerning the problem, no answer, no bug label etc... Ok, now we have the problem with that corona virus... Nevertheless: only-office could be a great feature, but useless with that risk of data loss in productive enviroment - sorry again ! lg. Joe

[timo12357]

I am afraid no one from the actual developer team is reading this thread making all this discussion useless. @icewind1991 @rullzer @nickvergessen @blizzz would you be so kind and instruct us how to contact the actual development team to get attention to this issue?

[joequant]

Hi. Looking over the code, since I'm getting hit by this bug.

What should happen is that during the cleanup process, the system will go through all of the documents and any documents that are not active will be saved. When you do a "documentserver:flush" a similar thing happens, except that all of the files are saved.

What seems to be the problem is that when you close the edit, the system thinks that the document is active when in fact it is not. So what I'll try doing is to look at the database to see the list of active documents, and see how that list changes.

Would help if anyone else has some insights on the code. Can't promise an ETA, but I'm stuck at home because of COVID.

[joequant]

Okay, I see a problem. I start up an editor. It adds an entry to the database table oc_documentserver_sess

''' nextcloud=# select * from oc_documentserver_sess; session_id | document_id | user | user_original | last_seen | readonly | user_index | username
------------+-------------+------------------+------------------+------------+----------+------------+------------- szvqefn5 | 1268142295 | ocwi3yinn69e_joe | ocwi3yinn69e_joe | 1586266102 | f | 1 | Joseph Wang '''' Now when I close the editor, it should remove this table, but it doesn't......

[joequant]

Okay. I've established that the entries are getting removed from the session table. The way that the system works is that there is an entry for the last time there is a change, and after 100 seconds after the last time it sees the session, it removes it from the database.

What should happen then is that the entries get flushed when cleanup is called by the cron job. That isn't happpening.

[joequant]

Okay it seems to work on my setup, but there is a big gotcha.

I'm using the cron backend. The sequence of events is

1) when you edit the file it puts an entry into oc_documentserver_sess 2) the system monitors for updates and changes. If it goes100 seconds without a change then it expires the session 3) then when the system runs the cleanup backend process, it saves the file nextcloud. I have my cron server run every minute.

The gotcha is that there is a lag of about three minutes after I save the file, before it commits the save, and that assumes that everything goes right. If you have an AJAX backend, and you don't touch the system, then nothing gets run, and the system might never commit.

One thing that could be done is to reduce the expires timeout from 100 seconds to something like 15 or 30 seconds.

One thing that someone can try with the AJAX backend is to come up with some event that will cause the backend process to run. Altenatively, we can add a "refresh" command or API call that runs cleanup and see if that forces a save.

One question that I have is does anyone have this working at all? Also, now that I kind of know what's happening behind the scenes, I can run some more experiments.

[joequant]

By the way, I'm packaging a ready to run system as a docker image under joequant/bitstation in the directory nextcloud. Also I'm in the process of applying for a government grant from the Hong Kong government to work on tools for physicists and so this is part of that effort.

[nickvergessen]

If you have an AJAX backend, and you don't touch the system, then nothing gets run, and the system might never commit.

See this comment https://github.com/nextcloud/documentserver_community/issues/100#issuecomment-604908119

One thing that someone can try with the AJAX backend is to come up with some event that will cause the backend process to run. Altenatively, we can add a "refresh" command or API call that runs cleanup and see if that forces a save.

We have such a backend process, its called cron. Sorry if that sounds sarcastic, but what you are askign for is cron and that is exactly what should be used. Set up systemcron to run every 5 minutes as per documentation and succeed.

[timo12357]

We have such a backend process, its called cron. Sorry if that sounds sarcastic, but what you are askign for is cron and that is exactly what should be used. Set up systemcron to run every 5 minutes as per documentation and succeed.

We have cron run every 5 minutes and unfortunately that does not work. That is exactly how we very sadly lost two days of expensive consultant work.

[FreemoX]

Can verify, the use of cron does not resolve this issue, as per my observations on my production server as well as my test / development server. Cron is set to run at 5 min intervals, as suggested by Nextcloud documentation. I haven't messed around with the other backends though

[joequant]

There are two issues:

1) first having a system that doesn't commit changes for X minutes is very uninitutive

2) one problem that I've run into is that if there is an exception while the cleanup is being called, then it's possible that the files won't get saved at all. I think I ran into this problem when one of the files causes binaryConverter to choke and because the exceptions is not caught, then the file saving stops

If I can confirm that it works on my setup, then I'll try to expose an API to run "cleanup" manually, and then people can see if that forces a save or if that chokes.

[nickvergessen]

By the way, I'm packaging a ready to run system as a docker image under joequant/bitstation in the directory nextcloud. Also I'm in the process of applying for a government grant from the Hong Kong government to work on tools for physicists and so this is part of that effort.

If you are setting up for a bigger environment like that, you should just use the Original Docker: https://github.com/nextcloud/documentserver_community#onlyoffice-components This app here is for people that can not setup docker stuff themselves to have a one-click office thing.

[joequant]

btw the use case that gave me a lot of headaches was:

1) edit a form 2) immediately make multiple copies of the form to fill out

Even a 5 minute save cycle causes a lot of confusion, since it's not obvious that once I save a file I have to wait 5 minutes before I can copy it. But now I know what's going on, I can stare at it and see if it works.

Also I tried using the original docker, but it turned out to be huge. I'm using this as part of a larger system (using jupyter and being a front end to supercomputing), and I found that the app was a lot slimmer and less resource intensive.

[joequant]

Okay it seems to be working with a long lag.

Is there any way of manually triggering a run of cron.php from the web interface? Being able to trigger that manually will help debug a lot of the complaints people are having.

[timo12357]

We are in a very deep rabbit hole: this is still not tagged as a bug.

[FreemoX]

We are in a very deep rabbit hole: this is still not tagged as a bug.

Wow, it is now

[jospoortvliet]

We are in a very deep rabbit hole: this is still not tagged as a bug.

I've tagged it as a bug, but there is nothing we can do. ONLYOFFICE has to fix this and they said they will so now we just have to wait.

Okay it seems to be working with a long lag.

Is there any way of manually triggering a run of cron.php from the web interface? Being able to trigger that manually will help debug a lot of the complaints people are having.

Only from the command line, by just running it.

Also I tried using the original docker, but it turned out to be huge. I'm using this as part of a larger system (using jupyter and being a front end to supercomputing), and I found that the app was a lot slimmer and less resource intensive.

Please understand something: This was developed for private home users who can't install a docker image but do occasionally want to edit a DOCX or PPTX file. Nothing more.

Any other use case we will never put any resources in. This is for home users and it will never be anything else. You wonder why?

1. Changing this is very hard (we already put in 1 year just to develop this). To fix this problem (and similar issues you inevitably have to start changing ONLYOFFICE itself and without a team of at least 6-7 full time people, it is impossible to maintain that. We're not going to for this. This was a gift to home users, nothing more.
2. There is no need for it. ONLYOFFICE and Collabora are perfectly fine, they are just hard to install for private users and THAT is what we fixed with this. (If there's ONE high priority thing to improve/fix, it would be support for running this on a Raspberry Pi, if you ask me...)

So, if you're not a private user and this doesn't work for you, use the docker, use Collabora, buy ONLYOFFICE and get packages - loads of options. Don't ever expect us to support this for anything other than a casual, private home use case.

Of course, this is open source, you can fork and fix it yourself. And of course we'll accept pull requests that improve things for non-home use, if they are easy to maintain at least. But that last part is going to be hard - we're really not prepared to patch ONLYOFFICE and you can't fix 90% of the issues without doing that.