search

gramps-project / gramps-web-api

A RESTful web API for Gramps - backend of Gramps Web
https://www.grampsweb.org
GNU Affero General Public License v3.0
82 stars 44 forks source link

Sync between Desktop and Gramps Web does not work as expected #503

Open hartenthaler opened 7 months ago

hartenthaler commented 7 months ago

I have an empty tree at GrampsWeb and my local tree with about 12000 persons in Gramps Desktop. When starting "Gramps Web Sync" the process starts after authorisation. Then it idles for several minutes and finally reports that there are deleted persons at Gramps Web. Then I select the option "Sync local to Web" (the second of the three options) in order to copy all the local records to Gramps Web. This stops immediately with the error message "Server authorisation error".

DavidMStraub commented 7 months ago

It's hard to diagnose without more details.

In any case, it will be much more efficient to first import a Gramps XML file in Gramps Web before the first sync. The sync addon is meant to sync two trees that are derivatives of each other, and having one completely blank side is not a good starting point.

hartenthaler commented 7 months ago

I tried now the other way as you suggested. However, importing the Gramps XML file in GrampsHub was very slow. After some hours the import was frozen at 52%. Finally this morning it reached 100%. I started syncing the now identical trees (local and GrampsHub) again. After a short time, the process stopped with a "504" error when accessing the server. I tried it a second time with the same error.

DavidMStraub commented 7 months ago

Thanks for the info. You are stress testing Gramps Web, which is useful :slightly_smiling_face:

According to the logs the import took 12.5 hours and completed without errors. I suspect the time was spend mostly updating the search index, not importing the objects (actually the new progress indicator shows what it's currently doing when you hover over it).

It would be useful if you could trigger an "update search index" by clicking the button in the admin settings and tell me at what time you clicked it. Then we would find out how long the reindex on its own takes for your tree.

hartenthaler commented 7 months ago

I started the "update search index" at 24.03.2024 17:58. The process stoped successfully at 18:05 (7 minutes later).

DavidMStraub commented 7 months ago

Thanks! So it's the XML import after all that takes so long. Good to know.

DavidMStraub commented 6 months ago

This is still not resolved, is it?

hartenthaler commented 3 months ago

Yes, this is still open. Started from scratch again, and uploaded XML, using the actual versions of Gramps Desktop 5.2.3 and GrampsHub.

DavidMStraub commented 3 months ago

I think the slow XML import is now solved by #530.

And I think I also understood the 504 on sync - it is because when using sync, all changes are done as part of a single HTTP request, not as a background task like the XML import, so depending on server configuration, that will time out after 1 or 2 minutes. That means that the sync addon is not suited to synchronize an entire tree from A to B. The right workflow would be to import an XML file and then synchronize. We should make sure our documentation reflects this fact.

Going forward, an option could be to have the sync addon offer an import if it finds the remote side is empty, but that would require some work and anyway shouldn't be tracked here.

So I'll change the label to documentation and close once I've checked that it's documented correctly.

hartenthaler commented 3 months ago

The SYNC problem is not a documentation problem. It is a still existing real problem. I followed your recommendation: export of XML at Gramps Desktop and import of XML in an empty tree at GrampsHub. Finally, I was uploading all media files successfully to GrampsHub. Rebuilding search index at GrampsHub. So both databases should now be identical. Starting SYNC at Gramps Desktop still results in a 504 error after about 60 seconds. Tested the SYNC again today and the 504 is still there.

DavidMStraub commented 3 months ago

It may be a real problem, but of https://github.com/DavidMStraub/gramps-web-sync, not https://github.com/gramps-project/gramps-web-api :slightly_smiling_face: - That's what this issue is about.

DavidMStraub commented 3 months ago

To be precise, there is nothing in Gramps Web API that can be improved to resolve this sync problem.

Illia-M commented 2 weeks ago

I have same problem with 504 because nginx in my self-deployed Gramps Web cut requests by timeout, solved by nginx config adjustment.

But my problem with sync next come in timeout of gunicorn workers that "hardcoded" to 120 seconds. I see in DB that transactions produced by Gramps Web Sync with file export need more time to read full tree data.

Tree 31k+ persons. Postgres DB image from your docs.

Gramps 5.2.2 Gramps Web API 2.4.2 Gramps Web Frontend 24.8.0 Gramps QL 0.3.0 locale: en multi-tree: false task queue: true

DavidMStraub commented 2 weeks ago

Yes, this is a problem in particular when syncing large amounts of data. The solution will be to use the background worker for the /api/transactions/ endpoint. I will open a separate issue for that.

PS: 31k+ persons is a very large database, so it's possible you might hit various limitations. It's good to get feedback what works and what doesn't.

Illia-M commented 2 weeks ago

Wait a lot of reports 😁27 жовт. 2024 р. о 10:20 David Straub @.***> пише: Yes, this is a problem in particular when syncing large amounts of data. The solution will be to use the background worker for the /api/transactions/ endpoint. I will open a separate issue for that. PS: 31k+ persons is a very large database, so it's possible you might hit various limitations. It's good to get feedback what works and what doesn't.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>