search

moqui / moqui-framework

Use Moqui Framework to build enterprise applications based on Java. It includes tools for databases (relational, graph, document), local and web services, web and other UI with screens and forms, security, file/resource access, scripts, templates, l10n, caching, logging, search, rules, workflow, multi-instance, and integration.
http://www.moqui.org
Other
279 stars 199 forks source link

ES sync fail #592

Open daiagou opened 1 year ago

daiagou commented 1 year ago

Moqui synchronization is a single synchronization. If the data volume is relatively large, such as tens of thousands of product data or party data. So there will be a timeout situation. Can we add a batch synchronization or a synchronization failure retry operation for failed data.

apathforward commented 1 year ago

Yes, you can add it. Submit a PR for review.

On Thu, Jun 1, 2023 at 8:14 PM abner @.***> wrote:

Moqui synchronization is a single synchronization. If the data volume is relatively large, such as tens of thousands of product data or party data. So there will be a timeout situation. Can we add a batch synchronization or a synchronization failure retry operation for failed data.

— Reply to this email directly, view it on GitHub https://github.com/moqui/moqui-framework/issues/592, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFK5KLXGBCDP5F26NCR3Z4LXJE46RANCNFSM6AAAAAAYXVDTFI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- Guru Dharam Singh Khalsa

eigood commented 1 year ago

Yes, this happens to us too, when restoring database snapshot dumps in staging/developer environments. The ES data folders are not saved, so at load time(from empty database), hundreds of thousands of WorkEfforts are sent over to ES, and it takes a very very long time.

For us, staging/developer stations have configured Postgresql to have extremely long commit times, so it runs a bit faster, but it still takes quite a bit of time.

This has been on my radar as something I want to fix, such that my local moqui has an uncommitted series of changes that print out a few basic things in this area of the code.