search

moqui / moqui-framework

Use Moqui Framework to build enterprise applications based on Java. It includes tools for databases (relational, graph, document), local and web services, web and other UI with screens and forms, security, file/resource access, scripts, templates, l10n, caching, logging, search, rules, workflow, multi-instance, and integration.
http://www.moqui.org
Other
284 stars 204 forks source link

Data import does not support batch insertion operations #593

Open daiagou opened 1 year ago

daiagou commented 1 year ago

Moqui's tools include dataimport, but they do not support batch operations. When there is a large amount of data, there will be timeout/failure issues. At this point, the timeout time can be increased (default 60s), But there are several issues:

  1. When deploying online, Moqui is usually not external, and the outermost layer usually has nginx/address. However, changing the Moqui timeout does not take effect.
  2. Due to the lack of batch operations, the import process is very slow. Even if Ops personnel agree to adjust the timeout time, this time cannot be estimated due to the amount of data. Usually, it is not recommended to adjust the timeout to a very large value. So it is recommended to determine whether the imported data is duplicate and can support batch inserts? Especially for importing in CSV format.
eigood commented 1 year ago

Could you provide some further examples of what exactly is occurring? Repeatable steps, so that others can help to isolate what the problem(s) actually are?