search

apache / inlong

Apache InLong - a one-stop, full-scenario integration framework for massive data
https://inlong.apache.org/
Apache License 2.0
1.4k stars 530 forks source link

[INLONG-111] Improve filter performance and increase the number of concurrent filters #710

Closed dockerzhang closed 3 years ago

dockerzhang commented 3 years ago

At present, the number of concurrent filtering copies supported by the system is about 20 under 1k Topic. This function is very popular with users after it is launched. If we can increase the number of concurrent filtering consumption, we can greatly help the business.

1. Structured index storage: optimize the current index storage, for example, increase the structured index storage, which can be quickly retrieved through the index when in use to quickly locate the data; the increase in the index structure may make the write request slower, At the same time, it takes more time to check and restore the index when the system restarts

2. Cross-partition consumption ability: For data that is not sensitive to partitions, when filtering consumption, it should support cross-partition consumption instead of specifying partition consumption to improve the overall filtering consumption capacity, but there must also be a way to limit this Mechanism, the specific choice should be controlled by the user

JIRA link - [INLONG-111] created by gosonzhang

gosonzhang commented 3 years ago

Closed due to long periods of inactivity

Please reopen it if needed