search

dockerzhang / incubator-inlong

Apache InLong - a one-stop data streaming platform
https://inlong.apache.org/
Apache License 2.0
0 stars 0 forks source link

[INLONG-111] Improve filter performance and increase the number of concurrent filters #111

Open dockerzhang opened 3 years ago

dockerzhang commented 3 years ago

At present, the number of concurrent filtering copies supported by the system is about 20 under 1k Topic. This function is very popular with users after it is launched. If we can increase the number of concurrent filtering consumption, we can greatly help the business.

1. Structured index storage: optimize the current index storage, for example, increase the structured index storage, which can be quickly retrieved through the index when in use to quickly locate the data; the increase in the index structure may make the write request slower, At the same time, it takes more time to check and restore the index when the system restarts

2. Cross-partition consumption ability: For data that is not sensitive to partitions, when filtering consumption, it should support cross-partition consumption instead of specifying partition consumption to improve the overall filtering consumption capacity, but there must also be a way to limit this Mechanism, the specific choice should be controlled by the user

JIRA link - [INLONG-111] created by gosonzhang

github-actions[bot] commented 2 years ago

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

github-actions[bot] commented 2 years ago

This issue is stale because it has been open 60 days with no activity.

github-actions[bot] commented 2 years ago

This issue is stale because it has been open for 60 days with no activity.