search

opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
9.73k stars 1.8k forks source link

Fix uncommittedOperations and uncommittedSizeInBytes in translog stats for remote enabled indexes #9927

Open ashking94 opened 1 year ago

ashking94 commented 1 year ago

Is your feature request related to a problem? Please describe. In local translog, the uncommitted generation is derived using local checkpoint of safe commit, but for remote store it would depend on the minRemoteGenReferenced and minSeqNoToKeep we have introduced in RemoteFsTranslog class which is incremented on account of segment upload. So, what we show basis the local translog is not really committed in remote store yet.

Describe the solution you'd like Use minRemoteGenReferenced and minSeqNoToKeep in RemoteFsTranslog to derive the translog stats for uncommitted operations / bytes.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.

ohbus commented 4 months ago

If this is available to pick up can you elaborate more so I can work on this issue?

I'm unsure if this is a beginner-friendly issue as I'm looking for my first contribution to OpenSearch.

ashking94 commented 4 months ago

If this is available to pick up can you elaborate more so I can work on this issue?

I'm unsure if this is a beginner-friendly issue as I'm looking for my first contribution to OpenSearch.

Thanks @ohbus for your interest on picking this up. This is available to be picked up. I can elaborate more about this issue. You can get started by exploring the TranslogStats class. You can also refer the public documentation here. Node stats api today provides important stats for translog that helps understand the uncommitted operations and the uncommitted size in bytes of these operations. Operations become committed in non-remote store indices after flush, but with remote store indexes, they become committed after refreshes. This is the difference that we want to also incorporate in the translog stats.

ashking94 commented 4 months ago

@ohbus This might be slightly above beginner friendly, but you can always get the information & help that you need for your contribution.

ohbus commented 4 months ago

Thank you so much for the explanation and clarification @ashking94

I would go through the resources and ask you for more clarification.

Just to know, is there a deadline for this issue or some specific timeframe when it is due?