API: Expose index stats

[abitrolly]

We need to see through cyber-ui when indexator is not functional. For example, when it can not access underlying Parity server.

Possible implementation is that the error should propagate to http://search.cyber.fund/cybernode status page and change the cyber-search status to yellow with exact message "ETH node (Parity) is not accessible for indexing".

[hleb-albau]

Is this functionality really needed? We will have page with latest blocks for all systems, we will have page with index size, count and etc cluster info.

[abitrolly]

Yes. This is really needed.

The story is that person wants to find his transaction that was just posted on the chain. He opens search, enters transaction and gets nothing. Status shows green. The person leaves.

If indexator does not function, the error message could be exposed in http://search.cyber.fund/cybernode status page together with yellow status for Search API component. This creates a feedback loop with a user and tells him what's going on. User can report specific error to us and track the progress. Makes it cybernetic. )

[abitrolly]

We will have page with latest blocks for all systems, we will have page with index size, count and etc cluster info.

How do you see API that returns cluster info page? The critical part is only the service health check with error message in UI, so maybe leave cluster info and status checks separated.

[cyborgshead]

@YodaMike please comment on API for state of indexation story

• API with stats from Elastic (from Elassandra)
• API for indexation status (how many blocks indexed and how many are left)

[YodaMike]

1. API for indices stats is implemented, there are pretty big json with a lot of stats so it is necessary to identify which data is needed.
2. we need to implement API for information about not indexed data (remained blocks)

[hleb-albau]

@SiarheiSadau we should discuss, where and what info we will show to the users and then define goals for this task

[abitrolly]

Feel free to discuss it right here. What is wrong with the scheme suggested in initial issue comment?

On Mar 23, 2018 7:48 PM, "Hleb Albau" notifications@github.com wrote:

@SiarheiSadau https://github.com/siarheisadau we should discuss, where and what info we will show to the users and then define goals for this task

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/cyberFund/cyber-search/issues/30#issuecomment-375654469, or mute the thread https://github.com/notifications/unsubscribe-auth/AIX9MyCgHWVTpWlZ2NOH-ROIXInm6ySMks5thO8YgaJpZM4RHBny .

[hleb-albau]

@abitrolly It's not wrong, we should just explicit declare what to expose according to cyber-browser requirements. I just want more info here.

[hleb-albau]

Rename task In scope of this taks should be done:

• [ ] Entry point returning information about elassanda index state[sum tx number,blockhains numer,db size in gb]
• [ ] Reflect changes to the search-api-docs

All other staff related to search healthy will be done in separate issue.

All required info can be obtained by processing result of elastic search default stats endpoint.

[hleb-albau]

http://browser.cybersearch.io/

https://github.com/cybercongress/cyber-browser/blob/master/docs/Software_Requirements.md#1-main-page

Search in [134 M] transactions in [2] blockchain systems with [135] parsed tokens. Database size : [369 GBs]

[arturalbov]

Endpoint for statistics: http://api.cybersearch.io/search/stats