[Feature] Self-Observability of the query subsystem in BanyanDB

apache / skywalking

APM, Application Performance Monitoring System

https://skywalking.apache.org/

Apache License 2.0

23.54k stars 6.47k forks source link

[Feature] Self-Observability of the query subsystem in BanyanDB #10561

Open lujiajing1126 opened 1 year ago

lujiajing1126 commented 1 year ago

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description

In order to evaluate the performance of BanyanDB, in particular the query subsystem, with real traffic and load, EXPLAIN is necessary. This would show how physical plans are organized hierarchically according to the data distribution and predefined optimization rules.

Also, to better understand how various (internal) physical plan works, self-observability (both metrics and tracing) has to been built. Basically, the following tasks should be done,

Support EXPLAIN[1] for both measure query and stream query
Add self-observability including trace and metrics for query subsystem
Support EXPLAIN in the client SDK & CLI and add query plan visualization in the UI

[1]: EXPLAIN in MySQL

Use case

No response

Related issues

No response

Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Nageshbansal commented 1 year ago

Hello @lujiajing1126, I am interested in working on this idea for GSoC. Can you provide some guidance on how to approach this project and what its basic requirements are? I would appreciate it if you could suggest some issues that I can work on to gain more experience for this project. Thank you.

lujiajing1126 commented 1 year ago

Hello @lujiajing1126, I am interested in working on this idea for GSoC. Can you provide some guidance on how to approach this project and what its basic requirements are?

The basic requirement can be found here, https://issues.apache.org/jira/browse/GSOC-125

Familiar with Go
Have a basic understanding of database query engine
Have an experience of Apache SkyWalking or other APMs

Please note this project is a full-time project and surely more demanding than #10562

I would appreciate it if you could suggest some issues that I can work on to gain more experience for this project. Thank you.

You may filter by database tag in the GitHub issues. Besides, I would remind you that the deadline for the application this year is APRIL 05, 2023 2:00 AM.

Nageshbansal commented 1 year ago

Thanks, @lujiajing1126. Please let me know if there is a particular template for the GSOC Project proposal or any specific requirements that need to be included in the project proposal. And yes, I am aware of the deadline, and I'll try to send you the proposal as soon as possible.

Superskyyy commented 1 year ago

Thanks, @lujiajing1126. Please let me know if there is a particular template for the GSOC Project proposal or any specific requirements that need to be included in the project proposal. And yes, I am aware of the deadline, and I'll try to send you the proposal as soon as possible.

No there isn't a particular template though I will highly suggest referring to any successful GSOC proposal from previous years. (There are plenty when you search on GitHub :))

Nageshbansal commented 1 year ago

Hello @lujiajing1126, I am Nagesh Bansal and I would like to confirm my participation in the GSOC 2023 project that we discussed.

lujiajing1126 commented 1 year ago

Hello @lujiajing1126, I am Nagesh Bansal and I would like to confirm my participation in the GSOC 2023 project that we discussed.

Assigned to you

Nageshbansal commented 1 year ago

Hello @lujiajing1126, I was wondering if there are any other tasks I can assist with or if I should proceed with working on the proposed idea. Sorry for not getting back to you sooner, as I was busy with my university exams. :)

lujiajing1126 commented 1 year ago

Hello @lujiajing1126, I was wondering if there are any other tasks I can assist with or if I should proceed with working on the proposed idea. Sorry for not getting back to you sooner, as I was busy with my university exams. :)

I've just replied your mail few seconds ago. Take your time.

lujiajing1126 commented 1 year ago

The project has been rejected in GSoC 2023 while @Nageshbansal will continue working on this project.

wu-sheng commented 11 months ago

Is this still on process?

gavinnwang commented 3 months ago

Hi!

I am an prospective GSoC contributor with a profound interest in database internals. I am enthusiastic about the prospect of dedicating my efforts to this topic this summer.

I engaged in detailed discussion with @lujiajing1126 about specific implementation and requirements.

My proposal has been carefully crafted and duly submitted via the GSoC application portal. I am excited for the opportunity to contribute to this project!

wu-sheng commented 1 month ago

As this is a part of BanyanDB 0.7 and SkyWalking 10.1, I have assigned this to @hanahmily and add the milestone.