Code Insights for Apache StreamPipes

bossenti commented 1 year ago

Body

Apache StreamPipes

Apache StreamPipes (incubating) is a self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams. StreamPipes offers several modules including StreamPipes Connect to easily connect data from industrial IoT sources, the Pipeline Editor to quickly create processing pipelines and several visualization modules for live and historic data exploration. Under the hood, StreamPipes utilizes an event-driven microservice paradigm of standalone, so-called analytics microservices making the system easy to extend for individual needs.

Background

StreamPipes is grown significantly throughout recent years. We were able to introduce a lot of new features and attracted both users and contributors. Putting the cherry on the cake, we were graduated as an Apache top level project in December 2022. We will of course continue developing new features and never rest to make StreamPipes even more amazing. Although, since we are approaching with full stream towards our 1.0 release, we want to project also to get more mature. Therefore, we want to address one of our Achilles' heels: our test coverage.

Don't worry, this issue is not about implementing myriads of tests for our code base. As a first step, we would like to make the status quo transparent. That means we want to measure our code coverage consistently across the whole codebase (Backend, UI, Python library) and report the coverage to codecov. Furthermore, to benchmark ourselves and motivate us to provide tests with every contributing, we would like to lock the current test coverage as an lower threshold that we always want to achieve (meaning in case we drop CI builds fail etc). With time we then can increase the required coverage lever step to step.

More than monitoring our test coverage, we also want to invest in better and more clean code. Therefore, we would like to adopt sonarcloud for our repository.

Tasks

[ ] calculate test coverage for all main parts of the repo
[ ] send coverage to codeCov
[ ] determine coverage threshold and let CI fail if below
[ ] include sonarcloud in CI setup
[ ] include automatic coverage report in PR validation (see an example here) -> optional
[ ] include automatic sonarcloud report in PR validation -> optional
[ ] what ever comes in your mind 💡 further ideas are always welcome

❗ Important Note ❗

Do not create any account in behalf of Apache StreamPipes in Sonarcloud or in CodeCov or using the name of Apache StreamPipes for any account creation. Your mentor will take care of it.

Relevant Skills

basic knowledge about GitHub worfklows

Learning Material

GitHub workflow docs
Apache StreamPipes workflows
Sonarcloud for Monorepos
Using code cov for a monorepo: https://www.curtiscode.dev/post/tools/codecov-monorepo/ & https://docs.codecov.com/docs/flags

Mentor

Tim Bossenmaier (bossenti[at]apache.org).

StreamPipes Committer

I acknowledge that I am a maintainer/committer of the Apache StreamPipes project.

bossenti commented 1 year ago

This issue is intended to be an idea for this year's GSOC. If you are interested in taking part at GSOC via the ASF, please have a look on this guide: https://community.apache.org/gsoc.html In case there are any questions related to the GSOC process or this issue in particular, feel free to reach out to us 🤝

Kshitiz-Mhto commented 1 year ago

hello @bossenti , i am interested in this project to contribute in this issue for GSOC 2023, Currently i am exploring its components and playing with it a little bit and understanding it. thank you!

bossenti commented 1 year ago

Welcome to StreamPipes @Kshitiz-Mhto 👋🏼 Great that you have discovered our project and are interested in this issue Getting a basic understanding of the project is indeed a good first step. If you need any help for both getting familiar with StreamPiper and the GSOC procedure, please feel free to reach out to me 🙂

bossenti commented 1 year ago

PS: If not already done, it would be great if you could subscribe to our mailing list (dev@streampipes.apache.org)

bossenti commented 1 year ago

@Kshitiz-Mhto if you are still interested in working on this topic as part of this year's GSOC, I would like to kindly remind you to write your project proposal and submit it by 2023/04/04 (https://summerofcode.withgoogle.com/programs/2023). Here is more information on how to write a proposal: https://google.github.io/gsocguides/student/writing-a-proposal In case there any open questions, feel free to reach out Good luck 🙂

Kshitiz-Mhto commented 1 year ago

Thank you @bossenti , i would love to be part of StreamPipes and GSOC. I have already started writing proposal. One question, for early review where/to whom should i send my proposal for review?

any any further doubt, i will reach out to you. thank you

Kshitiz-Mhto commented 1 year ago

hello @bossenti
would you provide your feedback on my proposal demo link:
https://docs.google.com/document/d/1Prp3sv5IVTnNNliwr58FOcvAUwqjyuhc7qaqLy3vYrg/edit?usp=share_link

Schedule of Delivery hasnot be fixed yet, i m working on it., Thank you

bossenti commented 1 year ago

Hi @Kshitiz-Mhto I've read your proposal Thanks for putting that much effort into it. As of now I'm not sure if I'm allowed go provide you with feedback policy-wise (both Google and ASF) I'll add some comments in the Google doc if it's possible Otherwise I would strongly encourage you to submit your proposal as it is before the deadline passes 🙂

Kshitiz-Mhto commented 1 year ago

Hi @Kshitiz-Mhto I've read your proposal Thanks for putting that much effort into it. ....

Thank you! some comments in the google docs would be wonderful. I would really appreciate your feedback and will make changes according to your feedback. Thank you. further, i have submitted the proposal on Apache foundation org in GSOC org. page.

bossenti commented 1 year ago

@Kshitiz-Mhto can you provide me with a link that allows me to add comments to your GDoc? 🙂

Kshitiz-Mhto commented 1 year ago

@Kshitiz-Mhto can you provide me with a link that allows me to add comments to your GDoc? slightly_smiling_face

here it is @bossenti https://docs.google.com/document/d/1Prp3sv5IVTnNNliwr58FOcvAUwqjyuhc7qaqLy3vYrg/edit?usp=sharing thank you !

Kshitiz-Mhto commented 1 year ago

hello @bossenti , unfortunately i didnt get selected this year but can i still work on this issues ?

bossenti commented 1 year ago

Hey @Kshitiz-Mhto that's sad to hear, but really great that you are still motivated to take on this task. We would really appreciate any contribution here though 🚀

With which part of the whole aspect would like to start? I would suppose to define some dab tasks for this quite large topic. For example, we could start with the Python part, try everything out and then adopting for the UI and the backend.

bossenti commented 1 year ago

@Kshitiz-Mhto since I didn't hear from you again, I'm assuming you are no longer willing to support here, right?

apache / streampipes