search

openSUSE / mentoring

The openSUSE Developer Mentoring Program
http://101.opensuse.org
60 stars 49 forks source link

Analytics Edge Ecosystem Workloads #186

Open bwgartner opened 2 years ago

bwgartner commented 2 years ago

Project Title: Create open source sample microservice workload deployments and interfaces, spanning distributed edge-core-cloud infrastructure to address market verticals to address analytics per market verticals

Description: In the context of this Analytics Edge Ecosystem, such a digital transformation approach to cloud-native containerizing addresses how one can help strategically iterate and perform the necessary and even more comprehensive business analytics requirements. This can be done across a truly distributed compute infrastructure (with various resource options), leveraging compute elements closer to where data is generated or provided initially. Then you can more rapidly adapt the reaction, formulate predictions and forecasts, and have a larger context of all the elements that can potentially impact your business positively and internally, plus for consumers, providers, and partners.

Deliverable: Provide an opensource-based example deployment of a trained/tested/functional workload available for use cases:

Mentor: @bwgartner + more/others (Ann, Krish, Navin, Terry)

Skills / Layers / (Roles):

Skill Level: Easy, Medium

Get started:

ahmadateya commented 2 years ago

Hello, I am coming from GSoC/openSUSE 101 I am interested in this project but I didn't get the idea of it, would you provide more Description about it? and how the skills needed related to the project especially the Data/Ops, Data Analysis, and MLOps Part?

abhi-bhatra commented 2 years ago

Hi @bwgartner @mrjazzcat This projects really interests me. The required skills completely aligns with my tech stack. I had a good working experience with Microservice architectures, k8s, Python, and currently I am working as a DevOps Engineer Intern with a Telecom Blockchain based company, and again it's an open-source organization. I studied about Rancher, it is itself one of the leading multi-cloud container management platform. I just need to know more about the task! Is it like that, I have to create an example use-case where I could create a sample workload which shows the use of Rancher on Multi-cloud platforms. If this is the case, I have an implementation idea of Telecom structure and how Rancher could be used !

bwgartner commented 2 years ago

Hello, I am coming from GSoC/openSUSE 101 I am interested in this project but I didn't get the idea of it, would you provide more Description about it? and how the skills needed related to the project especially the Data/Ops, Data Analysis, and MLOps Part?

Thanks for reaching out @ahmadateya, glad you have interest in this.

The goal is to create some functional example workloads (mostly microservices) that span the distributed computing infrastructure (multiple edge tiers and types of nodes to core/cloud) and effectively address the complete data science pipeline aspect for any of the business verticals to potentially create viable analytics.

So this can be a way to learn any portion of those roles/frameworks

to more effectively help complete the ecosystem's context.

bwgartner commented 2 years ago

Hi @bwgartner @mrjazzcat This projects really interests me. The required skills completely aligns with my tech stack. I had a good working experience with Microservice architectures, k8s, Python, and currently I am working as a DevOps Engineer Intern with a Telecom Blockchain based company, and again it's an open-source organization. I studied about Rancher, it is itself one of the leading multi-cloud container management platform. I just need to know more about the task! Is it like that, I have to create an example use-case where I could create a sample workload which shows the use of Rancher on Multi-cloud platforms. If this is the case, I have an implementation idea of Telecom structure and how Rancher could be used !

Sounds like you are totally able to contribute to a project like this, @abhi-bhatra. And yes, much of the technolgy infrastructure is available in Rancher for the distributed compute aspects of an edge ecosytem, so would like to have a sample set of such example workloads to show how this can be effectively done to help such a digital transformation for improved business analytics.

abhi-bhatra commented 2 years ago

That's really insightful Bryan ! I will look into it, research and develop more used cases

abhi-bhatra commented 2 years ago

Hi @bwgartner & mrjazzcat, I am building a sample blockchain protocol application, and using Rancher to manage the cluster.

  1. The cluster is created on Azure AKS
  2. Rancher is deployed on Azure VM
  3. Accessing Rancher Dashboard from VM's IP
  4. Deployed Blockchain based Microservie to AKS
  5. I am trying to write a detailed sample, I will share it with you.

It would be great if you visit this link and provide insightful feedback on whether my proposal fits to the problem proposed. Here's the link: https://github.com/abhi-bhatra/Rancher-Demo

Current Problems and Plans:

  1. TCP connection timed out when I try to register and import Rancher to the cluster
  2. Currrently, I have used only one protocol, i.e., ETH, planned to expand it over various other protocols and want Rancher to provision other protocols. Regards @abhi-bhatra
bwgartner commented 2 years ago

hello @abhi-bhatra

Thanks and quite impressed with what you have done so far ... documenting the steps, configs and capturing images.

In the next day or so, I'll try your step 5 (across the frontend/backend/database) on some Rancher managed clusters that I have already setup (non-Azure). Will also drill into the port/protocol aspect, too and will share what I find.

Some intial observations:

abhi-bhatra commented 2 years ago

Hi @bwgartner Thank you for this overwhelmed response. I really appreciate the time you put into the project. And also:

  1. I am planning to add the Dockerfile for the respected protocols.
  2. I will make the Blockchain section more descriptive to give you deeper insights over what I am aiming to achieve with this use case.
  3. And yes for PVC, it is used by "ethereumdb" mentioned in ./database/postgresql.yaml
  4. I am working on adding more protocols such as Fabric, Phaeton, Polygon, Corda, etc...
  5. I have created helm charts as well.
  6. Rancher's Fleet is something, I am studying of today, throughout the day. What I was thinking of that, I should setup one environment, let's say staging and then create helm chart for the same, then I will integrate Rancher to manage my cluster. Also, I am learning the Fleet, so that I can setup another environment, let's say production using Rancher on different namespace or a cluster ! I want to ask that can Fleet works on managing different namespaces for different environments on a single cluster or we need to set-up separate cluster to show the power of fleet. What will be more impactful ?
navin772 commented 2 years ago

Hey, I came across this project in Gsoc 2022 under openSUSE and it seems quite interesting to me. I have intermediate knowledge in docker, containers and kubernetes and expertise in python and handling virtual machines. I want to learn more about your Rancher project and enhance my skills by working on this project. Could you please help me get started. Where and how can I contribute ?

abhi-bhatra commented 2 years ago

Hi @bwgartner I navigated through openSUSE mentoring repo, but I did not find any good-first issue. I want to contribute in this project and already building a use-case. It would be great if you could point me to any bug, or good-first issues, so that I can start contributing and prepare my proposal ASAP !!

bwgartner commented 2 years ago

Hey, I came across this project in Gsoc 2022 under openSUSE and it seems quite interesting to me. I have intermediate knowledge in docker, containers and kubernetes and expertise in python and handling virtual machines. I want to learn more about your Rancher project and enhance my skills by working on this project. Could you please help me get started. Where and how can I contribute ?

Hello @navin772 ... apologies for the laggy reply, for some reason this didn't show up in my inbox.

Given that you cite intermediate knowledge around the basic technologies, feel free to visit https://rancher.com/learn-the-basics (plus Rodeos and Master Class series) and look at the Learn pull-down for Grow your skills as well.

And with the GSOC apply timeline in progress, I'll likely start putting some issue requests out there, both for prep and for actual project deliverables, plus I'll likely have some regular "sprint-like" meetings for joint discussions.

bwgartner commented 2 years ago

Hi @bwgartner I navigated through openSUSE mentoring repo, but I did not find any good-first issue. I want to contribute in this project and already building a use-case. It would be great if you could point me to any bug, or good-first issues, so that I can start contributing and prepare my proposal ASAP !!

Hi @abhi-bhatra ... at the end of this week, I'll start posting some issues (with requested deliverables) for contributors to begin reviewing and opting it to with their proposals. As a reminder, there will be a general request and then multiple parallels for the various verticals to try to emulate and provide example deliverables for.

abhi-bhatra commented 2 years ago

Hi @bwgartner today is the deadline of the proposal for GSoC, so I need to attach the proposal for the same Project Idea by today itself. Then as soon as you post the issues, I will start working on the issues. Here is the proposal I am submitting, could you please review it once, if time allows: https://docs.google.com/document/d/1vG8T4jRYwFMq5CU0UboPwQoOiH-xhG9TKKi0-u7MYoQ/edit?usp=sharing

bwgartner commented 2 years ago

Hi @bwgartner today is the deadline of the proposal for GSoC, so I need to attach the proposal for the same Project Idea by today itself. Then as soon as you post the issues, I will start working on the issues. Here is the proposal I am submitting, could you please review it once, if time allows: https://docs.google.com/document/d/1vG8T4jRYwFMq5CU0UboPwQoOiH-xhG9TKKi0-u7MYoQ/edit?usp=sharing

Looking very good and relevant to the DataOps aspect of Edge computing/analytics, for multiple verticals

abhi-bhatra commented 2 years ago

Hi @bwgartner thanks for selecting me for this Project. I wanted to start my work on this project. But before starting, I want you to guide me for some initials, on how to proceed and what will be the possible next steps. Also, I wanted to know about our communication channel so that we can sync-up on a regular basis !!

Coldman22 commented 1 year ago

Hi @bwgartner + @mrjazzcat I am new to this community. I am have basic knowledge about docker, ansible, Linux, git & GitHub, right now I am still learning Kubernetes. I find your project very appealing and intriguing , and peaked my interest towards it. I always wanted to contribute to open source and was finding my niche where I would be able to contribute and learn. I hope you can guide me through this process of contribution and learning.

bwgartner commented 1 year ago

Hi @bwgartner + @mrjazzcat I am new to this community. I am have basic knowledge about docker, ansible, Linux, git & GitHub, right now I am still learning Kubernetes. I find your project very appealing and intriguing , and peaked my interest towards it. I always wanted to contribute to open source and was finding my niche where I would be able to contribute and learn. I hope you can guide me through this process of contribution and learning.

Hello @Coldman22 ... thanks for reaching out on this. Feel free to send me an email, describing more of what you'd like to learn about. I can certainly share some existing classes, material, and documents to help you move forward and potentially we can interact on what kinds of topics you'd like to try and contribute to.

Also an upcoming discussion/session - https://news.opensuse.org/2022/12/14/project-plans-workshop-to-grow-mentorship-efforts/ which you can join and suggest some project ideas as well

ggold7046 commented 1 year ago

Hello @bwgartner , I have basic knowledge of python, git,linux. Will I be able to contribute to this project ?

harshalmittal4 commented 1 year ago

Hello @bwgartner, I find this project interesting and wish to contribute to this idea as part of GSoC. I will learn from the attached Getting Started docs. Can you also please help me with finding some specific issues that I must start with.

Thanks!

ggold7046 commented 1 year ago

Hello @bwgartner , I have basic knowledge of python, git,linux. Will I be able to contribute to this project ?

Hello @bwgartner and @mrjazzcat , any update on this ?

bwgartner commented 1 year ago

Hello @bwgartner , I have basic knowledge of python, git,linux. Will I be able to contribute to this project ?

That seems like a good basis for contributing, @ggold7046 (with quite a bit to learn and play with). Feel free to join via the GSOC 2023 option.

bwgartner commented 1 year ago

Hello @bwgartner, I find this project interesting and wish to contribute to this idea as part of GSoC. I will learn from the attached Getting Started docs. Can you also please help me with finding some specific issues that I must start with.

Hi @harshalmittal4 ... I've also updated the main content with more things for you to review and try. And hoping GSOC 2023 contribution would work for you

Thanks!

harshalmittal4 commented 1 year ago

Thanks @bwgartner , I will check the docs and deliverables and pick some issues in the project. My job will not hinder me from contributing to the project because I find Rancher very interesting and wish to spend time learning and contributing to it, and my job is flexible.

KrishAryan commented 1 year ago

Hello, @bwgartner I want to take contribute to this idea as part of GSoC. I find this project interesting as I know the basics of Linux, Docker, Git, Gitops, Kubernetes, and helm but have no experience and no projects. I want to solidify my skills by working on this project please help me with finding some specific issues that I must start with.

bwgartner commented 1 year ago

Hello, @bwgartner I want to take contribute to this idea as part of GSoC. I find this project interesting as I know the basics of Linux, Docker, Git, Gitops, Kubernetes, and helm but have no experience and no projects. I want to solidify my skills by working on this project please help me with finding some specific issues that I must start with.

Hi @KrishAryan ... looking forward to having you apply for this year's GSOC and then learn/contribute/share things

jxr98 commented 1 year ago

Hi everyone. I am Xinrui, and I am currently a graduate student in system design engineering. I am interested in learning more about Rancher. Could someone please suggest some easy issues for me to start with? I noticed that all issues under the 'good-first-issue' label are closed. Thanks.

bwgartner commented 1 year ago

Hi everyone. I am Xinrui, and I am currently a graduate student in system design engineering. I am interested in learning more about Rancher. Could someone please suggest some easy issues for me to start with? I noticed that all issues under the 'good-first-issue' label are closed. Thanks.

Hello @jxr98 / Xinrui. You can certainly do a quick learn/try via https://www.tryrancher.com/ and also review the docs at https://rancher.com/docs/ . Feel free to reach out to me with any more questions

bwgartner commented 1 year ago

Hello, @bwgartner I want to take contribute to this idea as part of GSoC. I find this project interesting as I know the basics of Linux, Docker, Git, Gitops, Kubernetes, and helm but have no experience and no projects. I want to solidify my skills by working on this project please help me with finding some specific issues that I must start with.

Hi @KrishAryan ... looking forward to having you apply for this year's GSOC and then learn/contribute/share things

Feel free to post questions here @KrishAryan about what help you'd need and I can drop pointers here for you. Also sent you an invite to last year's GH project to review some of that content

bwgartner commented 1 year ago

Hello @bwgartner I found this project really interests to me. The required skills completely match with my interest and tech stack I generally use. I have worked with kubernetes, Docker, Gitops, cicd, Python, ml, javascript and currently I am exploring DevOps into more depth and also integrating various tools.

Awesome, sounds like a great fit and collaboration, @bishal7679

I do really want and also interested to work on this project in Gsoc23 under your mentoring and guide.

I studied about Rancher, it is itself one of the leading multi-cloud container management platform. Currently I am trying to use rancher on one of my project and running it from docker with the bwlow command

docker run -d --restart=unless-stopped \
  -p 8080:80 -p 8443:443 \
  --privileged \
  rancher/rancher:latest

and its running on localhost:8443 which is good. But whenever I am going to import existing EKS cluster or creating ec2 cluster from rancher it gets stucked. you can see in below image. Here one EKS, one custom minikube and ec2 I am trying to import. Mar-03-12-01

So let's start with some questions here, about your deployment

bishal7679 commented 1 year ago

So let's start with some questions here, about your deployment

Here is the aws IAM instance profile name rules rancher-demo-ec2 rule 1

4

5 8 7

bishal7679 commented 1 year ago

Hello @bwgartner I have implemented one demo ML workload on healthcare business vertical using various algorithm features, and using Rancher to manage the cluster.

It would be great if you visit this link and provide your insightful feedback on whether my proposal fits to the problem proposed for the business vertical.

Here's the demo-project link: https://github.com/bishal7679/Rancher-MLalgo4Health

And also Here is my proposal which I've prepared. Could you please review it once and do any correction if its needed!

Here is the proposal link : https://docs.google.com/document/d/1Uhnmm4Iswhsp0fQJDiAWrQ3MTcbCAGzlAjptCcdozq4/edit?usp=sharing

Also suggest me if anything I should add. I would be really thankful to you!!

KrishAryan commented 1 year ago

@bishal7679 opensuse have biweekly meetings on Tuesday and Thursday. You can join and show your demo there too.

KrishAryan commented 1 year ago

https://news.opensuse.org/2023/02/22/awareness-sessions-about-projects-available-to-gsoc/

bishal7679 commented 1 year ago

@bishal7679 opensuse have biweekly meetings on Tuesday and Thursday. You can join and show your demo there too.

Thnx!

bwgartner commented 1 year ago

@bwgartner patiently waiting for your reply! Please provide any feedback

Hi @bishal7679 ... have reviewed your proposal (overall very good), so let me know how to send you that copy. Also your LinkedIn connection request didn't allow me to connect to you, so sent one from my side.

kparnami1 commented 1 year ago

Hello @bwgartner , I hope you are doing well! In regard to the above problem statement I have tried to implement a banking solution, expanded across multiple pods in the Kubernetes cluster, and can scale to a huge volume. The Kubernetes cluster is imported in rancher running on GCP VM. Below are the steps I have performed while building the demo -

Cluster Cluster_pods RancherInstance ApplicationCapture1 ApplicationCapture2

Currently, I have one issue with importing all clusters successfully to Rancher, I'm working on it. Below is the screenshot of the error.

RancherLandingPage

Please guide me if I'm going in the right direction. Thank you. Regards, Kunal Parnami.

bishal7679 commented 1 year ago

Please guide me if I'm going in the right direction. Thank you. Regards, Kunal Parnami. Hello @kparnami1

kparnami1 commented 1 year ago

Hey @bishal7679, Thanks for responding, I have used docker to install Rancher, and I have also discussed the above demo in Thursday's weekly call, as per feedback, we should ideally install Rancher in Kubernetes. I'm continuously researching this area, the above issue is mostly due to GKE security.

Regards, Kunal.

AkashKumar7902 commented 8 months ago

Hey @bwgartner

I am interested in working on this project. I have expertise in many programming languages like go, python and java. I have worked on various cloud native projects and have a deep understanding of kubernetes. I am currently an intern at CNCF landscape and GSOC project, keploy. I have also been selected in term 1 of the LFX mentorship program for working in kcl language package manager.

Is there any communication channel where I can discuss more about the proposal ?

Thanks Akash Kumar

navin772 commented 8 months ago

Communication channel for project- https://join.slack.com/t/opensuse-group/shared_invite/zt-2dwsqm6nb-iNJifnVbQ5zvzCYA_Q2FXQ

aakshitaa commented 7 months ago

Project Title: Create open source sample microservice workload deployments and interfaces, spanning distributed edge-core-cloud infrastructure to address market verticals to address analytics per market verticals

Description: In the context of this Analytics Edge Ecosystem, such a digital transformation approach to cloud-native containerizing addresses how one can help strategically iterate and perform the necessary and even more comprehensive business analytics requirements. This can be done across a truly distributed compute infrastructure (with various resource options), leveraging compute elements closer to where data is generated or provided initially. Then you can more rapidly adapt the reaction, formulate predictions and forecasts, and have a larger context of all the elements that can potentially impact your business positively and internally, plus for consumers, providers, and partners.

Deliverable: Provide an opensource-based example deployment of a trained/tested/functional workload available for use cases:

  • General

    • Interact in sprint meetings, sharing progress, issues, and plans (via lightning talk approach) and peer reviews of other contributors or examples
    • Provide content in a usable, accessible GitHub repo example
    • (Bonus Points) : publish blog/demo/documentation or webinar/conference sessions with an overview of why certain paths were chosen and how it was done

  • Phase I (Bonding Period)

    • Learn and interact plus become familiar with the overall application infrastructure layers (network + hardware + storage technology, operating system, containers, kubernetes, AI/ML examples)

  • Phase II (first coding)

    • Pick a target role plus an interesting business vertical, research an existing example and deploy to learn about what underlying aspects are needed and what to provide above it
    • Then either stay in same role, picking another vertical and retry or adopt another role to more completely adapt or provision it

  • Phase III (second/final coding)

    • Brainstorm and research an end-to-end example approach for a desired business vertical of your own personal interest
    • Potentially port aspects/portions of other found examples to further improve the functionality and address better analytics plus multiple user needs
    • Deploy in multiple environments to validate usability

Mentor: @bwgartner + more/others (Ann, Krish, Navin, Terry)

Skills / Layers / (Roles):

  • Application Infrastructure (ITOps, AIOps, Platform Engineering

    • Network/Computing/Storage Platforms

    • Operating System

    • Programming languages (your choice)

    • Bash, JavaScript, Java, Jupyter, Python, R, Ruby, ...

    • Build packages, virtual machines, container runtime and image creation, manifests/helm charts

    • Cloud-Native Kubernetes leveraging Rancher offerings

    • Data Pipeline (Data/Ops Analysis)

    • Ingest, cleanse, validate, stream, flow, curate, conform

    • Machine Learning Pipeline (MLOps)

    • Train, test, flow, evaluate, secure AI (Artificial Intelligence)/ML (Machine Learning)/DL (Deep Learning)/LLM (Large Language Models)

    • User, Consumer, Operator, Programmer Interfaces, (DevOps/GitOps)

  • Example business verticals

    • agriculuture/environmental/weather, automotive, healthcare, retail, federal/military, finance, manufacturing, media/entertainment, smart building/city/home/light/parking, telecommunications, ...

Skill Level: Easy, Medium

Get started:

hello @bwgartner I am looking forward to work with you on this project. I am a DevOps student and have decent knowledge about kubernetes helm linux docker and working with containers. I will be applying for the project, is there anything you would want me to know before I apply. Also can you tell me that this project lies under which category for project size- small / medium / large ? Hoping to work with you soon!

navin772 commented 7 months ago

@aakshitaa this is a large-sized project.

aakshitaa commented 7 months ago

Communication channel for project- https://join.slack.com/t/opensuse-group/shared_invite/zt-2dwsqm6nb-iNJifnVbQ5zvzCYA_Q2FXQ

Hello @navin772 , It says that this link is expired, can you please provide a new one. Also thanks for your previous reply. I have submitted my application for GSOC, looking forward to working with you all in a team!

ddemaio commented 7 months ago

Communication channel for project- https://join.slack.com/t/opensuse-group/shared_invite/zt-2dwsqm6nb-iNJifnVbQ5zvzCYA_Q2FXQ

Hello @navin772 , It says that this link is expired, can you please provide a new one. Also thanks for your previous reply. I have submitted my application for GSOC, looking forward to working with you all in a team!

This should work https://join.slack.com/t/opensuse-group/shared_invite/zt-2gjmiqtoy-7ZCnoTUuSK0waesomQ0EYA

aakshitaa commented 7 months ago

Communication channel for project- https://join.slack.com/t/opensuse-group/shared_invite/zt-2dwsqm6nb-iNJifnVbQ5zvzCYA_Q2FXQ

Hello @navin772 , It says that this link is expired, can you please provide a new one. Also thanks for your previous reply. I have submitted my application for GSOC, looking forward to working with you all in a team!

This should work https://join.slack.com/t/opensuse-group/shared_invite/zt-2gjmiqtoy-7ZCnoTUuSK0waesomQ0EYA

Yes joined. Thankyou 😊