search

kubeflow / website

Kubeflow Website
https://www.kubeflow.org
Creative Commons Attribution 4.0 International
149 stars 773 forks source link

Really need a Kubeflow Ecosystem Section to the Website, (Note list is attached I found who are in the Ecosystem) #2024

Closed gstoner closed 3 years ago

gstoner commented 4 years ago

I was looking at Kubeflow to understand who working with it yesterday. I thought it might be helpful for others as well who want to look at what solutions are available in the market. Realy it would be good to have a section on the website which looks at the Kuebflow ecosystem. It also helps show momentum

I did not add it bellow but also track VAR, System integrator Consutlanting Firms as well which support it. Make it easy for people register their solution with the Kubeflow community.

Here is list I pulled toghter for who I found in ecosystem so far

External Tool and Framework which integrate with Kubeflow

Feature Store • Hopsworks Feature store https://www.logicalclocks.com/event/hopsworks-feature-store-for-kubeflow-and-on-premises-clusters-hadoop • Feast - in development

Model management • ModelDB https://github.com/VertaAI/modeldb • Pachyderm https://www.pachyderm.com • DVC https://dvc.org via - Valohai https://valohai.com

Data management • Pachyderm https://www.pachyderm.com • Arrikto ROK https://www.arrikto.com/kubeflow/ https://www.arrikto.com/overview/ • Netapp https://blog.netapp.com/ai-control-plane

Metadata management • MLMD https://github.com/google/ml-metadata/blob/master/g3doc/get_started.md

Training Run Orchestration • Polyaxon https://polyaxon.com/integrations/kubeflow/ https://mc.ai/polyaxon-v0-4-2-kubeflow-horovod-and-mpi-integrations/

Schedulers Co-Scheduling and Improved Queuing • Volcano https://volcano.sh CNCF project for Machine Learning and Deep Learning Scheduler • Cloudera/Apache Yunikorn https://yunikorn.apache.org

Data Science Experience Optimization for Kubefow Pipelines • Kubeflow Automated pipelines Engine https://github.com/kubeflow-kale/kale • Ciao https://github.com/caicloud/ciao Tools • H20 https://github.com/h2oai/h2o-kubeflow • Kubeflow tools https://github.com/canonical-labs/kubeflow-tools • MAPR https://mapr.com/blog/kubernetized-machine-learning-and-ai-using-kubeflow/ • Valohai https://valohai.com/product/ Machine orchestration, version control, and pipeline management for machine

Learning • Hydrosphere https://hydrosphere.io Serving, Monitoring And Interpreting Your Machine Learning Flow • D2IQ Kudo for Ksphere https://d2iq.com/solutions/ksphere/kudo-kubeflow end-to-end machine learning pipeline • Agile Stacks https://www.agilestacks.com/solutions/machine-learning • Neural Network Interface https://nni.readthedocs.io/en/latest/index.html AutoML, automate Feature Engineering, Neural

Architecture Search, Hyperparameter Tuning, and Model Compression • MiniKF https://www.arrikto.com/minikf/ • Iguazio Platform https://www.iguazio.com Data science Platform that integrates with Kubeflow • Iguazio MLRun https://www.iguazio.com/open-source/mlrun/ AutoML, Pipeline Orchestration, Feature Store, Workflow with Kubeflows. • Iguazio Nuclio https://www.iguazio.com/open-source/nuclio/ Serverless Pipeline Automation and Model Serving • Docker "Depend on Docker for Kubeflow" https://www.docker.com/blog/depend-on-docker-for-kubeflow/ • Calicloud https://intl.caicloud.io/products/clever • One Convergence DKube https://www.oneconvergence.com/dkube/ • Datawire https://www.datawire.io • Alegro Ai Trains https://github.com/allegroai/trains-agent https://allegro.ai/enterprise/

OEM's • Dell EMC + Comet ML https://www.dellemc.com/resources/en-us/asset/white-papers/products/ready-solutions/dell-comet-machine-learning-wp.pdf. o Backed by Canonical’s Charmed Kubernetes and Kubeflow o Install Instructions https://infohub.delltechnologies.com/l/dell-and-comet-experiment-management-for-machine-learning-projects/technical-installation-specifications o Dell white paper https://www.dellemc.com/resources/en-us/asset/briefs-handouts/products/ready-solutions/dell_kubeflow_solution_brief.pdf • IBM https://developer.ibm.com/blogs/kubeflow-and-ibm-an-open-source-journey-to-10/ • Cisco Kubeflow Starter pack https://github.com/CiscoAI/cisco-kubeflow-starter-pack • HPE https://developer.hpe.com/blog/hpe-brings-a-little-magic-to-kubecon-cloudnativecon-europe o Topology Aware Scheduling using Prometheus and Telemetry Aware Scheduler o Taming Data/State Challenges for ML Applications and Kubeflow

Database • MySQL https://www.mysql.com • Minio https://docs.minio.io

Model Serving • Seldon https://www.seldon.io

Service Fabrics • Itsio https://www.kubeflow.org/docs/other-guides/istio-in-kubeflow/ • Redhat OpenShift Fabric https://developers.redhat.com/blog/2020/06/26/migrating-a-namespace-scoped-operator-to-a-cluster-scoped-operator/ Kubeflow Kubernetes Stack Integration • Redhat Openshift https://www.businesswire.com/news/home/20200324005432/en/Red-Hat-Accelerates-AIML-Workflows-Delivery-AI-Powered • Rancher https://rancher.com/blog/2020/introduction-to-machine-learning-pipeline • Ubuntu MicroK8S https://microk8s.io • Ubuntu Kubeflow https://ubuntu.com/kubeflow with Ubuntu Kubernetes • VMware vMLP https://flings.vmware.com/vmware-machine-learning-platform • BanziCloud https://banzaicloud.com/blog/kubeflow-deployment/ • Nvidia DeepOps https://github.com/NVIDIA/deepops • SuSE https://suse.com/c/kubeflow-data-science-on-steroids/ • Ubuntu JUJU charms https://jaas.ai/kubeflow/bundle/29

Cloud Version of Kubeflow • GCP o Codelab Kubeflow https://codelabs.developers.google.com/codelabs/kubeflow-introduction/index.html?index=..%2F..index#0 o Spotify Terraform ( Kubeflow Soluton) runs on GKE terraform-gke-kubeflow-cluster https://github.com/spotify/terraform-gke-kubeflow-cluster • Azure o Azure Kubeflow Labs https://github.com/Azure/kubeflow-labs o Azure Deploy Tool https://github.com/microsoft/DeployDLKubeflowAKS • AWS o Kubeflow Service - https://aws.amazon.com/blogs/opensource/kubeflow-amazon-eks/ o EKS Workshop https://github.com/aws-samples/eks-kubeflow-workshop o Sagemaker componets for Kubeflow https://aws.amazon.com/blogs/machine-learning/introducing-amazon-sagemaker-components-for-kubeflow-pipelines/ • IBM Cloud o https://developer.ibm.com/technologies/artificial-intelligence/tutorials/train-and-serve-a-machine-learning-model-using-kubeflow-in-ibm-cloud/ • Alibaba Cloud o https://github.com/AliyunContainerService/kubeflow-aliyun o https://www.alibabacloud.com/blog/build-a-machine-learning-system-using-kubernetes_595961 • Paterson Consulting http://www.pattersonconsultingtn.com/offerings/managed_kubeflow.html

Kubeflow YouTube Videos • Kubeflow: Machine Learning on Kubernetes https://www.youtube.com/watch?v=HBxyLnEzyhw • Kubeflow Pipelines https://www.youtube.com/watch?v=TZ1lGrJLEZ0 • Kubeflow 101 Videos by Google o Introduction to Kubeflow https://www.youtube.com/watch?v=cTZArDgbIWw o Getting Started https://www.youtube.com/watch?v=SRq21GRT31g&t=98s o Kubeflow Componets https://www.youtube.com/watch?v=dC659IsHNyg o Introduction to Pipelines https://www.youtube.com/watch?v=_AY8mmbR1o4&t=87s o Jupyter Notebooks https://www.youtube.com/watch?v=eEsfPL6SvJc o Hyperparameter Tuning with Katib https://www.youtube.com/watch?v=nIKVlosDvrc • Machine Learning using Kubeflow and Kubernetes - AWS - Arun Gupta https://www.youtube.com/watch?v=t5zJm1D_a04&t=2s • Null Channel: Basic Kubeflow and Kubernetes Install https://www.youtube.com/watch?v=8BSbpJc8FTw&t=80s • Understanding the Earth: ML With Kubeflow Pipelines https://www.youtube.com/watch?v=DjP24I_WKWw • Building a Medical AI with Kubernetes and Kubeflow - Jeremie Vallee, Babylon Health https://www.youtube.com/watch?v=nD_GXaW_A-s&list=PLj6h78yzYM2NDs-iu8WU5fMxINxHXlien&index=253&t=0s • Advanced Model Inferencing Leveraging KNative, Istio & Kubeflow Serving - Animesh Singh & Clive Cox https://www.youtube.com/watch?v=YaGASyU88dQ&list=PLj6h78yzYM2NDs-iu8WU5fMxINxHXlien&index=186&t=0s • Supercharge Kubeflow Performance on GPU Clusters - Meenakshi Kaushik & Neelima Mukiri, Cisco https://www.youtube.com/watch?v=UYhBaXzD0hA&list=PLj6h78yzYM2NDs-iu8WU5fMxINxHXlien&index=255&t=0s • Measuring and Optimizing Kubeflow Clusters at Lyft - Konstantin Gizdarski & Richard Liu https://www.youtube.com/watch?v=IKubDSIivR8&list=PL1gDNeYq1To-VSpPOGk7dYTvqXsQ6snwo&index=46&t=0s • Panel: Enterprise-grade, On-prem Kubeflow in the Financial Sector https://www.youtube.com/watch?v=SmrW_RyQM1c&list=PLj6h78yzYM2NDs-iu8WU5fMxINxHXlien&index=201&t=0s • Scaling Machine Learning on Kubernetes and Kubeflow with SageMaker - AWS Online Tech Talks https://www.youtube.com/watch?v=LKmkiUdhV58 • Kubeflow Overview by Redhat https://developers.redhat.com/devnation/tech-talks/kubeflow-kubernetes/ • PyTorch + TensorFlow + KubeFlow + RedisAI + Streams - Advanced Spark and TensorFlow Pipeline AI https://www.youtube.com/watch?v=6RYdyU71Fl8

Kubeflow Presentations • https://www.oliverwyman.com/content/dam/oliver-wyman/v2/events/2018/March/Google_London_Event/Public%20Introduction%20to%20Kubeflow.pdf • Kubeflow in Financial Sector https://static.sched.com/hosted_files/kccncna19/76/KubeCon%20-%20US%20Bank%20%26%20JPMorgan%20%284%29.pdf Youtube talk on managing GPU resource in K8 • Lightning Talk: Sharing a GPU Among Multiple Containers - Patrick McQuighan, Algorithmia

Books • Oreilly Books "Kubeflow Operations Guide" https://www.oreilly.com/library/view/kubeflow-operations-guide/9781492053262/ • Oreilly Books "Kubeflow for Machine Learning: From Lab to Production" https://www.amazon.com/Kubeflow-Machine-Learning-Lab-Production/dp/1492050121

Kubeflow Training Programs • Noble Programing Kubeflow Training Courses https://www.nobleprog.com/cc/kubeflow?participants=1&how=private • Katacoda https://www.katacoda.com/kubeflow • Pluralsight's https://www.pluralsight.com/courses/building-end-to-end-machine-learning-workflows-kubeflow • Learnk8s https://learnk8s.io Kubeflow Classes • ePlus https://www.eplus.com/docs/default-source/training-documents/kubeflow-end-user-training.pdf?sfvrsn=19bd46b1_2

Kubeflow Consulting Team • Growing Data https://growingdata.com.au/mlops-ci-cd-for-machine-learning-pipelines-model-deployment-with-kubeflow/ • Ubuntu https://ubuntu.com/kubeflow/consulting • Paterson Consulting http://www.pattersonconsultingtn.com/blog/trends_2019_kubeflow.html • Noble Prog https://www.nobleprog.com/consulting/kubeflow • Maven Code https://www.mavencode.com • Cloud Ops https://www.cloudops.com/2018/09/machine-learning-tensorflow-kubernetes-and-kubeflow/ • Pipeline AI https://pipeline.ai

Kubeflow Hiring Activity • https://www.indeed.com/q-Kubeflow-jobs.html • https://www.monster.com/jobs/search/?q=Kubeflow&where= • https://www.linkedin.com/search/results/all/?keywords=kubeflow&origin=GLOBAL_SEARCH_HEADER

issue-label-bot[bot] commented 4 years ago

Issue-Label Bot is automatically applying the labels:

Label Probability
area/docs 1.00
kind/feature 0.88

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! Links: app homepage, dashboard and code for this bot.

jlewi commented 4 years ago

This a great list; any interest in opening up a PR to add it to the website?

/cc @8bitmp3

g-angstrom commented 4 years ago

I see the website was developed in Hugo, I need to look at the website to see where it best fits in before I would be comftable with a PR.

Update List I found a few more new items- Note the list below is now markdown text so it can be lifted into a webpage

External Tool and Framework which integrate with Kubeflow

FeatureStore

Editor Integration

Jupyter NoteBook Extensions

Model management

Data management

Metadata management

Training Run Orchestration

Schedulers

Co-Scheduling and Improved Queuing

Data Science Experience Optimization for Kubefow Pipelines

Tools

OEM's

Database

Model Serving

Service Fabrics

Hyperparameter Services integrated into Katlib

Kubeflow Kubernetes Stack Integration

Cloud Version of Kubeflow

Kubeflow YouTube Videos

Kubeflow Presentations

Youtube talk on managing GPU resource in K8

Books

Kubeflow Training Programs

Kubeflow Consulting Team

Kubeflow Hiring Activity

8bitmp3 commented 4 years ago

@gstoner @g-angstrom @jlewi I am with you. I suggest we make this explicit, so it's easily identifiable on the site. The less steps someone has to take by browsing through "nested folders" the better, IMHO. Some of the best open source projects I've been using have relatively simple websites/docs with intuitive navigation.

Because the word "Ecosystem" may be too broad (it's a good word though), I propose to add to separate this extensive list into two sections:

See my bad attempt at graphics design below: image

Regarding the Kubeflow hiring section, I may be wrong but I think "experience with deployment of pipelines" (and/or other combinations of words can hide the fact that an employer is potentially looking for Kubeflow and Kubernetes experience. What we can do is ask the community to post jobs in GitHub Discussions instead? @jlewi

I can start with a PR with a couple of Markdown files and let you co-edit them, so you get the credit @gstoner @g-angstrom. Or you can begin the work and I can review/edit it.

g-angstrom commented 4 years ago

That sound good to me, I edit with my gstoner github profile, g-angstrom is for my job. I will say I really like the project,

Here some recent new on Kubeflow as well Some recent market activities AWS • AWS SageMaker recently announced it integration into Kubeflow https://aws.amazon.com/blogs/machine-learning/introducing-amazon-sagemaker-components-for-kubeflow-pipelines/
D2IQ ( Formally Mesoshphere ) • D2iQ. Has new product called new called Konvoy launched announced back in March 2020. On announced 26th of June they announced they are rolling out their own Kubeflow solution called Kudo. https://devclass.com/2020/06/26/kudos-for-kubeflow-d2iq-preps-machine-learning-platform-for-enterprise/ IBM · June 26th 2020 IBM offers open source toolkit for Covid-19 data analysis. The data analysis steps and data processing pipelines using Elyra Notebook Pipelines Visual Editor and KubeFlow Pipelines. https://developer.ibm.com/blogs/open-source-jupyter-notebooks-analyze-covid-19-data/ Dell · On March 23rd 2020, Dell Launched Reference Architecture for Kubeflow using Charmed Kubernetes and Dell EMC Infrastructure https://www.delltechnologies.com/content/dam/digitalassets/active/en/unauth/white-papers/products/ready-solutions/dell-canonical-kubeflow-charmed-kubernetes-ra.pdf · March 30th 2020 Dell and Comet.ai announce Reference Architecture on K8 + Kubeflow https://towardsdatascience.com/dell-emc-and-comet-release-kubernetes-reference-architecture-1706c1575530

jbottum commented 4 years ago

@g-angstrom thanks for collecting and organizing this material, very impressive. I think you are moving in the right direction and that we should organize some folks to help. Maybe we can ask for support at the Kubeflow Community meeting ? In addition, we have a Kubeflow Product Management meeting on Tuesdays at 10am PST, which might be a good place to discuss (in length). A few things that we probably need to consider - what are the section names, what order are those listed in, and what is the order of the items in each section (i.e. some like videos or blog posts might be by date, whereas others like a list of companies might be by alpha).

g-angstrom commented 4 years ago

yes, that was raw data list and aware of it, It was a bit of work finding it all as you can imagine.

Yes it now needs descriptors, dates, metadata, and tags

On Categories here is starting list you can work with

I looked at the current Video list on the Kubeflow site, it really needs a bit more work to break them out into categories as well.

jbottum commented 4 years ago

@g-angstrom Hey Gregory - I propose that we set-up some time to discuss. Are you open tomorrow (Friday) for a call ? Josh, josh.bottum@arrikto.com

g-angstrom commented 4 years ago

@jbottum set you an email on this

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

oluies commented 3 years ago

Is this online somewhere now?