Conference Notes

[banditelol]

Conference Notes

This Issue will contains all the notes I took and additional interesting ideas I found while watching conferences either in person or online on Youtube

[banditelol]

Makefiles: One Great Trick for Making Your Conda Environments More Managable | PyData Global 2021

Pydata is one of the better source of videos out there, while I still need to curate the content from time to time, but this is one of the best and practical one that I found. Also this pushed me to add Kjell Wooding to list of followed people.

In this talk he basically guide us step-by-step on the points that makes make a good tool for data science workflow and how to implement them. In total there are 7 points:

1. Use git to version your project
2. Use virtual environment when using python
3. Use your git repo name for your env name
4. Check in with your virtual environment. In this case you need environment.yaml for your conda stuffs
5. Use make for your environment management, because face it you don't remember the snippet to create new environment from environment.yaml don't you?
6. Never install package manually e.g. edit environment.yaml + make update_env or in poetry case you can use poetry add to do similar stuff. The neat thing is both can be abstracted behind makefile
7. Use Auto documentation, either in the makefile with adding comment before the command, or in-script by using docstring. Keep the docs near your code automatically, and only manually write down docs that has long shelf life
8. Separate what you want from what you need. This is the basic of lockfiles, you still write what you want in human readable interface (environment.yaml, requirements.txt, etc) but you need additional file for the actual installed dependencies (env.lock.yaml, req-v1-2.lock.yaml, etc)
9. If you implement all of those, don't be afraid to nuke it all from the orbit if you messed up a lot of things.

This talk also linked with Love Your (Data Scientist) Neighbour - Amy Wooding | PyData Global 2021, which defines it with 6 stages of reproducibility issues:

1. Where do I start: README and LICENSE
2. Where do I go next: organization + workflow via make
3. What to install: create_env and env.yaml
4. Where do i find the data: Dataset recipe ?
5. Does it work as expected: make test
6. Whats the purpose: use src module by editable install accompanying repo can be found here

[banditelol]

Kei Nemoto- Gentle introduction to scaling up ML service with Kubernetes + Mlflow | PyData NYC 2022

I've been wanting to implement MLFlow for managing my ML services. But I haven't gotten the cognitive bandwidth yet to do it for real. So for now I'll go on to my consumption mode and enjoy the talk. Anyway I'll try to go along with the hands on if possible.

Kei Nemoto (github.com/box-key) is DS in Montefiore Einstein Center for health. And the code for this talk is in box-key/pydata-kubernetes-mlflow

1. e.g. we create a ML service with stack of SKlearn-Flask-Gunicorn-Docker. In this case we have single container deployment "pattern".
2. What's the problem?
   1. Host downtime
   2. Update downtime
   3. Resources Limitation (service is limited by host resource)
3. Kubernetes supposedly solve these 3 problems. And watch kubernetes documentary by honeypot.

   What's additional problems introduced by this solution?

4. Kube: Control Plane + Workers. Kubelet is

   What is workers vs node? And What is kubelet?

5. Kubernets solve this problem by using deployments.
   1. Host downtime doesn't matter as long at least one worker is active
   2. Rolling update is built-in into the kubernetes
   3. Scalability is "easy" as we can just replicate as much as load
6. Create deployment is done by using kubectl apply and a deployment will generate several pods. So this assume Nodes and Control plane is already set up?
7. Cluster IP (the service) Service (resource kind, on the level of Deployment, Job, etc.) is load balancer which decide which pods will get the request.

   But how much of a bottleneck is this service? and how to scale this s rvice? Also is this only available inside Kubernetes

8. There's also NodePort Service. This can be done by exposing specific port. And as the request hit the nodeport it'll be redirected to cluster IP.
9. Load Balancer Service is an external LB for the nodes (as opposed to ClusterIP role as LB for the pods).
10. Should we run 4 pods? Aren't we wasting resources? Knative! a serverless service. Wait, so is it a replacement for LB?
11. Now that we've covered the service itself, how do we handle the model binary? We could build image for each binary model. But it's hard to track the lineage of each model. So MLFlow Model Registry!
12. Data Scientist -> Model Repostory -> ML Eng -> Embed Connection Link to Image (URI) as env variable -> Build the image

    

13. Model version is several version of the same model.

    How should we define what is "the same model"

14. K3S (lite) in Digital ocean native VM using 3 nodes (2 workers (1GB-1CPU) and 1 Control Plane (2GB-1CPU). Running Knative. Image Repos DockerHub, MLFlow (1GB-1CPU) for registry + server.
15. QnA
    1. Challenges in Realistic Scenario: Hard to setup K3S and faced a lot of network issues. Also it's possible you have multiple VMs in one rack and all hell break loose
    2. KNative practicality for latency : warm start vs cold start, how about the downtime? For big model it can take a long time. And you can also have 1 pods on standby.
    3. Production Distribution: We have to on prem kubes cluster for health domain.
    4. Load Balancer Ips/Subnet: ClusterIP can have a CIDR. Kubernetes doesn't setup subnet, it uses existing subnet.
    5. MLFlow setup multitenancy: Not yet.
    6. Kubernetes vs GCloud : not apple to apple, but compared to Mezos, Yarn, swarm its known to be more stable.

Welp. It's a lot more Kubernetes related than I expected. I thought it'll be more of a talk about practical MLflow-Kubernetes hands-on. Anyway the question on CIDR and how kubernetes network work lead me to this AWS page about VPC. It' worth a read to better understand how the IP Addressing component interact.