Monitoring Machine Learning Models in Production

[yeomko22]

1. The ML system life cycle

• continuous delivery for machine learning (CD4ML)

1. Model building

   • understanding the problem
   • data preparation
   • feature engineering
   • initial code, rough jupyter notebook

2. Model Evaluation and Experimentation

   • feature selection
   • hyper paramter tuning
   • comparing the effectiveness of different algorithms on the given problem
   • notebooks with stats, graph evaluating feature weights, accuracy, precision, Receiver Operating Characteristics

3. Productionize Model

   • preparing it so it can be deployed
   • production grade code
   • different programming language and framework

4. Testing

   • ensuring that production code behaves in the way we expect it to
   • test cases

5. Deployment

   • API for accessing the model

6. Monitoring and Observability

   • ensure our model is doing what we expect it to in production.

[yeomko22]

2. What makes ML system monitoring hard

• ML system: data + model + code(and config)
• entanglement
  • when input feature is changed, then the importance, weights or the remaining features may all shift as well.
  • “changing anything changes everything” issue.
• configuration
  • Because model hyperparameters, versions and features are often controlled in the system config, the slightest error here can cause radically different system behavior that won't be picked up with traditional software tests.

[yeomko22]

3. Why you need monitoring

data skew

• Distribution of the variables in the training data do not match the distribution of the variables in the live data.
• A feature is not available in production
• Research/Live data mismatch
• Data dependency. 외부의 데이터 소스에서 데이터를 가져오는데, 이 외부 시스템이 변하는 경우

Model Staleness

• shifts in the environment.
• changes in consumer behavior
• adversarial scenarios

Negative Feedback Loops

• data가 corrupt 된 상태에서 자동으로 모델을 학습시킬 경우 상황은 더 복잡해진다.
• 배포된 모델 자체가 데이터에 영향을 줄 수 있다.

[yeomko22]

4. Key Principles for monitoring your ML system

• monitor 1: dependency changes result in a notification
• monitor 2: data invariants hold in training and serving inputs
• monitor 3: training and serving features compute the same values
• monitor 4: models are not too stale
• monitor 5: the model is numerically stable
• monitor 6: the model has not experienced dramatic or slow-leak regressions in training speed, serving latency, throughput, or RAM usage
• monitor 7: the model has not experienced a regression in prediction quality on served data

[yeomko22]

5. Understanding the spectrum of ML Risk Management

• testing/monitoring의 궁극적인 목표는 risk management이다.
• test와 monitoring은 함께 가야한다.
• data test, skew test, ML infrastructure test, model test, unit test, integration test 등을 거친다.
• running system에서는 data monitoring, prediction monitoring, system monitoring이 필요하다.

[yeomko22]

6. Data Science Monitoring

• 프러덕션 환경에서 모델의 정확도를 즉각적으로 알 수 있는 방법이 없다. 왜냐? 모델의 예측이 정확한지 알기 위해서는 시간이 필요하기 때문.
• 그래서 모델의 예측이 정확한지를 모니터링하기 위한 보조 지표(proxy values)을 측정한다.
  • model prediction distribution (regression) or frequencies (classification)
  • model input distribution (numerical features) or frequencies (categorical features)
  • model versions

model input monitoring

• input 값이 허용된 범위 안으로 들어오는지 체크

model prediction monitoring

• 기초 통계 지표 측정 (median, mean, standard deviation, max/min values)
• training과 live 사이에 차이가 있는지를 보아라. 만일 다르다면 어떤 문제가 있는 것이다.

advanced techniques

• t-test
• ANOVA
• 비모수테스트 like Kruskal Wallis, Kolmogorov Smirnov
• 구현은 가능하지만 매우 어렵다. 그래서 보통 기본 통계값만 추출하고 ad-hoc manual testing을 한다.

[yeomko22]

7. Operations Monitoring Concerns

Operational concern around ML system

• System performance (latency, IO/ Memory / Disk Utilization)
• System Reliability (Uptime)
• Auditability (though this applies also to our model)

Three pilars of observability

• metrics
• logs
• distributed traces
• profiling (optional)
• alerting/visualization (optional)

Monitoring vs Observability

• observability is a supurset of monitoring and testing

[yeomko22]

8. Bringing Ops & DS Together - Metrics with Prometheus & Grafana

Pros

• low overhead than log saving
• longer retention of data, easier querying. well suited to craete dashboard
• ideal for stastical transformations

Cons

• cardinality issue, timeseries DB에 부담을 줄 수 있음
• scoped to one system

Metrics for Machine Learning

• latency when calling ML API endpoints
• memory/CPU usage when performing prediction
• disk utilization
• median & mean prediction values over a given timeframe
• min/max prediction values
• standard deviation over a given timeframe

[yeomko22]

9. Bringing Ops & DS Together - Logs

Pros

• easy to generate
• detail context를 제공, 이는 평균 같은 값이 알려주지 않음!

Cons