ML Model Dev Roadmap

[pomkos]

Goal

We want to predict which time segments in a stream has the potential to go viral from the chat.

Training Flow

• [x] Create a dataset of viral CCCs and their chat logs (done)
• [x] Create training feature set from those viral CCCs
  • [x] Create a new brain to coordinate feature extraction
  • [ ] Run each CCC chat log through brain to get features for each chat log
  • [x] Assign label of TRUE to timestamps that the CCC was in
  • [x] Assign label of FALSE for everything else in the same chat log
• [ ] Split up feature set into training/validation/testing
• [ ] Train predictor_model_training on the training data set split
• [ ] Test the results on the validation data set split until we get our desired results
• [ ] Validate the model on the testing set split

Output: we now have a validated and trained model! Yay!

Post-Training TODO:

• [ ] Create unit tests for each step in previous flow
• [ ] Document the pipeline
• [ ] Migrate to metaflow for reproducibility

Production Flow:

• [ ] Vod_poller and chat downloader lambdas create the chat log dataset
• [ ] Create a feature set by running the chat logs through the brain used in training pipeline.
• [ ] Run predictor_model to assign a binary true/false value (and a percentage of confidence) to each time segment.
• [ ] Create Clip Objects. Assign start/end times for all the time segments that have TRUE (optional: true && passes threshold of confidence)
  • [ ] TODO: Make the creation of clip ranges configurable + hard coded
  • [ ] TODO: Potentially explore writing another ML model that predicts start and end times for each clip

Model (Pre-Production) Deployment

• [ ] Set up CI/CD/CT for model training via metaflow
• [ ] Export model artifact and deploy on ML_Infra stack (SNS topic + lambda)
• [ ] Create usability testing + acceptance tests
• [ ] Get a baseline measurement of user engagement by interviewing moderators / streamers on how many clips they would use with our current algo1
• [ ] Deploy via Metaflow for easy rollbacks + versioning etc
• [ ] Track users engagement in Pillar app
  • [ ] Thumbs up / thumbs down ratings for clips, where exporting a clip == mega thumbs up.
  • [ ] Create a way to view the analytics + see the performance of the model

Maintenance + Monitoring

• [ ] Expose model performance in a dashboard (Hubspot? )
• [ ] Expose comparison of model performance to baseline user engagement
• [ ] Send alert if model performance creates lower user engagement versus the baseline

[pomkos]

@RusseII @chand1012 let us know what you think!

Steve and I came up with this plan today.

[pomkos]

OHIO GOZAIMUSU

Status

Currently we have all features/algos completed. What this means is that:

• Each algo was converted to a class called featureFinder. To run the algorithms you will have to run it as a class ff = algo1.featureFinder() and then run the run function new_feature = ff.run()
• There are no more default options @RusseII, this is to avoid bugs where the user expects one thing but the algo does something else

New Brain

Brain is being rewritten from the ground up in the new_brain branch. What this means is that:

• Brain will no longer find the common timestamps between algorithms, instead it is used to create a new feature dataset
• Brain will run each algorithm using appropriate parameters (need to run a grid search on what those parameters should be), then creates a new dataset using each algorithms output
• This new dataset will be used to train a supervised categorical machine learning algorithm to predict which timestamp is a CCC or not
• The trained algorithm will then be used in prod

Sample new dataset created with brain:

vid_id	Start	End	algo1 output	algo2 output	algo3_0 output	algo3_5 output	algo3_6 output	algo4 output	CCC	Genre	View Count
920260466	2021-02-19 19:54:30.649	2021-02-19 19:54:40.157	0.213	12.620951	120	210	0.885	0.123	False	MMO	69

New Feature Ideas

Our first attempt is not likely to be too accurate. This can be because of inaccurate features, so here are some potentially useful new ones.

1. Automated genre categories, probably from steam. Start with general like "strategy", but may have to be more specific like "Grand Strategy" or "Historical"
2. Labeling of each timestamp with a topic, perhaps using more advanced NLP, Brown Clustering, Butter
3. Interpretability, speed, F1 score as per this repo

To Do

Brain

The current to do list on brain:

Create Brain which will run all algorithms in order to give features to each desired timestamp

• [x] Each algo should have the same parameters in same order for simplicity 3_6
  • data: list
  • limit: int
  • sort_by: int
• [x] Replace sort_by with select. We no longer need sort_by parameter in any algo since the goal is to label each chunk, not to return top clips. We do need to select features of interest
• Algo3: gets the number of words/emoji/both sent by each chatter, compares to the most spammer chatter
  • [x] Find the chatter who sent the most number of words
  • [x] Return average weight of chatters per chunk
  • [x] Return average number of words/emoji/both sent per chatter per chunk
• Take out finalizer from each algo, it is now part of data_handler and used at brain level
  • [x] algo1
  • [x] algo2
  • [x] algo3
  • [x] algo3.5
  • [x] algo3.6
  • [x] algo4
• [x] Each algo should accept a list of chunks to analyze
  • [x] algo1
  • [x] algo2
  • [x] algo3
  • [x] algo3.5
  • [x] algo3.6
  • [x] algo4
• [x] Each algo should NOT create its own chunks
  • [x] algo1
  • [x] algo2
  • [x] algo3
  • [x] algo3.5
  • [x] algo3.6
  • [x] algo4
• [x] Add optional progress bars to each algo
• [x] Algo1 only returns 10 rows. Expected: 215
• [x] Double check that algo2 creates the desired output
• [x] Double check that algo3 creates the desired output
• [x] Algo3.5 does not return start/end timestamps with the feature
• [x] Update docstring of algo3_6
• [x] Algo4 returns NoneType
• [x] Brain needs to label each timestamp as True/False for CCC timestamp overlap
• [x] Rewrite brain to follow metaflow outline

Topic Selection

@gatesyp has too many ideas!!! These need to be investigated:

• [ ] Brown clustering link
• [ ] Butter tool link
• [ ] Section 2.2 of research article contains an overview of Twitch research link

ML Options

I know we said supervised categorization ML, but within that category the following might make sense:

1. SGDClassifier. Link <- our first attempt, super simple
2. DecisionTreeClassifier. Link <- second attempt maybe
3. Use Metaflow to select models

[pomkos]

Not sure how to receive data (line 102 of new_brain) @gatesyp

[pomkos]

Updated todo. We now can start coding brain to label each timestamp as overlapping CCC or not.

[pomkos]

Updated readme to describe how the flow works, how to call it, and the required/optional parameters for brain.

Current Status

• [x] Migrate brain to metaflow
• [x] Have cccLabeler() class accept dataframe instead of string
• [x] Update docstrings
• [x] Update README
• [ ] Set up deploy settings locally
• [ ] Test brain on AWS

To Do

• [ ] Create new metaflow for train/test/validating model
• [ ] Repeat with other models
• [ ] Deploy best model