Revamp Demo

[slopp]

Best to review the first 3 commits, the rest are fixing nits.

• [x] Added an EDA notebook to reflect the beginning of the data science journey, the notebook uses tidymodels to create a simple first xgboost model. This EDA notebook is not setup to be Git backed.

• [x] Added a RMD that generates a scheduled email to alert on bike availability. Station is selected as a parameter. Predictions use the deployed model API. Email sends a plot and some prose along with a custom subject line. Setup for Git deploys.

• [x] Greatly modified the model creation and scoring pipeline. The pipeline now has only 3 steps: raw intake -> clean -> model scoring. Model training is handled on an ad-hoc basis. Only 2 pins are created, the model pin and the stations pin. Scoring always takes the latest model and rescores the entire table. Ingest continues to append new records. The new content still uses the helper package but to a far lesser degree in an attempt to keep the main RMDs readable without referencing the package (essentially only the feeds and oos_metrics functions are used). No changes to the package were made, but a majority of the code in the package is now dead code.

There is a bit of a chicken and egg situation here... I am hoping we can merge everything, have RSC pick up the updates, and then be off to the races. But I suspect once everything deploys we'll need a few more tweaks to be sure it all hangs together. I also need to update the readme and solutions page to reflect the new state.

I have turned off the main model update and model metric RMDs for now (both were broken anyhow).The staged versions are running, as are the original data ingest reports. The original model monitoring app is broken, but the new one is staged.

I have also modified the vanity URL so that the main client app points to the staged version of the plumber API. This allows the main client app to work, as the original plumber API is currently broken (due to changes in xgboost package versions).

Staged versions:

• Model Scoring Report, scheduled daily and updates DB
• Model Train Report, scheduled monthly and updates pin
• Model Monitor App
• Plumber API , currently using the vanity URL associated with the client app

TODO after approval:

• [x] Merge and wait for RSC to pull the master of everything
• [x] Swap the vanity URL back to the "master" API
• [x] Turn off the "staging" schedules and turn back on the "master" schedules
• [ ] Deploy and schedule a "master" version of the email report, and deprecate the push button published version
• [x] Update content names if we want
• [ ] Update solutions article about bikes demo
• [ ] Experiment with pointblank ?

[edavidaja]

The EDA notebook doesn't run all the way through in my dev environment--it might be helpful if there was a renv lockfile in the repo.

[slopp]

Thank you for the review @akgold! It sounds like the biggest question is on model training, which is really interesting.

In my experiences interviewing customers with Julia, it appears the most common pattern is for organizations to monitor model metrics and if they slide then go through a process of re-building the model. I haven't heard of any organizations that actively re-train the model weights every day, have you?

I think there are benefits to separating model training and model scoring.

I did leave the model training RMD scheduled to run on a monthly basis. I think you could make a hand waving argument that this frequency is a compromise that avoids model drift while also avoiding the pitfalls of "real time training". But I'd love to get your take - is real time training critical to the demo you had envisioned?

[akgold]

Definitely not critical, and I don't think I've heard of customers doing automated re-training. I think doing model scoring daily meets the need of "automated model production things happening".

Some Thoughts That Are Not Blockers for Merging:

Thought this raised -- what if there were another scheduled RMD email that alerts on some measure of model drift? Since we're re-assessing model quality daily but maybe not retraining, an automated alert if model drift is happening seems really useful; that is something that's come up with customers.

The existing RMD email is directed at the business user, while this one would be directed at the publisher. Worth having both kinds in here?

I'd also like to think more about how we onboard people to this material. Now that there's something here that I didn't create, I'm feeling the pain of not being 100% comfortable with the material.

[slopp]

Love it! I think pointblank table + email could be a really powerful combo for model monitoring...

Perhaps on the knowledge share meeting we can see if any SE would like to take that project. I'd personally enjoy it, but it seems like a fun excuse for an SE to write some code, learn the demo (like I have through this process), and get acquainted with Rich!

Re: onboarding ... do you mean introducing an SE to all the moving parts as a developer? Or introducing the assets? I don't have many ideas besides doing more work on the README...

I'll work on getting this all merged now 🤞