search

newrelic-experimental / ml-performance-monitoring

A Python package for sending model inference data, data metrics, and model metrics
Apache License 2.0
33 stars 19 forks source link

Example for a deep learning model #55

Open JustasB opened 1 year ago

JustasB commented 1 year ago

Summary

Can you please provide an example of logging inference predictions for a deep learning model (e.g. image/audio classification).

Desired Behavior

NOTE: # Would like to see example of how non-tabular, high-dimensional inputs (e.g. image, audio, text) would be logged to NR

Possible Solution

NOTE: # If high-dimensional inputs are not supported, provide a way to send just predictions (e.g. just y_pred, without X)

Additional context

Currently, at least one feature is required to log with the library, when the inputs are high-dimensional (e.g. images, audio), sending them to NR would not be useful. Would like a way to sent just predictions without inputs.

ghost commented 1 year ago

Thanks for the feedback @JustasB. You are correct. We thought about sending the characteristics of the data instead of the data itself as X. We will also check the option of not sending X at all and will update you soon.

Thanks!