search

google / nitroml

NitroML is a modular, portable, and scalable model-quality benchmarking framework for Machine Learning and Automated Machine Learning (AutoML) pipelines.
Apache License 2.0
41 stars 6 forks source link

How to use it? #50

Open pplonski opened 3 years ago

pplonski commented 3 years ago

How can I use this benchmark suite? I'm working on AutoML https://github.com/mljar/mljar-supervised and would love to have a quick way to run tests.

Does it integrate with GitHub Actions?

cweill commented 3 years ago

Note: We're still in the pre-release stage, so caveat emptor. :)

@pplonski: Thanks for reaching out. If you want to try running our benchmark suite, follow the instructions at https://github.com/google/nitroml/tree/master/examples. This suite is designed to be run on GCP using KubeFlow.

Integration is straightforward if you are already using TensorFlow, i.e. accepting TensorFlow Examples as inputs, and exporting a TensorFlow SavedModel. Does MLJar support that? If so I can point you in the right direction. Otherwise, we are planning to make this this framework more ML framework agnostic in Q4 2020, i.e. accepting raw CSV data as inputs, and exporting numpy predictions.

cweill commented 3 years ago

@pplonski: You mentioned integrating with GitHub actions which sounds like a good idea: could you explain what you're envisioning?