search

allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.58k stars 645 forks source link

powerbi integration #281

Open oak-tree opened 3 years ago

oak-tree commented 3 years ago

Hey,

We are thinking about integrate trains-clearml selected data/chart into a report in Powerbi. Any idea what would be the best approach to do so ?

Thanks!

bmartinn commented 3 years ago

Hi @oak-tree, It sounds very inserting ! Please tell me more :)

Are we talking "exporting" metrics from clearml into powerbi? Or is is the opposite ? How would the imported/exported data will be used ? Is it a one time thing or a running "cron alike" job? Basically any background on the use case will be helpful.

oak-tree commented 3 years ago

Hey @bmartinn 2 different goals of integrations between powerbig and clearml:

bmartinn commented 3 years ago

@oak-tree nice!

  1. What's the powerbi format / API for importing data ? (I'm assuming it will not be able to directly send RestAPI requests to query the cleaml-server). What I have in mind is a service running in the background, querying the clearml-server then sending the data to the powerbi with some RestAPI calls. This service should not be complicated to implement with clearml interface, probably with direct APIClient calls like in this Slack example based on the Monitoring class
  2. I guess this one is another flavor of the suggested service of (1) exporting different data into the powerbi system.

What do you think?

oak-tree commented 3 years ago

Hey, The slack example is cool! Do you have a telegram example as well ?

  1. It makes sense to have a middleware in the background for fetching data. I guess this is the options for us
  2. For the 2nd issue, I thought more like analytics of the usages of clearml. Maybe there is a built in already? tracking how many experiments, project and etc.. for each user?
bmartinn commented 3 years ago

The slack example is cool! Do you have a telegram example as well ?

No currently we do not, but it will be great to add. Do they have a python package? How easy is it to integrate with ?

For the 2nd issue, I thought more like analytics of the usages of clearml. Maybe there is a built in already? tracking how many experiments, project and etc.. for each user?

Sure you can basically query everything in the system through the RestAPI, or the python interface to the RestAPI RestAPI documentation available here

To access the API from a nice pythonic interface:

from clearml.backend_api.session.client import APIClient
client = APIClient()
client.tasks.get_all(...)

You can use the cleanup service as a simple APIClient usage example.

What queries did you have in mind?

oak-tree commented 3 years ago

As for telegram seems like it has support in many different languages. See here for more details: https://core.telegram.org/bots/samples

We will give the RestAPI test to see if it fits to our needs.

For the report service the aim is to mainly get data, like graphs and others metrics, from specific experiments based on name/tag or other identifier.

For the analytics, it can be interesting to understand how many experiments we are doing in a week/month/qouter/year. To I think it can help us to understand if a new/different MLops scheme that we add is making any difference. Moreover, I think it can reveal bottleneck in our research pipelines.

Anyideas on this? maybe you already collect/ analyze clearml usages?

bmartinn commented 3 years ago

For the report service the aim is to mainly get data, like graphs and others metrics, from specific experiments based on name/tag or other identifier.

This is fully supported,

from clearml import Task
task_scalars_dict = Task.get_task(project_name='project', task_name='exp1').get_reported_scalars(max_samples=10000, x_axis='iter')

For the analytics, it can be interesting to understand how many experiments we are doing in a week/month/qouter/year...

I agree that would be a very cool dashboard to add. The API already supports these queries. I'm thinking of a service App that updates a dash dashboard with all these stats, maybe something similar to this one (code) It might be simpler to build with bokeh , I'm actually not sure ... Anyhow currently there is no dashboard example like that, but feel free to PR if you have a working draft :)