allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.45k stars 644 forks source link

Task logger flattens pandas dataframes pivoted in the WebUI #792

Open H4dr1en opened 1 year ago

H4dr1en commented 1 year ago

Describe the bug

Task logger flattens pandas dataframes pivoted in the WebUI

To reproduce

from clearml import Task
import pandas as pd
df = pd.DataFrame({'foo': ['one', 'one', 'one', 'two', 'two',
                           'two'],
                   'bar': ['A', 'B', 'C', 'A', 'B', 'C'],
                   'baz': [1, 2, 3, 4, 5, 6],
                   'zoo': ['x', 'y', 'z', 'q', 'w', 't']})
df_pivot = df.pivot(index='foo', columns='bar', values=['baz', 'zoo'])
task.get_logger().report_table("df_pivot", "df_pivot", 0, df_pivot)

Expected behaviour

The table is logged with the same multi index (columns) layers as the str representation:

>>> df_pivot
    baz       zoo      
bar   A  B  C   A  B  C
foo                    
one   1  2  3   x  y  z
two   4  5  6   q  w  t

But in the WebUI, the table has all the columns flattened:

newplot

Environment

erezalg commented 1 year ago

Thanks @H4dr1en, we'll let you know once this is solved