[App]: Unable to create custom chart using summary key that doesn't exist for all runs

[eito-fis]

Current Behavior

Including a summary key in the query that isn't shared by all runs grays out the Ok button and makes it so I am unable to create a custom chart. Filtering the query so that only runs have logged using the specified summary are selected does not solve the issue.

Expected Behavior

I should be able to use summary keys in my custom charts that aren't set by all runs.

Steps To Reproduce

See sweep workspace: https://wandb.ai/step-emg/Audio%20Signal%20Processing/sweeps/7hf398t3/

Screenshots

Using _timestamp, no problems:

Using Acc (only set by some runs) - Ok grayed out and unable to make chart:

Tested with filtering for non-null Acc:

Grouped bar chart editor, no errors?:

Environment

OS: Ubuntu 20.04

Browsers: Firefox 99.0

Additional Context

No response

[ramit-wandb]

Hi @eito-fis,

I'm sorry this is happening to you. This is a known issue, and I have bumped the priority on the internal ticket we have tracking issue. I'll let you know when there is some progress made on this.

Thanks, Ramit

[ramit-wandb]

Hi @eito-fis,

This issue should now be resolved. Please let me know if you still face issues with creating custom charts.

Thanks, Ramit

[ramit-wandb]

Closing this issue out. If you would like to continue this conversation, please let me know!

[eito-fis]

Unfortunately, I'm still running into the same problem, here is a screenshot from today where the ok button is still grayed out:

Moreover, I don't seem to be able to use the filter function anymore. Hitting the 0 filters dropdown causes the dropdown menu to disappear, leaving an empty filters: field:

[ramit-wandb]

Ah, I'm sorry to hear that. I'll report this to our engineering team and let you know how we can proceed over here.

Thanks, Ramit

[ramit-wandb]

Hi @eito-fis,

Quick question, how was this metric logged? Was this metric placed into the run using run.summary[key]=value instead of run.log({key, value})? This could be one potential reason for you to see this bug.

Thanks, Ramit

[eito-fis]

Hi Ramit,

In this particular case, the metric is added using wandb.summary.update(metric_dict). Could this be the root of the problem?

[ramit-wandb]

Hi @eito-fis,

Yes, I talked to some engineers on our end and looks like updating metrics directly through wandb.summary does not allow some keys to not form. As a test, could you try running wandb.log for this metric on another run and try again? I would like to confirm that this is the root cause.

We are looking into how we can resolve this formally, but if this is the issue, this might be the root cause behind it.

[abwilf]

Hi, was there any update on this? I like using summary metrics instead of logs for single values. Can I still do this and use custom charts?

[ramit-wandb]

Hey @abwilf,

This bug is being actively worked on as of now. There are certain cases in which this issue has been resolved, but others are still being looked into. Is there anything in particular you are doing with custom charts with which I can help?

[abwilf]

No I think that's it. Thank you so much!

[mario-dg]

Are there any news on this problem? I'm currently running into the same issue, where I can't add a similar grouped bar chart to my dashboard. I'm logging a wandb table using wandb.log in a PyTorch Lightning custom callback.

[alexaatm]

Having the same issues, when adding additional metrics to the summary using run.summary[key]=value andthen trying to plot them... See the plot in the custom chart editor but then cannot apply the changes and save it.

[alexaatm]

Here is a temporary workaround to be able to use new metrics with custom charts:

The idea is to resume the already finished runs and log the desired values there using wand.log(). Then they will appear in the UI, and plotting will be possible.

api = wandb.Api()
runs = api.runs(path=f"{ENTITY}/{PROJECT}") #can also filter here as usual
ALL_METRIC_NAMES = [ # I had a list of the metrics to use for calculated new summary metrics
    'mIoU',
    'Dice',
    'Pixel_Accuracy',
    'Recall',
    'Precision'
]

for run in tqdm(runs):

    run_id = run.id
    run_again = wandb.init(
        entity=ENTITY,
        project=PROJECT,
        id=run_id,
        resume='must' # This is important!
    )

    for metric in ALL_METRIC_NAMES:
        if metric not in run.history().columns:
            continue

        values = run.history()[metric].values
        values = values[~np.isnan(values)]

        # run.summary[f"total_{metric}"] = np.mean(values) # -> Won't be able to use it in wandb UI plots
        # run.summary[f"total_{metric}_std"] = np.std(values) # -> Won't be able to use it in wandb UI plots

        wandb.log({f"total_{metric}": np.mean(values)}) # -> Plotting works, because summary is updated during the run
        wandb.log({f"total_{metric}_std": np.std(values)})  # -> Plotting works, because summary is updated during the run

    # run.summary.update() #-> Needed for updating the summary, don't need if log using wandb.log()
    wandb.finish()

Related info that helped: https://community.wandb.ai/t/log-evaluation-to-finished-run/4648