HoloMap doesn't show new graph when previous graph was empty.

ALL software version info

holoviews: 1.18.3

Description of expected behavior and the observed behavior

I'm using HoloMap to switch between graphs with a slider. When I create a layout of 2 HoloMap's, these are automatically combined. This is not a real big issue, but when one HoloMap has fewer plots as the other HoloMap it can create the following issue which is also shown in the code below. Imagine there are 2 HoloMaps that are combined in a layout.

HoloMap A has 2 plots with HoloMap value 1 and 3.
HoloMap B has 3 plots with HoloMap value 1, 2 and 3. The layout will combine the plots and create 1 slider to guide through all the plots. HoloMap value 1 and 2 are shown correctly. HoloMap value 2 only has a plot for the HoloMap B which is correct. However HoloMap value 3 should have a graph for both HoloMap A and HoloMap B but this is not always the case. The graph for HoloMap A is only shown when you directly switch the HoloMap value 1 to 3. When you switch the HoloMap value from 2 to 3, there is no graph for the HoloMap B.

Complete, minimal, self-contained example code that reproduces the issue

import holoviews as hv
import pandas as pd
df = pd.DataFrame({'x': [1,2,3], 'y': [2,3,4],})
p1a = hv.Curve(df, kdims='x', vdims=['y'])
df = pd.DataFrame({'x': [1,2,3], 'y': [7,5,8],})
p3a = hv.Curve(df, kdims='x', vdims=['y'])
pa = hv.HoloMap({1:p1a, 3:p3a}, 'plots')

df = pd.DataFrame({'x': [1,2,3], 'y': [2,3,4],})
p1b = hv.Curve(df, kdims='x', vdims=['y'])
df = pd.DataFrame({'x': [1,2,3], 'y': [5,6,7],})
p2b = hv.Curve(df, kdims='x', vdims=['y'])
df = pd.DataFrame({'x': [1,2,3], 'y': [7,5,8],})
p3b = hv.Curve(df, kdims='x', vdims=['y'])
pb = hv.HoloMap({1:p1b, 2:p2b, 3:p3b}, 'plots')

p = (pa+pb).cols(1)
p

Screenshots or screencasts of the bug in action

The top graph should show a line for "Plots: 3" as shown here:

However, the top line is not shown when coming from "Plots: 2" where the top graph doesn't have any data.

holoviz / holoviews