jorisvandenbossche
commented
8 months ago Thanks for the list! I was just planning to look at ASV for CoW today. Investigation of a few of the list:
reshape.Unstack.time_full_product
m = 100
n = 1000
levels = np.arange(m)
index = pd.MultiIndex.from_product([levels] * 2)
columns = np.arange(n)
values = np.arange(m * m * n).reshape(m * m, n)
df = pd.DataFrame(values, index, columns)
%timeit df.unstack()I am using pandas 2.2.0 to easily be able to switch between then default and CoW mode. With that, I can reproduce a slowdown in unstack. Profiling this with CoW enabled, this seems to be caused by a slower DataFrame construction of the final result in Unstacker.get_result. I assume this is because of the DataFrame constructor now copying an input numpy array, and in this case we can pass copy=False as we know we created the new values ourselves:
arithmetic.FrameWithFrameWide.time_op_different_blocks op=; shape=(1000000, 10)
n_rows, n_cols = 1_000_000, 10
# construct dataframe with 2 blocks
arr1 = np.random.randn(n_rows, n_cols // 2).astype("f8")
arr2 = np.random.randn(n_rows, n_cols // 2).astype("f4")
df = pd.concat([DataFrame(arr1), DataFrame(arr2)], axis=1, ignore_index=True)
df._consolidate_inplace()
arr1 = np.random.randn(n_rows, max(n_cols // 4, 3)).astype("f8")
arr2 = np.random.randn(n_rows, n_cols // 2).astype("i8")
arr3 = np.random.randn(n_rows, n_cols // 4).astype("f8")
df2 = pd.concat(
[DataFrame(arr1), DataFrame(arr2), DataFrame(arr3)],
axis=1,
ignore_index=True,
)
df2._consolidate_inplace()
%timeit df > df2This I can also reproduce, and a profile indicates the slowdown comes entirely from the actual > operation on the underlying data. So the one difference seems to be that with CoW, the underlying block values seem to be row major (column major in the transposed Block.values), while without CoW it's the expected column major layout.
Simpler example:
>>> pd.options.mode.copy_on_write = False
>>> arr = np.random.randn(n_rows, n_cols)
>>> df = pd.concat([pd.DataFrame(arr), pd.DataFrame(arr)], axis=1, ignore_index=True)
>>> df._mgr.blocks[0].values.shape
(10, 1000000)
>>> df._mgr.blocks[0].values.flags
C_CONTIGUOUS : True
F_CONTIGUOUS : False
OWNDATA : False
WRITEABLE : True
ALIGNED : True
WRITEBACKIFCOPY : False
>>> pd.options.mode.copy_on_write = True
>>> df = pd.concat([pd.DataFrame(arr), pd.DataFrame(arr)], axis=1, ignore_index=True)
>>> df._mgr.blocks[0].values.flags
C_CONTIGUOUS : False
F_CONTIGUOUS : True
OWNDATA : False
WRITEABLE : True
ALIGNED : True
WRITEBACKIFCOPY : FalseSo something seems to go wrong in the DataFrame constructor. With CoW enabled, this should make a copy by default, but when making this copy, it seems to not use the correct C vs F order.
series_methods.ToNumpy.time_to_numpy
N = 1_000_000
ser = pd.Series(np.random.randn(N,))
%timeit ser.to_numpy()gives
In [80]: %timeit ser.to_numpy()
846 ns ± 57.3 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each) # witout CoW
2.38 µs ± 10.7 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each) # with CoWIn this case there is some extra work being done to return a read-only view:
So it might be this is just that (but haven't checked in detail, from a quick profile nothing stood really out apart from just the body of to_numpy taking a bit longer)
phofl
rhshadrach
PR #56633 may have induced a performance regression. If it was a necessary behavior change, this may have been expected and everything is okay.
Please check the links below. If any ASVs are parameterized, the combinations of parameters that a regression has been detected for appear as subbullets.
Subsequent benchmarks may have skipped some commits. The link below lists the commits that are between the two benchmark runs where the regression was identified.
Commit Range
cc @phofl