[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest version of Polars.
Reproducible example
import polars as pl
from datetime import datetime
import numpy as np
import random, string
def randomword(length):
letters = string.ascii_lowercase
return ''.join(random.choice(letters) for i in range(length))
dates = pl.datetime_range(datetime(2000, 1, 1), datetime(2022, 3, 1), "1d", eager=True).alias(
"date"
)
categories = []
for i in dates:
for counter in range(1, 2000):
categories.append(randomword(6))
all_dates = pl.concat([dates for i in range(1, 2000)])
cats = pl.Series(categories)
df = pl.DataFrame({"dates": all_dates , "categories": cats})
df = df.with_columns(value = pl.lit(np.random.rand(df.height)))
interval = "5y"
df = df.sort(by="dates")
grouped_dynamic = df.sort(by="dates") .rolling(index_column="dates", period=interval, group_by='categories').agg([
(pl.col('value') + 1).cum_prod().shift(fill_value=1).cum_sum().last()])
print(grouped_dynamic)
Log output
N/A
Issue description
Issue Description:
During the execution of rolling operations on a large window, memory consumption increases significantly, ranging from 3 to 10 times the initial memory footprint. The operation involves a DataFrame comprising approximately 2000 categories spanning a 22-year period, with an initial memory footprint of around 1 to 2 gigabytes on the user's machine. However, upon executing rolling and aggregation functions, memory utilization escalates dramatically, potentially exhausting all available system memory resources. Notably, on Windows platforms, the memory remains at an elevated level even after the operation completes.
Steps to Reproduce:
Instantiate a DataFrame with approximately 2000 categories representing data spanning a 22-year period.
Execute rolling and aggregation operations on the DataFrame.
Monitor memory consumption during the operation.
Expected behavior
Memory consumption should remain within reasonable bounds relative to the initial size of the DataFrame, even during intensive rolling and aggregation operations.
Checks
Reproducible example
Log output
Issue description
Issue Description:
During the execution of rolling operations on a large window, memory consumption increases significantly, ranging from 3 to 10 times the initial memory footprint. The operation involves a DataFrame comprising approximately 2000 categories spanning a 22-year period, with an initial memory footprint of around 1 to 2 gigabytes on the user's machine. However, upon executing rolling and aggregation functions, memory utilization escalates dramatically, potentially exhausting all available system memory resources. Notably, on Windows platforms, the memory remains at an elevated level even after the operation completes.
Steps to Reproduce:
Instantiate a DataFrame with approximately 2000 categories representing data spanning a 22-year period. Execute rolling and aggregation operations on the DataFrame. Monitor memory consumption during the operation.
Expected behavior
Memory consumption should remain within reasonable bounds relative to the initial size of the DataFrame, even during intensive rolling and aggregation operations.
Installed versions