Optimizing dictionary data structures fails when using a partially initialized function

Lightning-AI / litdata

Streamline data pipelines for AI. Process datasets across 1000s of machines, and optimize data for blazing fast model training.

Apache License 2.0

250 stars 24 forks source link

Optimizing dictionary data structures fails when using a partially initialized function #120

Open enrico-stauss opened 1 month ago

enrico-stauss commented 1 month ago

When passing fn=partial(my_func, arg1=arg1) to optimize, the line linked below will yield only the keys of the dictionary that my_funcreturns. I avoided the issue by not using partial for my usecase but I`d consider this a bug.

Kind regards :)

https://github.com/Lightning-AI/litdata/blob/a09de861eadcb8389d0bc3065a901113c9e72a5f/src/litdata/processing/functions.py#L138

github-actions[bot] commented 1 month ago

Hi! thanks for your contribution!, great first issue!

tchaton commented 1 month ago

Hey @enrico-stauss, Could you try out the latest version of litdata ?