Closed charismaticchiu closed 2 months ago
Hello @charismaticchiu,
Thank you very much for your interest in our work! Regarding the issue you mentioned, we are currently investigating the cause and expect to have an update within the next couple of days.
Thank you again for your interest!
Hello @charismaticchiu,
After investigating, we find that the original code saves all data into a single parquet file, which results in a file that is too large and likely causes the error you've encountered.
To address this, we update the inference logp code and the data reading code in the dataset. The logp files will now be stored in chunks of 5000 rows per file.
Please update your code to the latest version and see if this resolves the issue ~
If you have any other questions, please don't hesitate to ask. We're here to help!
Was using pickle as temporary hack. Also the new updates should work too. Thanks!
Hi I am getting this error loading the DPO dataset, does anyone know how to resolve it? Thank you!
I have this error even when my pandas version is 2.2.2