Closed jaidisido closed 1 year ago
Side note, one thing that has caused me trouble with the synthesizer is the output_file_path
argument. My understanding is that if left to None
, it will "sneakily" create a temporary .csv file locally.
This is fine as is, but in my example I am running multiple synthesizer instances at once. As result, they are concurrently trying to write to the same file. As a workaround I am using a random filename in the argument to generate separate files, but at that point I am responsible for cleaning them up once done with the sampling.
Is there a particular reason why the file must be created at all? Appreciate this is a separate thing though so can create a separate issue in the repo
Hi @jaidisido! It’s great to see your interest in the SDV ecosystem. This comment is a reminder to consult your legal before adopting the SDV into your project, as the recent versions of SDV have a new source-available license.
For more information, you can read through our license FAQs (not a legal advice). For any other questions, you can reach us at info@sdv.dev. You can also inquire about a commercial license to allow additional use
Thank you for sharing I'll keep that in mind
Problem Description
I have discovered SDV this week and I am impressed by its capabilities, well done!
One thing I couldn't find in the docs is how to scale the sampling process. As you know, Pandas data frames are limited to a single machine and run into OOM issues pretty easily.
Expected behavior
Ray and Modin are two frameworks that can help alleviate these issues.
Here is a quick and dirty example I have put together to showcase what I mean
Additional context
Some context:
My team looks after a popular Python library named AWS SDK for pandas (20M+ downloads/month). Our focus has been on pandas but lately have been working on a new version (currently a release candidate) which scales beyond pandas thanks to Ray and Modin.
Some links for more details: