Scit3ch
commented
1 month ago @srinify Thank you very much for investigating further and providing a minimum working example.
A thought about your second workaround which would be a good solution for my problem with one addition.
If I limit the sequence length via the parameter sequence_length of the synthesizer all sequences will have the same length, which doesn't represent the real data I have and others may also have very well.
What about a parameter to limit the maximum sequence length to a defined value, but still allow sequences to be shorter.
This could be achieved by finding the longest sequence in the training data lets say it's 100 and the maximum sequence length should be 50. So this means a 50% decrease for this sequence. This percentage could then be applied to all sequences (e.g. a sequence of 30 of the real data becomes 15).
This would preserve the relative ratio of the sequence lengths between the training and generated data.
Regarding your first workaround: Setting enforce_min_max_values to false will effect all columns. An option to exclude certain user definable columns (like sequence_index column in this case) would be beneficial.
srinify
Environment Details
Error Description
When the desired sequence length is higher than the real data's sequence length and min-max enforcement is enabled, PARSynthesizer can generate duplicate values. This seems to happen especially when the sequence index column is a datetime column. When synthesizing values for the sequence key column, PARSynthesizer runs into the max value and repeats it.
Steps to reproduce
Original Data
2 sequences, each with 5 unique values for the
visitscolumn (the sequence index)Synthetic Data
Synthetic Data example when you set
sequence_lengthparameter to 25:Synthetic Data example when you set
sequence_lengthparameter to 7:Full code in Internal Colab Notebook here
Workarounds
enforce_min_max_valuesasFalseand this will remove the max value ceiling for the datetime sequence_key column. But this will mean that the synthesized data will be less representative of your real data so this is a big tradeoff until this bug is fixed.num_sequencesto be identical to the number of rows in your smallest, least unique (when it comes to the sequence key column) sequence from your real data. E.g. if you have a small sequence with 5 unique values for the sequence key, don't generate more than 5 rows per sequence. But this is also a limitation of SDV until this bug is fixed.Original Discussion here: https://github.com/sdv-dev/SDV/issues/2004