Closed liu305 closed 6 months ago
Hello @liu305, thanks for sharing this! Could you please share a traceback of the error so I can see where in the code this is happening?
Hello. I did some debugging myself. After uncommenting line 441 of rtf_sampler.py, I no longer see such issues. So it seems that some invalid samples got leaked into the final stage and thus gave invalid string representation for a float number.
FYI, the trackback of the original error looks like this
Traceback (most recent call last):
File "/iter3/debugging/02_parent_model_training.py", line 17, in
Hello. In the parent model training during the validation phase when the model is generating synthetic data to compare with real raw data, the training is somehow terminated in the middle (usually around 45-50 epochs) by ValueError due to invalid string representation of float numbers. For example, I see something like "ValueError: could not convert string to float: '-0.-0.2'". I checked the model vocabulary and everything seems to be fine. For example, column_00 has tokens of "-0.", "01", -3.", and column_01 has tokens of "123","532","324". If for each processed column the model only generates tokens specific to that column, then why we would see '-0.-0.2'. It seems that tokens in column_00 is sampled for the location of column_01. Do you have any idea where the things could go wrong in the source code so that this issue could happen?