Open Futyn-Maker opened 1 year ago
The same problem, repeats almost everywhere, with the same conditions. OSs Ubuntu & Windows 11.
Hey @The-One-Who-Speaks-and-Depicts and @Futyn-Maker, Any luck in solving this issue as I am facing the same error during model_train
@Moustafa-Banbouk I have been experiencing this for the year or so, no ideas. I just switched the validation off in args, and called it a day.
I just switched the validation off in args, and called it a day.
The same for now, it didn't really interfere with the project I was working on at the time - but I consider it an extremely critical bug.
@The-One-Who-Speaks-and-Depicts @Futyn-Maker @Moustafa-Banbouk Can you guys try disabling multi-processing using,
use_multiprocessing = False
use_multiprocessing_for_evaluation = False
@DamithDR At least in my case it works, I created a PR.
@Futyn-Maker @Moustafa-Banbouk /fyi
@The-One-Who-Speaks-and-Depicts
Glad that it helped :)
About the PR, I think this issue only re-creates on servers which are having multiple GPUs. The real issue is in the Seq2SeqDataset
class where it initiates a pool of processes to get the sample list. A proper fix will have to look into this area.
@DamithDR I had this issue on my laptop, and on a server, where I used only one GPU.
Describe the bug During the training of Seq2seq-type models, with evaluation enabled, a Pandas error
ValueError: All arrays must be of the same length
occurs in Google Colab (free plan) during evaluation. To Reproduce Steps to reproduce the behavior:Expected behavior A clear and concise description of what you expected to happen.
Learning and evaluating without failures
Screenshots If applicable, add screenshots to help explain your problem. Not applicable
Desktop (please complete the following information):
Here are the reduced logs: