rb-determined-ai
commented
1 year ago The duplicate key problem is real. You have two containers, but one is reporting metrics before the other even says Creating _PyTorchTrialController with OneVarPytorchTrial.
The first time the second container reports metrics, you get the error.
Since I see the [rank=N] bits in the logs, and since you are using the legacy entrypoint format (that is, model_def:TrialClass), that means you are definitely inside of horovodrun, but somehow your PyTorchTrial isn't recognizing the fact that there's two workers that need to coordinate.
Have you modified the determined library by chance? This strikes me as an impossible bug.
If you haven't modified it, please add a print statement in your OneVarTrial.__init__():
print('rank', context.distributed.rank)
print('size', context.distributed.size)
print('local_rank', context.distributed.local_rank)
print('local_size', context.distributed.local_size)
print('cross_rank', context.distributed.cross_rank)
print('cross_size', context.distributed.cross_size)and share the resulting logs.
tzj-scau
Describe your question
I ran my own program using an image I built myself, and upon completion of the test I found the following problem that did not occur on a single GPU, when trying multiple GPUs (one GPU for each of the two machines)
determined.common.api.errors.APIException: {"error":{"code":13,"reason":"Internal","error":"failed to exec transaction (add training metrics): inserting metrics into raw_steps: ERROR: duplicate key value violates unique constraint \"steps_trial_id_total_batches_run_id_unique\" (SQLSTATE 23505)"}}
I tried searching for previous issues but did not find a specific solution, I am using version 0.22.2 of the determined version , using det deploy for deployment, in order to reproduce my error, I refer to a mod provided in the previous issue to try to reproduce my error, the error code as the following code reality
I'm a newbie and any tips and guidance would be greatly appreciated!
Checklist