Train fp16 interrupt

[5Yesterday]

Defaults for this optimization level are: enabled : True opt_level : O1 cast_model_type : None patch_torch_functions : True keep_batchnorm_fp32 : None master_weights : None loss_scale : dynamic Processing user overrides (additional kwargs that are not None)... After processing overrides, optimization options are: enabled : True opt_level : O1 cast_model_type : None patch_torch_functions : True keep_batchnorm_fp32 : None master_weights : None loss_scale : dynamic Selected optimization level O1: Insert automatic casts around Pytorch functions and Tensor methods.

Defaults for this optimization level are: enabled : True opt_level : O1 cast_model_type : None patch_torch_functions : True keep_batchnorm_fp32 : None master_weights : None loss_scale : dynamic Processing user overrides (additional kwargs that are not None)... After processing overrides, optimization options are: enabled : True opt_level : O1 cast_model_type : None patch_torch_functions : True keep_batchnorm_fp32 : None master_weights : None loss_scale : dynamic 1500 265 1500 265 Every epoch need 188 iterations Note that dataloader may hang with too much nworkers. DLoss: 6.0000 Reg: 0.0000

Process finished with exit code 139 (interrupted by signal 11: SIGSEGV) I installed apex, and use the fp16 config, output this.

[5Yesterday]

when I run in terminal, got 段错误(核心已转储). Maybe I should locate it step by step?

[layumi]

Hi @5Yesterday Please check the cuda version and your pytorch cuda version. Are they matched? And please check the installation of apex. Do you successfully compile the apex with gcc 5+ ?

[5Yesterday]

@layumi Thanks, it works. I have trained it when not use fp16, so it's the gcc problem, gcc-5 work.