Closed lxtGH closed 4 years ago
We trained our model using PyTorch 1.0.0 and the latest TensorboardX.
I find even using the official PyTorch tensorboard has the problem.
A workaround is try to comment this line: https://github.com/HRNet/Higher-HRNet-Human-Pose-Estimation/blob/ce3a264f7dc4db937e2e3742c6fc097394775f21/tools/dist_train.py#L195
Thanks!
@bowenc0221 Yes, just comment the line , since the line "writer_dict['writer'].add_graph(model, (dump_input, ))" is not important for training. Then you can train your model.
Thank you very much!
Thank you, this fix worked for me.
I following the steps in the readme file. However, I met such error.
Traceback (most recent call last): File "tools/dist_train.py", line 312, in
main()
File "tools/dist_train.py", line 113, in main
args=(ngpus_per_node, args, final_output_dir, tb_log_dir)
File "/home/user/torch1.3/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
while not spawn_context.join():
File "/home/user/torch1.3/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 118, in join
raise Exception(msg)
Exception:
-- Process 0 terminated with the following error: Traceback (most recent call last): File "/home/user/torch1.3/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap fn(i, *args) File "/home/user/project/Higher-HRNet-Human-Pose-Estimation/tools/dist_train.py", line 195, in main_worker writer_dict['writer'].add_graph(model, (dump_input, )) File "/home/user/torch1.3/lib/python3.6/site-packages/tensorboardX/writer.py", line 738, in add_graph self._get_file_writer().add_graph(graph(model, input_to_model, verbose, kwargs)) File "/home/user/torch1.3/lib/python3.6/site-packages/tensorboardX/pytorch_graph.py", line 240, in graph trace = torch.jit.trace(model, args) File "/home/user/torch1.3/lib/python3.6/site-packages/torch/jit/init.py", line 858, in trace check_tolerance, _force_outplace, _module_class) File "/home/user/torch1.3/lib/python3.6/site-packages/torch/jit/init.py", line 991, in trace_module module = make_module(mod, _module_class, _compilation_unit, tuple(inputs.keys())) File "/home/user/torch1.3/lib/python3.6/site-packages/torch/jit/init.py", line 709, in make_module return _module_class(mod, _compilation_unit=_compilation_unit) File "/home/user/torch1.3/lib/python3.6/site-packages/torch/jit/init.py", line 1462, in init_then_register original_init(self, *args, *kwargs) File "/home/user/torch1.3/lib/python3.6/site-packages/torch/jit/init.py", line 1462, in init_then_register original_init(self, args, kwargs) File "/home/user/torch1.3/lib/python3.6/site-packages/torch/jit/init.py", line 1733, in init self._modules[name] = make_module(submodule, TracedModule, _compilation_unit) File "/home/user/torch1.3/lib/python3.6/site-packages/torch/jit/init.py", line 709, in make_module return _module_class(mod, _compilation_unit=_compilation_unit) File "/home/user/torch1.3/lib/python3.6/site-packages/torch/jit/init.py", line 1462, in init_then_register original_init(self, *args, kwargs) File "/home/user/torch1.3/lib/python3.6/site-packages/torch/jit/init.py", line 1733, in init self._modules[name] = make_module(submodule, TracedModule, _compilation_unit) File "/home/user/torch1.3/lib/python3.6/site-packages/torch/jit/init.py", line 709, in make_module return _module_class(mod, _compilation_unit=_compilation_unit) File "/home/user/torch1.3/lib/python3.6/site-packages/torch/jit/init.py", line 1462, in init_then_register original_init(self, *args, *kwargs) File "/home/user/torch1.3/lib/python3.6/site-packages/torch/jit/init.py", line 1733, in init self._modules[name] = make_module(submodule, TracedModule, _compilation_unit) File "/home/user/torch1.3/lib/python3.6/site-packages/torch/jit/init.py", line 709, in make_module return _module_class(mod, _compilation_unit=_compilation_unit) File "/home/user/torch1.3/lib/python3.6/site-packages/torch/jit/init.py", line 1462, in init_then_register original_init(self, args, kwargs) File "/home/user/torch1.3/lib/python3.6/site-packages/torch/jit/init.py", line 1733, in init self._modules[name] = make_module(submodule, TracedModule, _compilation_unit) File "/home/user/torch1.3/lib/python3.6/site-packages/torch/jit/init.py", line 709, in make_module return _module_class(mod, _compilation_unit=_compilation_unit) File "/home/user/torch1.3/lib/python3.6/site-packages/torch/jit/init.py", line 1462, in init_then_register original_init(self, *args, *kwargs) File "/home/user/torch1.3/lib/python3.6/site-packages/torch/jit/init.py", line 1733, in init self._modules[name] = make_module(submodule, TracedModule, _compilation_unit) File "/home/user/torch1.3/lib/python3.6/site-packages/torch/jit/init.py", line 709, in make_module return _module_class(mod, _compilation_unit=_compilation_unit) File "/home/user/torch1.3/lib/python3.6/site-packages/torch/jit/init.py", line 1462, in init_then_register original_init(self, args, kwargs) File "/home/user/torch1.3/lib/python3.6/site-packages/torch/jit/init.py", line 1710, in init assert(isinstance(orig, torch.nn.Module)) AssertionError What is the pytorch version and tensorboardX version ??**