Closed sunzhe09 closed 9 months ago
Hi Thanks for your interest. Have you removed the comments here? https://github.com/DLYuanGod/TinyGPT-V/blob/12b036a34090fb1f06d81f25c388b13db4c21fe3/README.md?plain=1#L122 and could you please provide more detail about the bug?
yes,I have remove the comments .log belows:
module.llama_model.base_model.model.model.layers.31.input_layernorm.weight
module.llama_model.base_model.model.model.layers.31.post_layernorm.weight
module.llama_model.base_model.model.model.final_layernorm.weight
module.llama_proj.weight
module.llama_proj.bias
module.llama_proj2.weight
module.llama_proj2.bias
2024-01-08 08:08:56,852 [INFO] number of trainable parameters: 45266944
2024-01-08 08:08:56,854 [INFO] Start training epoch 0, 200 iters per inner epoch.
[W reducer.cpp:1300] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
[W reducer.cpp:1300] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
Traceback (most recent call last):
File "/home/notebook/code/personal/80239864/TinyGPT-V/train.py", line 104, in
can you reproduce the errors? I test it on 2 V100 GPUs
Hi
I don't meet these errors, Did you use our environment.yml to bulid the env?
no when I create the env I got a cmake error,so I just install it by pip
Hi
There is a "pip" part in the environment.yml, you can create an env python==3.9 and pip install the right version. These errors may be caused by dependence.
after create the environment,the error is missing.Thank you
Describe the bug A clear and concise description of what the bug is.
To Reproduce Steps to reproduce the behavior: 1.uncommond code in base_model.py 2.torchrun --nproc-per-node NUM_GPU train.py --cfg-path train_configs/tinygptv_stage3.yaml
Expected behavior train success