search

opendatalab / DocLayout-YOLO

DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse Synthetic Data and Global-to-Local Adaptive Perception
https://huggingface.co/spaces/opendatalab/DocLayout-YOLO
GNU Affero General Public License v3.0
516 stars 36 forks source link

onnx bug #23

Closed xiaodongxi121 closed 3 weeks ago

xiaodongxi121 commented 3 weeks ago

I encountered an error while trying to convert the training model into an onnx model. AttributeError: 'Conv' object has no attribute 'bn'

xiaodongxi121 commented 3 weeks ago

`from doclayout_yolo import YOLOv10 model=YOLOv10("/DocLayout-YOLO-main/unidoc1/yolov10yolov10_layout_0710_epoch500_imgsz1600_bs12_pretrain_unknown/weights/best.pt")

model.export(format='onnx')

help='torchscript, onnx, openvino, engine, coreml, saved_model, pb, tflite, edgetpu, tfjs, paddle'`

This is my conversion code,hope you can help me solve it. Thank you!

JulioZhao97 commented 3 weeks ago

This is due to the GL_CRM module uses a weight-shared convolution, which uses a torch.nn.Conv2D (do not have attribute bn) rather than Conv below in nn/modules/conv.py (have attribute bn)

https://github.com/opendatalab/DocLayout-YOLO/blob/5fbc138175099c98a42381d0753ff50ebc5fe494/doclayout_yolo/nn/modules/conv.py#L36-L54

To fix this, you can disable fuse parameter in model initialization, example is shown in engine/validator.py:

https://github.com/opendatalab/DocLayout-YOLO/blob/5fbc138175099c98a42381d0753ff50ebc5fe494/doclayout_yolo/engine/validator.py#L125-L134

Currently I did not test the onnx, maybe a few days later, if you succeed you can submit a PR, much appreciated!

arthursunbao commented 3 weeks ago

no, it does not work even if fuse is False

JulioZhao97 commented 3 weeks ago

no, it does not work even if fuse is False

Thanks for your reply, I will fix this in these two days

xiaodongxi121 commented 3 weeks ago

Yesterday I tried the fuse method and found that it didn't solve the problem, thanks for the reply, I will continue to try to solve this problem in the coming days as well, if there is any progress, I will write back to you, thank you for your help!

---- Replied Message ---- | From | @.> | | Date | 10/30/2024 09:48 | | To | @.> | | Cc | @.> , @.> | | Subject | Re: [opendatalab/DocLayout-YOLO] onnx bug (Issue #23) |

no, it does not work even if fuse is False

Thanks for your reply, I will fix this in these two days

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

JulioZhao97 commented 3 weeks ago

remove this line: https://github.com/opendatalab/DocLayout-YOLO/blob/5fbc138175099c98a42381d0753ff50ebc5fe494/doclayout_yolo/engine/exporter.py#L226 then export model to onnx format:

from doclayout_yolo import YOLOv10
model=YOLOv10("./doclayout_yolo_docstructbench_imgsz1024.pt")
model.export(format='onnx')

then you are good to go! just replace model with your onnx model in demo.py

onnx-related versions:

onnx                      1.14.0                   pypi_0    pypi
onnxruntime               1.15.1                   pypi_0    pypi
onnxruntime-gpu           1.16.3                   pypi_0    pypi
onnxslim                  0.1.31                   pypi_0    pypi

If there are any other bugs, please let me know! @xiaodongxi121 @arthursunbao

xiaodongxi121 commented 3 weeks ago
Thank you for providing me with your help. I have tried the method you provided and successfully solved the onnx problem. In my actual exploration, I have also found that take https://github.com/opendatalab/DocLayout-YOLO/blob/5fbc138175099c98a42381d0753ff50ebc5fe494/doclayout_yolo/nn/modules/g2l_crm.py#L36 Change to bn = nn.Identity() can also solve this problem. I will test onnx in the future and ask you for advice if you have any questions. Thank you for your help！ lfy

@. | ---- Replied Message ---- | From | @.> | | Date | 10/30/2024 11:21 | | To | @.> | | Cc | @.> , @.***> | | Subject | Re: [opendatalab/DocLayout-YOLO] onnx bug (Issue #23) |

remove this line: https://github.com/opendatalab/DocLayout-YOLO/blob/5fbc138175099c98a42381d0753ff50ebc5fe494/doclayout_yolo/engine/exporter.py#L226 then export model to onnx format:

fromdoclayout_yoloimportYOLOv10model=YOLOv10("./doclayout_yolo_docstructbench_imgsz1024.pt") model.export(format='onnx')

then you are good to go! just replace model with your onnx model in demo.py

onnx-related versions:

onnx 1.14.0 pypi_0 pypi onnxruntime 1.15.1 pypi_0 pypi onnxruntime-gpu 1.16.3 pypi_0 pypi onnxslim 0.1.31 pypi_0 pypi

If there are any other bugs, please let me know! @@.***

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

JulioZhao97 commented 3 weeks ago

Glad to see problem solved, I will close this now, for any further questions you can also post them here.

jiang-yw commented 2 weeks ago

Yesterday I tried the fuse method and found that it didn't solve the problem, thanks for the reply, I will continue to try to solve this problem in the coming days as well, if there is any progress, I will write back to you, thank you for your help!

I'm a beginner with YOLO and would appreciate it if you could provide the ONNX inference code for learning purposes.

JulioZhao97 commented 2 weeks ago

Yesterday I tried the fuse method and found that it didn't solve the problem, thanks for the reply, I will continue to try to solve this problem in the coming days as well, if there is any progress, I will write back to you, thank you for your help!

I'm a beginner with YOLO and would appreciate it if you could provide the ONNX inference code for learning purposes.

Thanks for your interest on our work! For inference you can still use demo.py and just change model path to the exported onnx model.

jiang-yw commented 1 week ago

Yesterday I tried the fuse method and found that it didn't solve the problem, thanks for the reply, I will continue to try to solve this problem in the coming days as well, if there is any progress, I will write back to you, thank you for your help!

I'm a beginner with YOLO and would appreciate it if you could provide the ONNX inference code for learning purposes.

Thanks for your interest on our work! For inference you can still use demo.py and just change model path to the exported onnx model.

That works, thanks a lot!

Scorpion-cg commented 1 week ago

remove this line:

https://github.com/opendatalab/DocLayout-YOLO/blob/5fbc138175099c98a42381d0753ff50ebc5fe494/doclayout_yolo/engine/exporter.py#L226

then export model to onnx format:

from doclayout_yolo import YOLOv10
model=YOLOv10("./doclayout_yolo_docstructbench_imgsz1024.pt")
model.export(format='onnx')

then you are good to go! just replace model with your onnx model in demo.py

onnx-related versions:

onnx                      1.14.0                   pypi_0    pypi
onnxruntime               1.15.1                   pypi_0    pypi
onnxruntime-gpu           1.16.3                   pypi_0    pypi
onnxslim                  0.1.31                   pypi_0    pypi

If there are any other bugs, please let me know! @xiaodongxi121 @arthursunbao

I'm a beginner too.These days i want to take a progress on the finetune item.But i have confronted with the same errors:AttributeError: 'Conv' object has no attribute 'bn' The whole error message below: Traceback (most recent call last): File "/nfs1/56dev/root/CG/DocLayout-YOLO/train.py", line 66, in results = model.train( File "/nfs1/56dev/root/CG/DocLayout-YOLO/doclayout_yolo/engine/model.py", line 660, in train self.trainer.train() File "/nfs1/56dev/root/CG/DocLayout-YOLO/doclayout_yolo/engine/trainer.py", line 214, in train self._do_train(world_size) File "/nfs1/56dev/root/CG/DocLayout-YOLO/doclayout_yolo/engine/trainer.py", line 473, in _do_train self.final_eval() File "/nfs1/56dev/root/CG/DocLayout-YOLO/doclayout_yolo/engine/trainer.py", line 630, in final_eval self.metrics = self.validator(model=f) File "/nfs1/56dev/root/CG/DocLayout-YOLO/doclayout_yolo/engine/validator.py", line 170, in call model.warmup(imgsz=(1 if pt else self.args.batch, 3, imgsz, imgsz)) # warmup File "/nfs1/56dev/root/CG/DocLayout-YOLO/doclayout_yolo/nn/autobackend.py", line 586, in warmup self.forward(im) # warmup File "/nfs1/56dev/root/CG/DocLayout-YOLO/doclayout_yolo/nn/autobackend.py", line 420, in forward y = self.model(im, augment=augment, visualize=visualize, embed=embed) File "/root/anaconda3/envs/doclayout_yolo/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/root/anaconda3/envs/doclayout_yolo/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, *kwargs) File "/nfs1/56dev/root/CG/DocLayout-YOLO/doclayout_yolo/nn/tasks.py", line 96, in forward return self.predict(x, args, kwargs) File "/nfs1/56dev/root/CG/DocLayout-YOLO/doclayout_yolo/nn/tasks.py", line 114, in predict return self._predict_once(x, profile, visualize, embed) File "/nfs1/56dev/root/CG/DocLayout-YOLO/doclayout_yolo/nn/tasks.py", line 136, in _predict_once x = m(x) # run File "/root/anaconda3/envs/doclayout_yolo/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/root/anaconda3/envs/doclayout_yolo/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, *kwargs) File "/nfs1/56dev/root/CG/DocLayout-YOLO/doclayout_yolo/nn/modules/g2l_crm.py", line 114, in forward y.append(m(y[-1])) File "/root/anaconda3/envs/doclayout_yolo/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/root/anaconda3/envs/doclayout_yolo/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, kwargs) File "/nfs1/56dev/root/CG/DocLayout-YOLO/doclayout_yolo/nn/modules/g2l_crm.py", line 77, in forward return x + self.cv2(self.dilated_block(self.cv1(x))) if self.add else self.cv2(self.dilated_block(self.cv1(x))) File "/root/anaconda3/envs/doclayout_yolo/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/root/anaconda3/envs/doclayout_yolo/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(args, kwargs) File "/nfs1/56dev/root/CG/DocLayout-YOLO/doclayout_yolo/nn/modules/g2l_crm.py", line 43, in forward dx = [self.dilated_conv(x, d) for d in self.dilation] File "/nfs1/56dev/root/CG/DocLayout-YOLO/doclayout_yolo/nn/modules/g2l_crm.py", line 43, in dx = [self.dilated_conv(x, d) for d in self.dilation] File "/nfs1/56dev/root/CG/DocLayout-YOLO/doclayout_yolo/nn/modules/g2l_crm.py", line 36, in dilated_conv bn = self.dcv.bn File "/root/anaconda3/envs/doclayout_yolo/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1931, in getattr raise AttributeError( AttributeError: 'Conv' object has no attribute 'bn'

The terminal bash order is : python train.py --data /nfs1/56dev/root/CG/DocLayout-YOLO/finetune_datas/dataset --model m-doclayout --epoch 100 --image-size 1120 --batch-size 10 --project public_dataset/doclaynet --plot 1 --optimizer SGD --lr0 0.02 --pretrain doclayout_yolo_docsynth300k_imgsz1600.pt

The methods above this page i have tried ,but no working.If you can help me ,i'd appreciate it ~~

JulioZhao97 commented 1 week ago

File "/nfs1/56dev/root/CG/DocLayout-YOLO/doclayout_yolo/engine/trainer.py", line 214, in train self._do_train(world_size) File "/nfs1/56dev/root/CG/DocLayout-YOLO/doclayout_yolo/engine/trainer.py", line 473, in _do_train self.final_eval()

Thanks for feedback! Your error also indicate that validation fail in the last evaluation:

File "/nfs1/56dev/root/CG/DocLayout-YOLO/doclayout_yolo/engine/trainer.py", line 214, in train
self._do_train(world_size)
File "/nfs1/56dev/root/CG/DocLayout-YOLO/doclayout_yolo/engine/trainer.py", line 473, in _do_train
self.final_eval()

can model evaluate perform well during training process? You can directly using val.py to get the results of best.pt

Scorpion-cg commented 1 week ago

File "/nfs1/56dev/root/CG/DocLayout-YOLO/doclayout_yolo/engine/trainer.py", line 214, in train self._do_train(world_size) File "/nfs1/56dev/root/CG/DocLayout-YOLO/doclayout_yolo/engine/trainer.py", line 473, in _do_train self.final_eval()

Thanks for feedback! Your error also indicate that validation fail in the last evaluation:

File "/nfs1/56dev/root/CG/DocLayout-YOLO/doclayout_yolo/engine/trainer.py", line 214, in train
self._do_train(world_size)
File "/nfs1/56dev/root/CG/DocLayout-YOLO/doclayout_yolo/engine/trainer.py", line 473, in _do_train
self.final_eval()

can model evaluate perform well during training process? You can directly using val.py to get the results of best.pt

Thanks for reply . I can successfully run 50 epoch train process and get the best.pt and the last.pt.And i can directly run the val.py with the best.pt.But use the bash terminal i can not run the validated part.These are the all situations i have met~

JulioZhao97 commented 1 week ago

@Scorpion-cg I don't quite understand your expression. Did you mean you specify --epochs 100 (train for 100 epochs) but evaluation fail at the 50th epoch? But the traceback indicates evaluation fail at self.final_eval().

Scorpion-cg commented 1 week ago

@Scorpion-cg I don't quite understand your expression. Did you mean you specify --epochs 100 (train for 100 epochs) but evaluation fail at the 50th epoch? But the traceback indicates evaluation fail at self.final_eval().

Sry for late reply.I have already gotten the best.pt of 100 train epoch.But use the terminal bash order ,the error appeared in the val part.So i take your advice of using best.pt with the val.py directly which can get the metrics.I mean the bash order only train successfully ,and fail in the val part.^.^