thorstenwagner
commented
3 years ago Hi,
training on non square images is not a problem. The error you get looks more like corrupted training data. It seems that your training data does not contain the box width and height.
How did you create the training data?
Best, Thorsten
07.07.2021 01:46:56 jianhaoc @.***>:
Hi, I have some issue on train model. It sounds like crYOLO use only squared micrographs to train (see below).
Jianhao
##################################################### Important debugging information. In case of any problems, please provide this information. ##################################################### /opt/conda3/envs/cryolo/bin/cryolo_gui.py config --train_image_folder /data/test-cryolo/i --train_annot_folder /data/test-cryolo/yolo-train-box --saved_weights_name cryolo_model-train.h5 -a PhosaurusNet --input_size 1024 -nm STANDARD --num_patches 1 --overlap_patches 200 --filtered_output filtered_tmp/ -f NONE --low_pass_cutoff 0.1 --janni_overlap 24 --janni_batches 3 --train_times 10 --batch_size 4 --learning_rate 0.0001 --nb_epoch 200 --object_scale 5.0 --no_object_scale 1.0 --coord_scale 1.0 --class_scale 1.0 --debug --log_path logs/ -- config_cryolo-train.json 256 #####################################################
Wrote config to config_cryolo-train.json ##################################################### Important debugging information. In case of any problems, please provide this information. ##################################################### /opt/conda3/envs/cryolo/bin/cryolo_gui.py train -c /data/test-cryolo/config_cryolo-train.json -w 5 -g 3 -nc -1 --gpu_fraction 1.0 -e 10 -lft 2 --seed 10 ##################################################### Using TensorFlow backend. 2021-07-06 16:43:59.903515: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA 2021-07-06 16:43:59.913070: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz 2021-07-06 16:43:59.915416: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55da2c789660 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2021-07-06 16:43:59.915454: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2021-07-06 16:43:59.918923: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2021-07-06 16:44:00.135270: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55da2c80cb50 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2021-07-06 16:44:00.135327: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5 2021-07-06 16:44:00.138565: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:af:00.0 2021-07-06 16:44:00.139173: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-07-06 16:44:00.143025: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-07-06 16:44:00.146532: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-07-06 16:44:00.147243: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-07-06 16:44:00.151588: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-07-06 16:44:00.154904: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-07-06 16:44:00.164289: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-07-06 16:44:00.169416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-07-06 16:44:00.169489: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-07-06 16:44:00.173723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-07-06 16:44:00.173751: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-07-06 16:44:00.173770: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-07-06 16:44:00.178740: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10311 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:af:00.0, compute capability: 7.5) ############################################### The following training image sizes were detected: 5760 x 4092 ( N: 99 )
crYOLO will train in mode: NON_SQUARE Your training input size 1024 was adjusted to [1024, 1440] (height, width) ############################################### Traceback (most recent call last): File "/opt/conda3/envs/cryolo/bin/cryolo_gui.py", line 8, in sys.exit(/main/()) File "/opt/conda3/envs/cryolo/lib/python3.7/site-packages/cryolo/cryolo_main.py", line 463, in /main/ kwargs File "/opt/conda3/envs/cryolo/lib/python3.7/site-packages/gooey/python_bindings/gooey_decorator.py", line 93, in return lambda *args, *kwargs: func(args, kwargs) File "/opt/conda3/envs/cryolo/lib/python3.7/site-packages/cryolo/cryolo_main.py", line 424, in main train.main(args) File "/opt/conda3/envs/cryolo/lib/python3.7/site-packages/cryolo/train.py", line 513, in main anchor_size=int(config["model"]["anchors"][0]), File "/opt/conda3/envs/cryolo/lib/python3.7/site-packages/cryolo/preprocessing.py", line 205, in parse_annotation boxes = CoordsIO.read_eman1_boxfile(boxpath) File "/opt/conda3/envs/cryolo/lib/python3.7/site-packages/cryolo/CoordsIO.py", line 195, in read_eman1_boxfile return [BoundBox(x=box[0], y=box[1], w=box[2], h=box[3]) for box in boxreader] File "/opt/conda3/envs/cryolo/lib/python3.7/site-packages/cryolo/CoordsIO.py", line 195, in return [BoundBox(x=box[0], y=box[1], w=box[2], h=box[3]) for box in boxreader] IndexError: index 2 is out of bounds for axis 0 with size 2
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub[https://github.com/MPI-Dortmund/cryolo/issues/7], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AAIP66JFYAJV5OJFELOFZLLTWOIW5ANCNFSM475QZMQQ]. [data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAD8AAAA/CAYAAABXXxDfAAAAAXNSR0IArs4c6QAAAARzQklUCAgICHwIZIgAAAAmSURBVGiB7cEBDQAAAMKg909tDwcUAAAAAAAAAAAAAAAAAAAAJwY+QwABivJx1AAAAABJRU5ErkJggg==###24x24:true###][Verfolgungsbild][https://github.com/notifications/beacon/AAIP66N7IKJQ2F52PFJJYHTTWOIW5A5CNFSM475QZMQ2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4N7N7TLQ.gif]
jianhaoc
Hi, I have some issue on train model. It sounds like crYOLO uses only squared micrographs to train (see below).
Jianhao
##################################################### Important debugging information. In case of any problems, please provide this information. ##################################################### /opt/conda3/envs/cryolo/bin/cryolo_gui.py config --train_image_folder /data/test-cryolo/i --train_annot_folder /data/test-cryolo/yolo-train-box --saved_weights_name cryolo_model-train.h5 -a PhosaurusNet --input_size 1024 -nm STANDARD --num_patches 1 --overlap_patches 200 --filtered_output filtered_tmp/ -f NONE --low_pass_cutoff 0.1 --janni_overlap 24 --janni_batches 3 --train_times 10 --batch_size 4 --learning_rate 0.0001 --nb_epoch 200 --object_scale 5.0 --no_object_scale 1.0 --coord_scale 1.0 --class_scale 1.0 --debug --log_path logs/ -- config_cryolo-train.json 256 #####################################################
Wrote config to config_cryolo-train.json ##################################################### Important debugging information. In case of any problems, please provide this information. ##################################################### /opt/conda3/envs/cryolo/bin/cryolo_gui.py train -c /data/test-cryolo/config_cryolo-train.json -w 5 -g 3 -nc -1 --gpu_fraction 1.0 -e 10 -lft 2 --seed 10 ##################################################### Using TensorFlow backend. 2021-07-06 16:43:59.903515: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA 2021-07-06 16:43:59.913070: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz 2021-07-06 16:43:59.915416: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55da2c789660 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2021-07-06 16:43:59.915454: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2021-07-06 16:43:59.918923: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2021-07-06 16:44:00.135270: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55da2c80cb50 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2021-07-06 16:44:00.135327: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5 2021-07-06 16:44:00.138565: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:af:00.0 2021-07-06 16:44:00.139173: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-07-06 16:44:00.143025: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-07-06 16:44:00.146532: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-07-06 16:44:00.147243: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-07-06 16:44:00.151588: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-07-06 16:44:00.154904: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-07-06 16:44:00.164289: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-07-06 16:44:00.169416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-07-06 16:44:00.169489: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-07-06 16:44:00.173723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-07-06 16:44:00.173751: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-07-06 16:44:00.173770: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-07-06 16:44:00.178740: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10311 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:af:00.0, compute capability: 7.5) ############################################### The following training image sizes were detected: 5760 x 4092 ( N: 99 )
crYOLO will train in mode: NON_SQUARE Your training input size 1024 was adjusted to [1024, 1440] (height, width) ############################################### Traceback (most recent call last): File "/opt/conda3/envs/cryolo/bin/cryolo_gui.py", line 8, in
sys.exit(main())
File "/opt/conda3/envs/cryolo/lib/python3.7/site-packages/cryolo/cryolo_main.py", line 463, in main
kwargs
File "/opt/conda3/envs/cryolo/lib/python3.7/site-packages/gooey/python_bindings/gooey_decorator.py", line 93, in
return lambda *args, *kwargs: func(args, kwargs)
File "/opt/conda3/envs/cryolo/lib/python3.7/site-packages/cryolo/cryolo_main.py", line 424, in main
train.main(args)
File "/opt/conda3/envs/cryolo/lib/python3.7/site-packages/cryolo/train.py", line 513, in main
anchor_size=int(config["model"]["anchors"][0]),
File "/opt/conda3/envs/cryolo/lib/python3.7/site-packages/cryolo/preprocessing.py", line 205, in parse_annotation
boxes = CoordsIO.read_eman1_boxfile(boxpath)
File "/opt/conda3/envs/cryolo/lib/python3.7/site-packages/cryolo/CoordsIO.py", line 195, in read_eman1_boxfile
return [BoundBox(x=box[0], y=box[1], w=box[2], h=box[3]) for box in boxreader]
File "/opt/conda3/envs/cryolo/lib/python3.7/site-packages/cryolo/CoordsIO.py", line 195, in
return [BoundBox(x=box[0], y=box[1], w=box[2], h=box[3]) for box in boxreader]
IndexError: index 2 is out of bounds for axis 0 with size 2