Open XXmm-s opened 1 month ago
请列出详细的报错信息以及配置文件,可能是DataSet中getitem递归最大限制的错误。
以下是配置文件
Global:
device: gpu
epoch_num: 20
log_smooth_window: 20
print_batch_step: 10
output_dir: ./output/rec/svtr_base_igtr_ug
save_epoch_step: 1
# evaluation is run every 2000 iterations
eval_batch_step: [0, 500]
eval_epoch_step: [0, 1]
cal_metric_during_train: True
pretrained_model:
checkpoints:
use_tensorboard: false
infer_img:
# for data or label process
character_dict_path: &character_dict_path ./tools/utils/charset_33.txt
# ./tools/utils/EN_symbol_dict.txt # 96en
# ./tools/utils/ppocr_keys_v1.txt # ch
max_text_length: &max_text_length 25
use_space_char: &use_space_char False
save_res_path: ./output/rec/predicts_igtr_ug.txt
use_amp: True
Optimizer:
name: AdamW
lr: 0.0005 # 2gpus 384bs/gpu
weight_decay: 0.05
filter_bias_and_bn: True
LRScheduler:
name: OneCycleLR
warmup_epoch: 1.5
cycle_momentum: False
Architecture:
model_type: rec
algorithm: BGPD
in_channels: 3
Transform:
Encoder:
name: SVTRNet2DPos
img_size: [32, -1]
out_char_num: 25
out_channels: 256
patch_merging: 'Conv'
embed_dim: [128, 256, 384]
depth: [6, 6, 6]
num_heads: [4, 8, 12]
mixer: ['ConvB','ConvB','ConvB','ConvB','ConvB','ConvB', 'ConvB','ConvB', 'Global','Global','Global','Global','Global','Global','Global','Global','Global','Global']
local_mixer: [[5, 5], [5, 5], [5, 5]]
last_stage: False
prenorm: True
use_first_sub: False
Decoder:
name: IGTRDecoder
dim: 384
num_layer: 1
ar: False
refine_iter: 0
# next_pred: True
next_pred: False
pos2d: True
ds: True
# pos_len: False
# rec_layer: 1
Loss:
name: IGTRLoss
PostProcess:
name: IGTRLabelDecode
character_dict_path: *character_dict_path
use_space_char: *use_space_char
Metric:
name: RecMetric
main_indicator: ACC
Train:
dataset:
name: RatioDataSet
ds_width: True
padding: &padding False
data_dir_list: ['/home/xmm/workspace/PARseq/U_datas1/train/real/aug/aug1',
'/home/xmm/workspace/PARseq/U_datas1/train/real/raw']
transforms:
- DecodeImage: # load image
img_mode: BGR
channel_first: False
- PARSeqAug:
- IGTRLabelEncode: # Class handling label
k: 8
prompt_error: False
character_dict_path: *character_dict_path
use_space_char: *use_space_char
max_text_length: *max_text_length
- KeepKeys:
keep_keys: ['image', 'label', 'prompt_pos_idx_list',
'prompt_char_idx_list', 'ques_pos_idx_list', 'ques1_answer_list',
'ques2_char_idx_list', 'ques2_answer_list', 'ques3_answer', 'ques4_char_num_list',
'ques_len_list', 'ques2_len_list', 'prompt_len_list', 'length'] # dataloader will return list in this order
sampler:
name: RatioSampler
scales: [[128, 32]] # w, h
# divide_factor: to ensure the width and height dimensions can be devided by downsampling multiple
first_bs: &bs 20
fix_bs: false
divided_factor: [4, 16] # w, h
is_training: True
loader:
shuffle: True
batch_size_per_card: *bs
drop_last: True
max_ratio: &max_ratio 4
num_workers: 2
Eval:
dataset:
name: RatioDataSet
ds_width: True
padding: *padding
data_dir_list: ['/home/xmm/workspace/PARseq/U_datas1/val']
transforms:
- DecodeImage: # load image
img_mode: BGR
channel_first: False
- ARLabelEncode: # Class handling label
character_dict_path: *character_dict_path
use_space_char: *use_space_char
max_text_length: *max_text_length
- KeepKeys:
keep_keys: ['image', 'label', 'length'] # dataloader will return list in this order
sampler:
name: RatioSampler
scales: [[128, 32]] # w, h
# divide_factor: to ensure the width and height dimensions can be devided by downsampling multiple
first_bs: 256
fix_bs: false
divided_factor: [4, 16] # w, h
is_training: False
loader:
shuffle: False
drop_last: False
batch_size_per_card: 20
max_ratio: *max_ratio
num_workers: 2
下面是报错信息
[2024/07/19 12:12:24] openrec INFO: valid dataloader has 9 iters
[2024/07/19 12:12:24] openrec INFO: {'Total': 24100888, 'Trainable': 24100888}
[2024/07/19 12:12:26] openrec INFO: train from scratch
[2024/07/19 12:12:26] openrec INFO: run with torch 2.2.0 and device cuda:0
[2024/07/19 12:12:26] openrec INFO: During the training process, after the 0th epoch, an evaluation is run every 1 epoch
[2024/07/19 12:12:26] openrec INFO: During the training process, after the 0th iteration, an evaluation is run every 500 iterations
0 1381 32.0 69 20
0 3768 64.0 188 20
0 1509 96.0 75 20
0 254954 128.0 12747 20
Traceback (most recent call last):
File "/home/xmm/M2/OpenOCR-main/tools/train_rec.py", line 37, in <module>
main()
File "/home/xmm/M2/OpenOCR-main/tools/train_rec.py", line 33, in main
trainer.train()
File "/home/xmm/M2/OpenOCR-main/tools/engine/trainer.py", line 241, in train
for idx, batch in enumerate(self.train_dataloader):
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 631, in __next__
data = self._next_data()
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1346, in _next_data
return self._process_data(data)
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1372, in _process_data
data.reraise()
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/_utils.py", line 722, in reraise
raise exception
RecursionError: Caught RecursionError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
data = fetcher.fetch(index)
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/xmm/M2/OpenOCR-main/tools/data/ratio_dataset.py", line 205, in __getitem__
return self.__getitem__([img_width, img_height, ids[0], ratio])
File "/home/xmm/M2/OpenOCR-main/tools/data/ratio_dataset.py", line 205, in __getitem__
return self.__getitem__([img_width, img_height, ids[0], ratio])
File "/home/xmm/M2/OpenOCR-main/tools/data/ratio_dataset.py", line 205, in __getitem__
return self.__getitem__([img_width, img_height, ids[0], ratio])
[Previous line repeated 953 more times]
File "/home/xmm/M2/OpenOCR-main/tools/data/ratio_dataset.py", line 194, in __getitem__
outs = transform(data, self.ops[:-1])
File "/home/xmm/M2/OpenOCR-main/openrec/preprocess/__init__.py", line 44, in transform
data = op(data)
File "/home/xmm/M2/OpenOCR-main/openrec/preprocess/rec_aug.py", line 32, in __call__
img = np.array(self.transforms(Image.fromarray(img)))
File "/home/xmm/M2/OpenOCR-main/openrec/preprocess/auto_augment.py", line 759, in __call__
img = op(img)
File "/home/xmm/M2/OpenOCR-main/openrec/preprocess/auto_augment.py", line 405, in __call__
return self.aug_fn(img, *level_args, **self.kwargs)
File "/home/xmm/M2/OpenOCR-main/openrec/preprocess/parseq_aug.py", line 101, in poisson_noise
return Image.fromarray(op(image=np.asarray(img)))
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/imgaug/augmenters/meta.py", line 2008, in __call__
return self.augment(*args, **kwargs)
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/imgaug/augmenters/meta.py", line 1979, in augment
batch_aug = self.augment_batch_(batch, hooks=hooks)
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/imgaug/augmenters/meta.py", line 641, in augment_batch_
batch_inaug = self._augment_batch_(
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/imgaug/augmenters/arithmetic.py", line 1725, in _augment_batch_
values = self.value.draw_samples(sample_shape, random_state=rs)
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/imgaug/parameters.py", line 412, in draw_samples
samples = self._draw_samples(
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/imgaug/parameters.py", line 2284, in _draw_samples
samples = self.other_param.draw_samples(size, random_state=rss[0])
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/imgaug/parameters.py", line 412, in draw_samples
samples = self._draw_samples(
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/imgaug/parameters.py", line 1019, in _draw_samples
lam = self.lam.draw_sample(random_state=random_state)
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/imgaug/parameters.py", line 387, in draw_sample
return self.draw_samples(1, random_state=random_state)[0]
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/imgaug/parameters.py", line 413, in draw_samples
size if not ia.is_single_integer(size) else tuple([size]),
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/imgaug/imgaug.py", line 256, in is_single_integer
return isinstance(val, numbers.Integral) and not isinstance(val, bool)
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/abc.py", line 98, in __instancecheck__
return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded
[2024-07-19 12:12:40,603] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 30401) of binary: /home/xmm/anaconda3/envs/openocr/bin/python
Traceback (most recent call last):
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/distributed/launch.py", line 198, in <module>
main()
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/distributed/launch.py", line 194, in main
launch(args)
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/distributed/launch.py", line 179, in launch
run(args)
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/distributed/run.py", line 803, in run
elastic_launch(
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 135, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
tools/train_rec.py FAILED
------------------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2024-07-19_12:12:40
host : xmm
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 30401)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
检查一下图片读取是否正常,报错是在parseq_aug.py", line 101, in poisson_noise函数中。或者试试python3.8
这可能是图片在做数据增强的时候报错,导致DataSet不能正常返回数据,训练无法进行。可以通过保存图片的形式检查一下图片是否读取正常。
相同的情况,只不过我的是在eval时报的错,猜测是图像处理的时候出的问题,把img mode由RGB改成GRAY不再报错。
大家有没有遇见这个错误RecursionError: maximum recursion depth exceeded while calling a Python object 造成这个错误的原因是什么呀。