RecursionError: maximum recursion depth exceeded while calling a Python object

[XXmm-s]

大家有没有遇见这个错误RecursionError: maximum recursion depth exceeded while calling a Python object 造成这个错误的原因是什么呀。

[Topdu]

请列出详细的报错信息以及配置文件，可能是DataSet中getitem递归最大限制的错误。

[XXmm-s]

以下是配置文件

Global:
  device: gpu
  epoch_num: 20
  log_smooth_window: 20
  print_batch_step: 10
  output_dir: ./output/rec/svtr_base_igtr_ug
  save_epoch_step: 1
  # evaluation is run every 2000 iterations
  eval_batch_step: [0, 500]
  eval_epoch_step: [0, 1]
  cal_metric_during_train: True
  pretrained_model:
  checkpoints:
  use_tensorboard: false
  infer_img:
  # for data or label process
  character_dict_path: &character_dict_path ./tools/utils/charset_33.txt
  # ./tools/utils/EN_symbol_dict.txt # 96en
  # ./tools/utils/ppocr_keys_v1.txt  # ch
  max_text_length: &max_text_length 25
  use_space_char: &use_space_char False
  save_res_path: ./output/rec/predicts_igtr_ug.txt
  use_amp: True

Optimizer:
  name: AdamW
  lr: 0.0005 # 2gpus 384bs/gpu
  weight_decay: 0.05
  filter_bias_and_bn: True

LRScheduler:
  name: OneCycleLR
  warmup_epoch: 1.5
  cycle_momentum: False

Architecture:
  model_type: rec
  algorithm: BGPD
  in_channels: 3
  Transform:
  Encoder:
    name: SVTRNet2DPos
    img_size: [32, -1]
    out_char_num: 25
    out_channels: 256
    patch_merging: 'Conv'
    embed_dim: [128, 256, 384]
    depth: [6, 6, 6]
    num_heads: [4, 8, 12]
    mixer: ['ConvB','ConvB','ConvB','ConvB','ConvB','ConvB', 'ConvB','ConvB', 'Global','Global','Global','Global','Global','Global','Global','Global','Global','Global']
    local_mixer: [[5, 5], [5, 5], [5, 5]]
    last_stage: False
    prenorm: True
    use_first_sub: False
  Decoder:
    name: IGTRDecoder
    dim: 384
    num_layer: 1
    ar: False
    refine_iter: 0
    # next_pred: True
    next_pred: False
    pos2d: True
    ds: True
    # pos_len: False
    # rec_layer: 1

Loss:
  name: IGTRLoss

PostProcess:
  name: IGTRLabelDecode
  character_dict_path: *character_dict_path
  use_space_char: *use_space_char

Metric:
  name: RecMetric
  main_indicator: ACC

Train:
  dataset:
    name: RatioDataSet
    ds_width: True
    padding: &padding False
    data_dir_list: ['/home/xmm/workspace/PARseq/U_datas1/train/real/aug/aug1',
                    '/home/xmm/workspace/PARseq/U_datas1/train/real/raw']
    transforms:
      - DecodeImage: # load image
          img_mode: BGR
          channel_first: False
      - PARSeqAug:
      - IGTRLabelEncode: # Class handling label
          k: 8
          prompt_error: False
          character_dict_path: *character_dict_path
          use_space_char: *use_space_char
          max_text_length: *max_text_length
      - KeepKeys:
          keep_keys: ['image', 'label', 'prompt_pos_idx_list',
          'prompt_char_idx_list', 'ques_pos_idx_list', 'ques1_answer_list',
          'ques2_char_idx_list', 'ques2_answer_list', 'ques3_answer', 'ques4_char_num_list',
          'ques_len_list', 'ques2_len_list', 'prompt_len_list', 'length'] # dataloader will return list in this order
  sampler:
    name: RatioSampler
    scales: [[128, 32]] # w, h
    # divide_factor: to ensure the width and height dimensions can be devided by downsampling multiple
    first_bs: &bs 20
    fix_bs: false
    divided_factor: [4, 16] # w, h
    is_training: True
  loader:
    shuffle: True
    batch_size_per_card: *bs
    drop_last: True
    max_ratio: &max_ratio 4
    num_workers: 2

Eval:
  dataset:
    name: RatioDataSet
    ds_width: True
    padding: *padding
    data_dir_list: ['/home/xmm/workspace/PARseq/U_datas1/val']
    transforms:
      - DecodeImage: # load image
          img_mode: BGR
          channel_first: False
      - ARLabelEncode: # Class handling label
          character_dict_path: *character_dict_path
          use_space_char: *use_space_char
          max_text_length: *max_text_length
      - KeepKeys:
          keep_keys: ['image', 'label', 'length'] # dataloader will return list in this order
  sampler:
    name: RatioSampler
    scales: [[128, 32]] # w, h
    # divide_factor: to ensure the width and height dimensions can be devided by downsampling multiple
    first_bs: 256
    fix_bs: false
    divided_factor: [4, 16] # w, h
    is_training: False
  loader:
    shuffle: False
    drop_last: False
    batch_size_per_card: 20
    max_ratio: *max_ratio
    num_workers: 2

下面是报错信息

[2024/07/19 12:12:24] openrec INFO: valid dataloader has 9 iters
[2024/07/19 12:12:24] openrec INFO: {'Total': 24100888, 'Trainable': 24100888}
[2024/07/19 12:12:26] openrec INFO: train from scratch
[2024/07/19 12:12:26] openrec INFO: run with torch 2.2.0 and device cuda:0
[2024/07/19 12:12:26] openrec INFO: During the training process, after the 0th epoch, an evaluation is run every 1 epoch
[2024/07/19 12:12:26] openrec INFO: During the training process, after the 0th iteration, an evaluation is run every 500 iterations
0 1381 32.0 69 20
0 3768 64.0 188 20
0 1509 96.0 75 20
0 254954 128.0 12747 20
Traceback (most recent call last):
  File "/home/xmm/M2/OpenOCR-main/tools/train_rec.py", line 37, in <module>
    main()
  File "/home/xmm/M2/OpenOCR-main/tools/train_rec.py", line 33, in main
    trainer.train()
  File "/home/xmm/M2/OpenOCR-main/tools/engine/trainer.py", line 241, in train
    for idx, batch in enumerate(self.train_dataloader):
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 631, in __next__
    data = self._next_data()
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1346, in _next_data
    return self._process_data(data)
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1372, in _process_data
    data.reraise()
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/_utils.py", line 722, in reraise
    raise exception
RecursionError: Caught RecursionError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/xmm/M2/OpenOCR-main/tools/data/ratio_dataset.py", line 205, in __getitem__
    return self.__getitem__([img_width, img_height, ids[0], ratio])
  File "/home/xmm/M2/OpenOCR-main/tools/data/ratio_dataset.py", line 205, in __getitem__
    return self.__getitem__([img_width, img_height, ids[0], ratio])
  File "/home/xmm/M2/OpenOCR-main/tools/data/ratio_dataset.py", line 205, in __getitem__
    return self.__getitem__([img_width, img_height, ids[0], ratio])
  [Previous line repeated 953 more times]
  File "/home/xmm/M2/OpenOCR-main/tools/data/ratio_dataset.py", line 194, in __getitem__
    outs = transform(data, self.ops[:-1])
  File "/home/xmm/M2/OpenOCR-main/openrec/preprocess/__init__.py", line 44, in transform
    data = op(data)
  File "/home/xmm/M2/OpenOCR-main/openrec/preprocess/rec_aug.py", line 32, in __call__
    img = np.array(self.transforms(Image.fromarray(img)))
  File "/home/xmm/M2/OpenOCR-main/openrec/preprocess/auto_augment.py", line 759, in __call__
    img = op(img)
  File "/home/xmm/M2/OpenOCR-main/openrec/preprocess/auto_augment.py", line 405, in __call__
    return self.aug_fn(img, *level_args, **self.kwargs)
  File "/home/xmm/M2/OpenOCR-main/openrec/preprocess/parseq_aug.py", line 101, in poisson_noise
    return Image.fromarray(op(image=np.asarray(img)))
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/imgaug/augmenters/meta.py", line 2008, in __call__
    return self.augment(*args, **kwargs)
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/imgaug/augmenters/meta.py", line 1979, in augment
    batch_aug = self.augment_batch_(batch, hooks=hooks)
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/imgaug/augmenters/meta.py", line 641, in augment_batch_
    batch_inaug = self._augment_batch_(
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/imgaug/augmenters/arithmetic.py", line 1725, in _augment_batch_
    values = self.value.draw_samples(sample_shape, random_state=rs)
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/imgaug/parameters.py", line 412, in draw_samples
    samples = self._draw_samples(
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/imgaug/parameters.py", line 2284, in _draw_samples
    samples = self.other_param.draw_samples(size, random_state=rss[0])
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/imgaug/parameters.py", line 412, in draw_samples
    samples = self._draw_samples(
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/imgaug/parameters.py", line 1019, in _draw_samples
    lam = self.lam.draw_sample(random_state=random_state)
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/imgaug/parameters.py", line 387, in draw_sample
    return self.draw_samples(1, random_state=random_state)[0]
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/imgaug/parameters.py", line 413, in draw_samples
    size if not ia.is_single_integer(size) else tuple([size]),
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/imgaug/imgaug.py", line 256, in is_single_integer
    return isinstance(val, numbers.Integral) and not isinstance(val, bool)
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/abc.py", line 98, in __instancecheck__
    return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded

[2024-07-19 12:12:40,603] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 30401) of binary: /home/xmm/anaconda3/envs/openocr/bin/python
Traceback (most recent call last):
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/distributed/launch.py", line 198, in <module>
    main()
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/distributed/launch.py", line 194, in main
    launch(args)
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/distributed/launch.py", line 179, in launch
    run(args)
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/distributed/run.py", line 803, in run
    elastic_launch(
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 135, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/xmm/anaconda3/envs/openocr/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
tools/train_rec.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-19_12:12:40
  host      : xmm
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 30401)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

[Topdu]

检查一下图片读取是否正常，报错是在parseq_aug.py", line 101, in poisson_noise函数中。或者试试python3.8

[Topdu]

这可能是图片在做数据增强的时候报错，导致DataSet不能正常返回数据，训练无法进行。可以通过保存图片的形式检查一下图片是否读取正常。

[Henryplay]

相同的情况，只不过我的是在eval时报的错，猜测是图像处理的时候出的问题，把img mode由RGB改成GRAY不再报错。