hello, I get an error when converting multiple pdf files in one directory to mkdown. How can I adjust the parameters and optimize them?

[xuboot]

python convert.py input/ output/ --workers 1 --metadata_file metadata.json 2024-05-08 18:12:49,703 INFO worker.py:1642 -- Started a local Ray instance. Loading detection model vikp/surya_det2 on device cuda with dtype torch.float16 Loading detection model vikp/surya_layout2 on device cuda with dtype torch.float16 Loading reading order model vikp/surya_order on device cuda with dtype torch.float16 Loading recognition model vikp/surya_rec on device cuda with dtype torch.float16 Loaded texify model to cuda with torch.float16 dtype Converting 11 pdfs in chunk 1/1 with 1 processes, and storing in /root/test/marker/output 0%| | 0/11 [00:00<?, ?it/s]Detecting bboxes: 0%| | 0/1 [00:00<?, ?it/s] Detecting bboxes: 100%|██████████| 1/1 [00:00<00:00, 2.93it/s] 0%| | 0/11 [00:07<?, ?it/s]Recognizing Text: 0it [00:00, ?it/s] Detecting bboxes: 0%| | 0/1 [00:00<?, ?it/s] Detecting bboxes: 100%|██████████| 1/1 [00:00<00:00, 6.73it/s] Finding reading order: 0%| | 0/1 [00:00<?, ?it/s] Finding reading order: 100%|██████████| 1/1 [00:00<00:00, 1.58it/s] 9%|███████████████▋ | 1/11 [00:12<00:57, 5.78s/it](process_single_pdf pid=44873) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44873) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44873) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44873) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44873) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44873) -> Cannot close object, library is destroyed. This may cause a memory leak! Detecting bboxes: 0%| | 0/1 [00:00<?, ?it/s] Detecting bboxes: 100%|██████████| 1/1 [00:00<00:00, 1.31it/s] Recognizing Text: 0it [00:00, ?it/s] Detecting bboxes: 0%| | 0/1 [00:00<?, ?it/s] Detecting bboxes: 100%|██████████| 1/1 [00:00<00:00, 2.20it/s] Finding reading order: 0%| | 0/1 [00:00<?, ?it/s] Finding reading order: 100%|██████████| 1/1 [00:01<00:00, 1.08s/it] 18%|███████████████████████████████▍ | 2/11 [00:38<02:35, 17.28s/it](process_single_pdf pid=44935) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44935) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44935) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44935) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44935) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44935) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44935) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44935) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44935) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44935) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44935) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44935) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44935) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44935) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44935) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44935) -> Cannot close object, library is destroyed. This may cause a memory leak! Detecting bboxes: 0%| | 0/1 [00:00<?, ?it/s] 27%|███████████████████████████████████████████████▏ | 3/11 [00:46<01:44, 13.02s/it]Detecting bboxes: 0%| | 0/1 [00:01<?, ?it/s] (process_single_pdf pid=44995) Error converting /root/test/marker/input/基于事件相机的目标检测算法研究_张亚丽.pdf: CUDA out of memory. Tried to allocate 2.06 GiB. GPU 0 has a total capacty of 22.19 GiB of which 356.19 MiB is free. Process 34176 has 9.43 GiB memory in use. Process 44636 has 2.16 GiB memory in use. Including non-PyTorch memory, this process has 10.24 GiB memory in use. Of the allocated memory 6.52 GiB is allocated by PyTorch, and 3.43 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF (process_single_pdf pid=44995) Traceback (most recent call last): (process_single_pdf pid=44995) File "/root/test/marker/convert.py", line 39, in process_single_pdf (process_single_pdf pid=44995) full_text, images, out_metadata = convert_single_pdf(fname, model_refs, metadata=metadata) (process_single_pdf pid=44995) File "/root/test/marker/marker/convert.py", line 77, in convert_single_pdf (process_single_pdf pid=44995) surya_detection(doc, pages, detection_model) (process_single_pdf pid=44995) File "/root/test/marker/marker/ocr/detection.py", line 16, in surya_detection (process_single_pdf pid=44995) predictions = batch_text_detection(images, det_model, processor) (process_single_pdf pid=44995) File "/opt/anaconda3/envs/pp/lib/python3.10/site-packages/surya/detection.py", line 113, in batch_text_detection (process_single_pdf pid=44995) preds, orig_sizes = batch_detection(images, model, processor) (process_single_pdf pid=44995) File "/opt/anaconda3/envs/pp/lib/python3.10/site-packages/surya/detection.py", line 53, in batch_detection (process_single_pdf pid=44995) pred = model(pixel_values=batch) (process_single_pdf pid=44995) File "/opt/anaconda3/envs/pp/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl (process_single_pdf pid=44995) return self._call_impl(*args, kwargs) (process_single_pdf pid=44995) File "/opt/anaconda3/envs/pp/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl (process_single_pdf pid=44995) return forward_call(*args, *kwargs) (process_single_pdf pid=44995) File "/opt/anaconda3/envs/pp/lib/python3.10/site-packages/surya/model/detection/segformer.py", line 135, in forward (process_single_pdf pid=44995) logits = self.decode_head(encoder_hidden_states) (process_single_pdf pid=44995) File "/opt/anaconda3/envs/pp/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl (process_single_pdf pid=44995) return self._call_impl(args, kwargs) (process_single_pdf pid=44995) File "/opt/anaconda3/envs/pp/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl (process_single_pdf pid=44995) return forward_call(*args, kwargs) (process_single_pdf pid=44995) File "/opt/anaconda3/envs/pp/lib/python3.10/site-packages/surya/model/detection/segformer.py", line 93, in forward (process_single_pdf pid=44995) hidden_states = self.linear_fuse(torch.cat(all_hidden_states[::-1], dim=1)) (process_single_pdf pid=44995) File "/opt/anaconda3/envs/pp/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl (process_single_pdf pid=44995) return self._call_impl(*args, *kwargs) (process_single_pdf pid=44995) File "/opt/anaconda3/envs/pp/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl (process_single_pdf pid=44995) return forward_call(args, kwargs) (process_single_pdf pid=44995) File "/opt/anaconda3/envs/pp/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 460, in forward (process_single_pdf pid=44995) return self._conv_forward(input, self.weight, self.bias) (process_single_pdf pid=44995) File "/opt/anaconda3/envs/pp/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 456, in _conv_forward (process_single_pdf pid=44995) return F.conv2d(input, weight, bias, self.stride, (process_single_pdf pid=44995) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.06 GiB. GPU 0 has a total capacty of 22.19 GiB of which 356.19 MiB is free. Process 34176 has 9.43 GiB memory in use. Process 44636 has 2.16 GiB memory in use. Including non-PyTorch memory, this process has 10.24 GiB memory in use. Of the allocated memory 6.52 GiB is allocated by PyTorch, and 3.43 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF (process_single_pdf pid=44995) (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=44995) -> Cannot close object, library is destroyed. This may cause a memory leak! Detecting bboxes: 0%| | 0/1 [00:00<?, ?it/s] Detecting bboxes: 100%|██████████| 1/1 [00:00<00:00, 2.92it/s] Recognizing Text: 0it [00:00, ?it/s] Detecting bboxes: 0%| | 0/1 [00:00<?, ?it/s] Detecting bboxes: 100%|██████████| 1/1 [00:00<00:00, 6.54it/s] Finding reading order: 0%| | 0/1 [00:00<?, ?it/s] 36%|██████████████████████████████████████████████████████████████▉ | 4/11 [00:57<01:26, 12.43s/it]Finding reading order: 100%|██████████| 1/1 [00:00<00:00, 1.39it/s] (process_single_pdf pid=45023) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45023) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45023) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45023) -> Cannot close object, library is destroyed. This may cause a memory leak! Detecting bboxes: 0%| | 0/1 [00:00<?, ?it/s] 45%|██████████████████████████████████████████████████████████████████████████████▋ | 5/11 [01:05<01:04, 10.71s/it](process_single_pdf pid=45086) Error converting /root/test/marker/input/基于知识图谱与深度学习的零件机加工艺设计方法_李建勋.pdf: CUDA out of memory. Tried to allocate 5.51 GiB. GPU 0 has a total capacty of 22.19 GiB of which 1.85 GiB is free. Process 34176 has 9.43 GiB memory in use. Process 44636 has 2.16 GiB memory in use. Including non-PyTorch memory, this process has 8.74 GiB memory in use. Of the allocated memory 8.42 GiB is allocated by PyTorch, and 34.84 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF (process_single_pdf pid=45086) outputs = self.segformer( (process_single_pdf pid=45086) encoder_outputs = self.encoder( (process_single_pdf pid=45086) layer_outputs = blk(hidden_states, height, width, output_attentions) (process_single_pdf pid=45086) self_attention_outputs = self.attention( (process_single_pdf pid=45086) self_outputs = self.self(hidden_states, height, width, output_attentions) (process_single_pdf pid=45086) attention_scores = attention_scores / math.sqrt(self.attention_head_size) (process_single_pdf pid=45086) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 5.51 GiB. GPU 0 has a total capacty of 22.19 GiB of which 1.85 GiB is free. Process 34176 has 9.43 GiB memory in use. Process 44636 has 2.16 GiB memory in use. Including non-PyTorch memory, this process has 8.74 GiB memory in use. Of the allocated memory 8.42 GiB is allocated by PyTorch, and 34.84 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF (process_single_pdf pid=45086) Traceback (most recent call last): (process_single_pdf pid=45086) File "/root/test/marker/convert.py", line 39, in process_single_pdf (process_single_pdf pid=45086) full_text, images, out_metadata = convert_single_pdf(fname, model_refs, metadata=metadata) (process_single_pdf pid=45086) File "/root/test/marker/marker/convert.py", line 77, in convert_single_pdf (process_single_pdf pid=45086) surya_detection(doc, pages, detection_model) (process_single_pdf pid=45086) File "/root/test/marker/marker/ocr/detection.py", line 16, in surya_detection (process_single_pdf pid=45086) predictions = batch_text_detection(images, det_model, processor) (process_single_pdf pid=45086) File "/opt/anaconda3/envs/pp/lib/python3.10/site-packages/surya/detection.py", line 113, in batch_text_detection (process_single_pdf pid=45086) preds, orig_sizes = batch_detection(images, model, processor) (process_single_pdf pid=45086) File "/opt/anaconda3/envs/pp/lib/python3.10/site-packages/surya/detection.py", line 53, in batch_detection (process_single_pdf pid=45086) pred = model(pixel_values=batch) (process_single_pdf pid=45086) File "/opt/anaconda3/envs/pp/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl [repeated 6x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/ray-logging.html#log-deduplication for more options.) (process_single_pdf pid=45086) return self._call_impl(*args, kwargs) [repeated 6x across cluster] (process_single_pdf pid=45086) File "/opt/anaconda3/envs/pp/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl [repeated 6x across cluster] (process_single_pdf pid=45086) return forward_call(*args, **kwargs) [repeated 6x across cluster] (process_single_pdf pid=45086) File "/opt/anaconda3/envs/pp/lib/python3.10/site-packages/transformers/models/segformer/modeling_segformer.py", line 209, in forward [repeated 6x across cluster] (process_single_pdf pid=45086) Detecting bboxes: 0%| | 0/1 [00:00<?, ?it/s] (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45086) -> Cannot close object, library is destroyed. This may cause a memory leak! Detecting bboxes: 0%| | 0/1 [00:00<?, ?it/s] Detecting bboxes: 100%|██████████| 1/1 [00:00<00:00, 1.44it/s] Recognizing Text: 0%| | 0/1 [00:00<?, ?it/s] Recognizing Text: 100%|██████████| 1/1 [00:03<00:00, 3.45s/it] Detecting bboxes: 0%| | 0/1 [00:00<?, ?it/s] Detecting bboxes: 100%|██████████| 1/1 [00:00<00:00, 2.48it/s] Finding reading order: 0%| | 0/1 [00:00<?, ?it/s] Finding reading order: 100%|██████████| 1/1 [00:00<00:00, 1.22it/s] 55%|██████████████████████████████████████████████████████████████████████████████████████████████▎ | 6/11 [01:36<01:28, 17.63s/it](process_single_pdf pid=45115) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45115) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45115) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45115) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45115) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45115) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45115) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45115) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45115) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45115) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45115) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45115) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45115) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45115) -> Cannot close object, library is destroyed. This may cause a memory leak! Detecting bboxes: 0%| | 0/1 [00:00<?, ?it/s] Detecting bboxes: 100%|██████████| 1/1 [00:00<00:00, 1.35it/s] Recognizing Text: 0it [00:00, ?it/s] Detecting bboxes: 0%| | 0/1 [00:00<?, ?it/s] Detecting bboxes: 100%|██████████| 1/1 [00:00<00:00, 2.21it/s] Finding reading order: 0%| | 0/1 [00:00<?, ?it/s] Finding reading order: 100%|██████████| 1/1 [00:01<00:00, 1.14s/it] 64%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 7/11 [01:50<01:06, 16.63s/it](process_single_pdf pid=45179) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45179) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45179) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45179) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45179) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45179) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45179) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45179) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45179) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45179) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45179) -> Cannot close object, library is destroyed. This may cause a memory leak! Detecting bboxes: 0%| | 0/1 [00:00<?, ?it/s] Detecting bboxes: 100%|██████████| 1/1 [00:00<00:00, 2.94it/s] Recognizing Text: 0it [00:00, ?it/s] Detecting bboxes: 0%| | 0/1 [00:00<?, ?it/s] Detecting bboxes: 100%|██████████| 1/1 [00:00<00:00, 6.74it/s] Finding reading order: 0%| | 0/1 [00:00<?, ?it/s] Finding reading order: 100%|██████████| 1/1 [00:00<00:00, 1.62it/s] 73%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 8/11 [02:03<00:46, 15.47s/it](process_single_pdf pid=45242) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45242) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45242) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45242) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45242) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45242) -> Cannot close object, library is destroyed. This may cause a memory leak! Detecting bboxes: 0%| | 0/1 [00:00<?, ?it/s] 82%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 9/11 [02:11<00:25, 12.95s/it]Detecting bboxes: 0%| | 0/1 [00:01<?, ?it/s] (process_single_pdf pid=45303) Error converting /root/test/marker/input/基于特征融合的低剂量CT图像降噪研究_冉瑞生.pdf: CUDA out of memory. Tried to allocate 1.93 GiB. GPU 0 has a total capacty of 22.19 GiB of which 866.19 MiB is free. Process 34176 has 9.43 GiB memory in use. Process 44636 has 2.16 GiB memory in use. Including non-PyTorch memory, this process has 9.74 GiB memory in use. Of the allocated memory 6.23 GiB is allocated by PyTorch, and 3.22 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF (process_single_pdf pid=45303) Traceback (most recent call last): (process_single_pdf pid=45303) File "/root/test/marker/convert.py", line 39, in process_single_pdf (process_single_pdf pid=45303) full_text, images, out_metadata = convert_single_pdf(fname, model_refs, metadata=metadata) (process_single_pdf pid=45303) File "/root/test/marker/marker/convert.py", line 77, in convert_single_pdf (process_single_pdf pid=45303) surya_detection(doc, pages, detection_model) (process_single_pdf pid=45303) File "/root/test/marker/marker/ocr/detection.py", line 16, in surya_detection (process_single_pdf pid=45303) predictions = batch_text_detection(images, det_model, processor) (process_single_pdf pid=45303) File "/opt/anaconda3/envs/pp/lib/python3.10/site-packages/surya/detection.py", line 113, in batch_text_detection (process_single_pdf pid=45303) preds, orig_sizes = batch_detection(images, model, processor) (process_single_pdf pid=45303) File "/opt/anaconda3/envs/pp/lib/python3.10/site-packages/surya/detection.py", line 53, in batch_detection (process_single_pdf pid=45303) pred = model(pixel_values=batch) (process_single_pdf pid=45303) logits = self.decode_head(encoder_hidden_states) (process_single_pdf pid=45303) hidden_states = self.linear_fuse(torch.cat(all_hidden_states[::-1], dim=1)) (process_single_pdf pid=45303) return self._conv_forward(input, self.weight, self.bias) (process_single_pdf pid=45303) File "/opt/anaconda3/envs/pp/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 456, in _conv_forward (process_single_pdf pid=45303) return F.conv2d(input, weight, bias, self.stride, (process_single_pdf pid=45303) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.93 GiB. GPU 0 has a total capacty of 22.19 GiB of which 866.19 MiB is free. Process 34176 has 9.43 GiB memory in use. Process 44636 has 2.16 GiB memory in use. Including non-PyTorch memory, this process has 9.74 GiB memory in use. Of the allocated memory 6.23 GiB is allocated by PyTorch, and 3.22 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF (process_single_pdf pid=45303) (process_single_pdf pid=45303) File "/opt/anaconda3/envs/pp/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl [repeated 3x across cluster] (process_single_pdf pid=45303) return self._call_impl(*args, kwargs) [repeated 3x across cluster] (process_single_pdf pid=45303) File "/opt/anaconda3/envs/pp/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl [repeated 3x across cluster] (process_single_pdf pid=45303) return forward_call(*args, **kwargs) [repeated 3x across cluster] (process_single_pdf pid=45303) File "/opt/anaconda3/envs/pp/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 460, in forward [repeated 3x across cluster] (process_single_pdf pid=45303) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45303) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45303) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45303) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45303) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45303) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45303) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45303) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45303) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45303) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45303) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45303) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45303) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45303) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45303) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45303) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45303) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45303) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45303) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45303) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45303) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45303) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45303) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45303) -> Cannot close object, library is destroyed. This may cause a memory leak! Detecting bboxes: 0%| | 0/1 [00:00<?, ?it/s] Detecting bboxes: 100%|██████████| 1/1 [00:00<00:00, 2.92it/s] Recognizing Text: 0it [00:00, ?it/s] Detecting bboxes: 0%| | 0/1 [00:00<?, ?it/s] Detecting bboxes: 100%|██████████| 1/1 [00:00<00:00, 6.76it/s] Finding reading order: 0%| | 0/1 [00:00<?, ?it/s] Finding reading order: 100%|██████████| 1/1 [00:00<00:00, 1.44it/s] 91%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 10/11 [02:23<00:12, 12.78s/it](process_single_pdf pid=45332) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45332) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45332) -> Cannot close object, library is destroyed. This may cause a memory leak! (process_single_pdf pid=45332) -> Cannot close object, library is destroyed. This may cause a memory leak! Detecting bboxes: 0%| | 0/1 [00:00<?, ?it/s] Detecting bboxes: 100%|██████████| 1/1 [00:00<00:00, 2.89it/s] Recognizing Text: 0%| | 0/1 [00:00<?, ?it/s] Recognizing Text: 100%|██████████| 1/1 [00:02<00:00, 2.95s/it] Detecting bboxes: 0%| | 0/1 [00:00<?, ?it/s] Detecting bboxes: 100%|██████████| 1/1 [00:00<00:00, 6.87it/s] Finding reading order: 0%| | 0/1 [00:00<?, ?it/s] Finding reading order: 100%|██████████| 1/1 [00:00<00:00, 1.70it/s] 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████

[VikParuchuri]

It looks like you're using the dev branch. The batch sizes on that branch are not yet optimized for batch conversion. That is coming soon.