search

gwxie / Document-Dewarping-with-Control-Points

Document Dewarping with Control Points
MIT License
156 stars 33 forks source link

[issues dataloader]Pickle data was truncated #19

Open Coder-C18 opened 2 years ago

Coder-C18 commented 2 years ago

i have issues when i trained model with 13k sample Traceback (most recent call last): File "train.py", line 319, in

<_io.BufferedReader name='/home/admin1/mnt_raid/source/caopv/dewaping/Source/Dataset/Train/color/452new_40_2_fold.gw'> train(args) File "train.py", line 143, in train for i, (images, labels, segment) in enumerate(trainloader): File "/mnt/raid1/software_app/anaconda3/envs/doctr/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 652, in __next__ data = self._next_data() File "/mnt/raid1/software_app/anaconda3/envs/doctr/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1327, in _next_data <_io.BufferedReader name='/home/admin1/mnt_raid/source/caopv/dewaping/Source/Dataset/Train/color/new_1013_19_curve.gw'> <_io.BufferedReader name='/home/admin1/mnt_raid/source/caopv/dewaping/Source/Dataset/Train/color/new_1324_17_fold.gw'> return self._process_data(data) File "/mnt/raid1/software_app/anaconda3/envs/doctr/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1373, in _process_data data.reraise() File "/mnt/raid1/software_app/anaconda3/envs/doctr/lib/python3.8/site-packages/torch/_utils.py", line 461, in reraise raise exception _pickle.UnpicklingError: Caught UnpicklingError in DataLoader worker process 3. Original Traceback (most recent call last): File "/mnt/raid1/software_app/anaconda3/envs/doctr/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop data = fetcher.fetch(index) File "/mnt/raid1/software_app/anaconda3/envs/doctr/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/mnt/raid1/software_app/anaconda3/envs/doctr/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in data = [self.dataset[idx] for idx in possibly_batched_index] File "/mnt/raid1/source/caopv/dewaping/Document-Dewarping-with-Control-Points/Source/dataloader.py", line 147, in __getitem__ perturbed_data = pickle.load(f) _pickle.UnpicklingError: pickle data was truncated
gwxie commented 2 years ago

hi, Check downloaded files for corruption, or provide more information.

love-ll commented 1 year ago

您好，我也遇到了这个问题，重新下载了pkl文件还是没有解决，请问是什么原因呢？错误输出如下：

Namespace(arch='Document-Dewarping-with-Control-Points', batch_size=8, data_path_test=PosixPath('/home/alyson/PycharmProjects/Document-Dewarping-with-Control-Points/Source/dataset/fiducial1024/png'), data_path_train='/media/alyson/DataDisk1/fiducial1024/fiducial1024/fiducial1024_v1', data_path_validate='/media/alyson/DataDisk1/fiducial1024/fiducial1024/fiducial1024_v1/validate', img_shrink=None, l_rate=0.0002, n_epoch=300, optimizer='adam', output_path=PosixPath('/home/alyson/PycharmProjects/Document-Dewarping-with-Control-Points/Source/flat'), parallel=['0'], print_freq=60, resume=PosixPath('/home/alyson/PycharmProjects/Document-Dewarping-with-Control-Points/Source/ICDAR2021/2021-02-03 16:15:55/143/2021-02-03 16_15_55flat_img_by_fiducial_points-fiducial1024_v1.pkl'), schema='train')

------load DilatedResnetForFlatByFiducialPointsS2------

Loading model and optimizer from checkpoint '/home/alyson/PycharmProjects/Document-Dewarping-with-Control-Points/Source/ICDAR2021/2021-02-03 16:15:55/143/2021-02-03 16_15_55flat_img_by_fiducial_points-fiducial1024_v1.pkl' Loaded checkpoint '2021-02-03 16_15_55flat_img_by_fiducial_points-fiducial1024_v1.pkl' (epoch 143)