ValueError: Cannot load file containing pickled data when allow_pickle=False

RodolfoFR commented 1 year ago

Search before asking

[X] I have searched the YOLOv8 issues and found no similar bug report.

YOLOv8 Component

No response

Bug

I'm trying to do a model training, but with 2% process of first epoch appear the following error:

Traceback (most recent call last): File "simple_train.py", line 16, in train = model.train( File "/usr/local/lib/python3.8/dist-packages/ultralytics/engine/model.py", line 372, in train self.trainer.train() File "/usr/local/lib/python3.8/dist-packages/ultralytics/engine/trainer.py", line 192, in train self._do_train(world_size) File "/usr/local/lib/python3.8/dist-packages/ultralytics/engine/trainer.py", line 315, in _do_train for i, batch in pbar: File "/usr/local/lib/python3.8/dist-packages/tqdm/std.py", line 1178, in iter for obj in iterable: File "/usr/local/lib/python3.8/dist-packages/ultralytics/data/build.py", line 38, in iter yield next(self.iterator) File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 633, in next data = self._next_data() File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 1325, in _next_data return self._process_data(data) File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 1371, in _process_data data.reraise() File "/usr/local/lib/python3.8/dist-packages/torch/_utils.py", line 644, in reraise raise exception ValueError: Caught ValueError in DataLoader worker process 3. Original Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop data = fetcher.fetch(index) File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/fetch.py", line 51, in data = [self.dataset[idx] for idx in possibly_batched_index] File "/usr/local/lib/python3.8/dist-packages/ultralytics/data/base.py", line 240, in getitem return self.transforms(self.get_image_and_label(index)) File "/usr/local/lib/python3.8/dist-packages/ultralytics/data/base.py", line 246, in get_image_and_label label['img'], label['ori_shape'], label['resized_shape'] = self.load_image(index) File "/usr/local/lib/python3.8/dist-packages/ultralytics/data/base.py", line 149, in load_image im = np.load(fn) File "/usr/local/lib/python3.8/dist-packages/numpy/lib/npyio.py", line 435, in load raise ValueError("Cannot load file containing pickled data " ValueError: Cannot load file containing pickled data when allow_pickle=False

i tried uniinstall package numpy and install numpy=1.22.2 (its specificed in yolov8 requirements), but don't worked

Environment

-OS: Ubuntu 20.04

Python 3.8.10

Minimal Reproducible Example

from ultralytics import YOLO

# Parameters for train the model dataset = "/home/rodolfo/desenvolvimento/project_dataset/dataset/data.yaml" weights = "/home/rodolfo/desenvolvimento/yolov8/train_640large_v2/result/weights/last.pt" epochs = 1500 batch_size = 16 img_size = 640 device = 0 result_folder = "/home/rodolfo/desenvolvimento/yolov8/train_640large_v2" verbose = True

model = YOLO(weights) train = model.train( data=dataset, epochs = epochs, batch = batch_size, imgsz = img_size, device = device, project=result_folder, name="result", pretrained = False, optimizer = "SGD", verbose=verbose, resume=True)

Additional

No response

Are you willing to submit a PR?

[X] Yes I'd like to help by submitting a PR!

glenn-jocher commented 1 year ago

@RodolfoFR i appreciate your detailed description of the issue and willingness to help by submitting a PR. Based on the error, it appears that the issue lies with loading pickled data in numpy where allow_pickle is set as False.

The YOLOv8 model pipelines directly utilize PyTorch Dataloader for data loading, they do not involve numpy's data loading functions, specifically np.load, as it seems like in your traceback. This suggests the issue might lie within your dataset and how it's formatted or saved rather than within YOLOv8 architecture.

Check if your dataset contains npy files, which are numpy arrays saved as files. From your traceback, it seems like your data loader is attempting to load a numpy file and encountering a pickle error. Numpy has disabled pickling by default from version 1.16.3 onwards for security reasons.

My recommendation would be to look into the specific content and format of your dataset files. If they are in numpy format (.npy), you might need to allow pickling to load them, or consider re-saving them in a different, non-pickled format.

I hope this helps guide your debugging process. Let us know if there are further issues.

glenn-jocher commented 1 year ago

@RodolfoFR hi I've tried to implement a fix for your issue in PR #4704. Can you install this branch and try to retrain your data to see if this fixes your error? The command to install this branch is:

pip install git+https://github.com/ultralytics/ultralytics.git@npsave

EDIT: Please delete all your *.npy files in your dataset directory first before retraining with the new PR, i.e.:

from pathlib import Path

for npy_file in Path('path/to/dataset').rglob('*.npy'):
    npy_file.unlink()

glenn-jocher commented 1 year ago

@RodolfoFR good news 😃! Your original issue may now be fixed ✅ in PR #4704.

To receive this update:

Git – git pull from within your yolov5/ directory or git clone https://github.com/ultralytics/yolov5 again
PyTorch Hub – Force-reload model = torch.hub.load('ultralytics/yolov5', 'yolov5s', force_reload=True)
Notebooks – View updated notebooks
Docker – sudo docker pull ultralytics/yolov5:latest to update your image

Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 🚀!

RodolfoFR commented 1 year ago

Glenn, thanks, I used the script to remove numpy files and it worked correctly. the training is normal.

glenn-jocher commented 1 year ago

@RodolfoFR i'm glad to hear that the script to remove numpy files solved your issue and that your training is now running smoothly. As you continue your work, please don't hesitate to reach out if you encounter any other concerns or issues. I am always here to help. Happy training with YOLOv8!

github-actions[bot] commented 11 months ago

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

Docs: https://docs.ultralytics.com
HUB: https://hub.ultralytics.com
Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐

ultralytics / ultralytics