search

sicara / easy-few-shot-learning

Ready-to-use code and tutorial notebooks to boost your way into few-shot learning for image classification.
MIT License
975 stars 134 forks source link

`my_first_few_shot_classifier.ipynb` raises pickling error on Apple Silicon Mac #152

Open NMZ0429 opened 2 weeks ago

NMZ0429 commented 2 weeks ago

Describe the bug

To Reproduce Steps to reproduce the behavior:

  1. Simply running all cells of the notebook
  2. At the following cell, the aforementioned exception is raised.
(
    example_support_images,
    example_support_labels,
    example_query_images,
    example_query_labels,
    example_class_ids,
) = next(iter(test_loader))

plot_images(example_support_images, "support images", images_per_row=N_SHOT)
plot_images(example_query_images, "query images", images_per_row=N_QUERY)

Error

PicklingError: Can't pickle <function <lambda> at 0x13fee15a0>: attribute lookup <lambda> on __main__ failed

Additional context

Seems like the cause of this error is the line that defines get_labels method of dataset class. Specifically,

# The sampler needs a dataset with a "get_labels" method. Check the code if you have any doubt!
test_set.get_labels = lambda: [
    instance[1] for instance in test_set._flat_character_images
]

As I searched the old issues and referred https://github.com/sicara/easy-few-shot-learning/issues/121 , I have solved the issue by passing Omniglot to WrapFewShotDataset and delete the above line. Yet, I wrote this issue as I do not know if this is the best solution.

ebennequin commented 2 weeks ago

Hi! Thanks for the very clear issue report.

It seems to be the same issue as #51. It is related to the multiprocessing in PyTorch's DataLoader. Setting num_workers=0 should solve this error.

NMZ0429 commented 1 week ago

Thank you for your prompt reply and sorry for raising a duplicated issue. Before closing this, would you mind if I write a PR that adds a comment line warning Mac/Win users of the multiprocessing bug? Something like the following should be helpful.

test_loader = DataLoader(
    test_set,
    batch_sampler=test_sampler,
    num_workers=12, # Set to 0 if you get a "PicklingError" in the next cell
    pin_memory=True,
    collate_fn=test_sampler.episodic_collate_fn,
)
ebennequin commented 1 week ago

Yes, adding a warning would be valuable! I believe it's best to link to the issue for completeness, and maybe put it in the markdown rather than in the code.