search

d2l-ai / d2l-en

Interactive deep learning book with multi-framework code, math, and discussions. Adopted at 500 universities from 70 countries including Stanford, MIT, Harvard, and Cambridge.
https://D2L.ai
Other
23.6k stars 4.32k forks source link

Chapter 15.4. Pretraining word2vec: AttributeError: Can't pickle local object 'load_data_ptb.<locals>.PTBDataset' #2478

Open km5ar opened 1 year ago

km5ar commented 1 year ago

AttributeError: Can't pickle local object 'load_data_ptb..PTBDataset'

image

can anyone help with this error?

HJL-echoandy commented 1 year ago

这个问题搞定了了吗 请问

cx-olquinjica commented 1 year ago

@HJL-echoandy to anyone who is facing this problem I sort of found a quick fix.... After struggling with this same issue on a MacBook Pro M2, according to what I found online, this error message was mostly related to how Jupyter handles multithreading, particularly pickle.

So this solution worked for me:

  1. Create a python file in the same folder where you have your word_embedding_dataset.ipynb file, let's say you'll call it ptb_dataset.py
  2. Move (i.e., delete from the word_embedding_dataset.ipynb) the class PTBDataset(torch.utils.data.Dataset), and the function def batchify. Move them both to the ptb_dataset.py
  3. Import the class PTBDataset and the function batchify now into the word_embedding_dataset.ipynb (i.e., from ptb_dataset import PTBDataset, batchify
  4. Run the notebook again, and you shall be fine.

Note: You can refer to my notes in case of doubt.

Another thing to pay attention is that the only way I found to deal with this issue was to combine both lesson 15.3 and 15.4 in one notebook (i.e., dataset_word_embedding.ipynb), because I defined load_data_ptb locally.