Error in loading the pretrained model

[sedflix]

While trying to load the model given here, I'm facing the following problem:

.....
.....
  File "/home/sid/text-segmentation/evaluate.py", line 13, in load_model
    model = torch.load(f)
  File "/home/sid/miniconda2/envs/textseg/lib/python2.7/site-packages/torch/serialization.py", line 261, in load
    return _load(f, map_location, pickle_module)
  File "/home/sid/miniconda2/envs/textseg/lib/python2.7/site-packages/torch/serialization.py", line 399, in _load
    magic_number = pickle_module.load(f)
cPickle.UnpicklingError: A load persistent id instruction was encountered,
but no persistent_load function was specified.

[koomri]

I could not reproduce this problem. Did you install the right torch package? can you send the full stack? can you load models from other flow paths (e.g test_accuracy.py)

[sedflix]

I tried loading the model through several flow paths. Each time the above part was common in them.

I installed the torch package given the README.md file. Right now, I'm using docker to run it (Dockerfile). I was getting the same error while using conda env.

Stack for test_accuracy.py:

  File "test_accuracy.py", line 178, in <module>
    main(parser.parse_args())
  File "test_accuracy.py", line 79, in main
    model = torch.load(f)
  File "/usr/local/lib/python2.7/site-packages/torch/serialization.py", line 261, in load
    return _load(f, map_location, pickle_module)
  File "/usr/local/lib/python2.7/site-packages/torch/serialization.py", line 399, in _load
    magic_number = pickle_module.load(f)
cPickle.UnpicklingError: A load persistent id instruction was encountered,
but no persistent_load function was specified.

Does it have something do with the way the model was saved? (Maybe they cannot be loaded if not present at the same path where they were saved initially) As given here in the last section.

However in this case, the serialized data is bound to the specific classes and the exact directory 
structure used, so it can break in various ways when used in other projects, or after some serious 
refactors.

[koomri]

I don't think that it is related, since I can run the flow on various machines. Can you add the command that you are running?

[sedflix]

It's same as given in the README file(with a proper model path).

I was able to use models trained by me locally using the very same command.

[sedflix]

Config file:

root@f2f8e97440dc:/datadrive/sid/text-segmentation# cat configgenerator.py
import json

jsondata = {
    "word2vecfile": "/datadrive/sid/embedding",
    "choidataset": "/datadrive/sid/text-segmentationl/data/choi",
    "wikidataset": "/datadrive/sid/wiki_727",
}

with open('config.json', 'w') as f:
    json.dump(jsondata, f)

Example 1:

root@f2f8e97440dc:/datadrive/sid/text-segmentation# python test_accuracy.py --model checkpoints/model_cpu.t7 --wiki --test
Running with threshold: 0.4
Loading word2vec ellapsed: 0.00523281097412 seconds
running on wikipedia
Traceback (most recent call last):
  File "test_accuracy.py", line 178, in <module>
    main(parser.parse_args())
  File "test_accuracy.py", line 79, in main
    model = torch.load(f)
  File "/usr/local/lib/python2.7/site-packages/torch/serialization.py", line 261, in load
    return _load(f, map_location, pickle_module)
  File "/usr/local/lib/python2.7/site-packages/torch/serialization.py", line 399, in _load
    magic_number = pickle_module.load(f)
cPickle.UnpicklingError: A load persistent id instruction was encountered,
but no persistent_load function was specified.

Example 2:

root@f2f8e97440dc:/datadrive/sid/text-segmentation# python run_web_server.py --model checkpoints/model_cpu.t7 --test
Traceback (most recent call last):
  File "run_web_server.py", line 16, in <module>
    from webapp import app
  File "/datadrive/sid/text-segmentation/webapp/__init__.py", line 6, in <module>
    from webapp import views
  File "/datadrive/sid/text-segmentation/webapp/views.py", line 15, in <module>
    model = evaluate.load_model()
  File "/datadrive/sid/text-segmentation/evaluate.py", line 13, in load_model
    model = torch.load(f)
  File "/usr/local/lib/python2.7/site-packages/torch/serialization.py", line 261, in load
    return _load(f, map_location, pickle_module)
  File "/usr/local/lib/python2.7/site-packages/torch/serialization.py", line 399, in _load
    magic_number = pickle_module.load(f)
cPickle.UnpicklingError: A load persistent id instruction was encountered,
but no persistent_load function was specified.

[erickrf]

I've seen the same error occurring when you open a pickle file and try to load the same data from it more than once.

[SamMohel]

i'm facing the same problem how can i fix it ? excuse me what do you mean by "open a pickle file and try to load the same data from it more than once" appreciating your help

[erickrf]

@SamMohel that happens if you open a file, call pickle.load (so the contents are loaded) and then call it again. Pickle has reached the end of the file and there's nothing left to read. If that is your case, you probably should close the file and open it again.

[SamMohel]

thanks for replying , excuse me you mean I should try like this import pickle f = open('data.pkl', 'rb') mydict = pickle.load(f) f.close() print (mydict) as i got the same error mydict = pickle.load(f) _pickle.UnpicklingError: A load persistent id instruction was encountered, but no persistent_load function was specified.

On Sat, Apr 3, 2021, 12:29 PM Erick Rocha Fonseca @.***> wrote:

@SamMohel https://github.com/SamMohel that happens if you open a file, call pickle.load (so the contents are loaded) and then call it again. Pickle has reached the end of the file and there's nothing left to read. If that is your case, you probably should close the file and open it again.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/koomri/text-segmentation/issues/1#issuecomment-812846335, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHBVOVG3LVGW6FMHE2H7REDTG3UZNANCNFSM4FAZZOXQ .

[nutshell999]

The problem has been solved ? I have the same.

[BitnaKeum]

In my case, it was not the problem of Pickle and was problem of the version of torch. I upgraded the version of torch then the problem has solved.