_pickle.UnpicklingError: pickle data was truncated

[xiongdemao]

(autoner) A@7420:~/AutoNER-master$ ./autoner_train.sh === Compilation === mkdir -p bin g++ -std=c++11 -Wall -O3 -msse2 -fopenmp -I.. -pthread -lm -Wno-unused-result -Wno-sign-compare -Wno-unused-variable -Wno-parentheses -Wno-format -o bin/generate src/generate.cpp === Generating Distant Supervision === loading KB... core dict inserted full dict marked cleaning stopwords... initialized! # of trie nodes = 23819 === Encoding Dataset === Traceback (most recent call last): File "preprocess_partial_ner/encode_folder.py", line 262, in w_emb = pickle.load(f) _pickle.UnpicklingError: pickle data was truncated

[shangjingbo1226]

Did you see the embedding file (.pk) successfully downloaded?

[xiongdemao]

thanks for your answer ,may i ask for about could i use part of embedding file(.txt) or (.pk) because my computer can`t skill so mach data.

[xiongdemao]

and ,anther question is pytorch must or not use GPU? thank you again for your answer!

[shangjingbo1226]

• Embedding: You can use your own (smaller) embedding file. For example, you can download the embedding.txt file, and then filter them based the training corpus. We actually did that during the preprocessing, but loaded all embedding into the memory.

• GPU or not?: We ran all the experiments on GPU, but the CPU version should work. If it doesn't, you can try to modify the code. It should be not difficult.

[xiongdemao]

thanks for your answer

[junglefish8086]

Did you see the embedding file (.pk) successfully downloaded?

I have encountered the same problem. How can I check the embedding file (.pk) successfully downloaded? I searched this problem probably because the file format does not satisfy pickle.