What's wrong

[thedeepdeepsky]

[tpgh24]

This looks like a Pythong library version issue. Are you running AG4Masses or the original AlphaGeometry? For AG4Masses, did you follow the setup instructions?

[thedeepdeepsky]

Hi I did so

[thedeepdeepsky]

ag.txt

[thedeepdeepsky]

I'm running AGM4

[thedeepdeepsky]

I did it step by step

[thedeepdeepsky]

The error message you're encountering, msgpack.exceptions.ExtraData: unpack(b) received extra data, suggests that there is an issue when trying to deserialize data using the msgpack library. This error occurs when msgpack.unpackb detects more data than it expects in the input.

Here's a breakdown of the traceback and potential areas to investigate:

1. File "/home/thedeepdeepsky/ag4masses/alphageometry/alphageometry.py", line 739, in main
   This is the entry point of your script where the main function is being called.

2. File "/home/thedeepdeepsky/ag4masses/alphageometry/alphageometry.py", line 652, in run_alphageometry
   The run_alphageometry function is called within the main function. This is where the initialization process starts.

3. File "/home/thedeepdeepsky/ag4masses/alphageometry/alphageometry.py", line 529, in bqsearch_init
   Within run_alphageometry, the bqsearch_init function is called. This function likely initializes some search-related components.

4. File "/home/thedeepdeepsky/ag4masses/alphageometry/alphageometry.py", line 213, in get_lm
   The get_lm function is responsible for loading the language model.

5. File "/home/thedeepdeepsky/ag4masses/alphageometry/lm_inference.py", line 62, in init
   The __init__ method of the LanguageModelInference class is invoked, initializing the language model.

6. File "/home/thedeepdeepsky/aglib/meliad/training_loop.py", line 391, in initialize_model
   The initialize_model method in the training_loop.py file is called, which initializes the model's training state.

7. File "/home/thedeepdeepsky/aglib/meliad/training_loop.py", line 436, in restore_checkpoint
   The restore_checkpoint method tries to restore the training state from a checkpoint.

8. File "/home/thedeepdeepsky/lib/python3.10/site-packages/flax/training/checkpoints.py", line 456, in restore_checkpoint
   The restore_checkpoint function from the flax library is used to restore the checkpoint.

9. File "/home/thedeepdeepsky/lib/python3.10/site-packages/flax/serialization.py", line 350, in msgpack_restore
   The msgpack_restore function tries to deserialize the checkpoint data.

10. File "msgpack/_unpacker.pyx", line 201, in msgpack._cmsgpack.unpackb
    The actual deserialization using msgpack.unpackb fails, resulting in the ExtraData exception.

Steps to Troubleshoot

1. Verify Checkpoint File: Ensure that the checkpoint file you are trying to load is not corrupted and contains only the expected data. You can try to load a different checkpoint to see if the issue persists.

2. Check Serialization/Deserialization: Verify that the way you serialize and deserialize the checkpoint data is consistent. Any discrepancy can cause issues. If you have control over how the checkpoint is saved, make sure it is being serialized correctly.

3. Debug the Checkpoint Content: You can manually inspect the checkpoint content before deserialization to ensure it matches the expected format. Adding debug statements to print the content of checkpoint_contents before calling msgpack.unpackb might help.

4. Library Versions: Ensure that the versions of the libraries you are using (e.g., flax, msgpack) are compatible with each other. Sometimes, library updates can introduce changes that break compatibility.

5. Partial Data Handling: Check if there is any mechanism that might have caused only a part of the checkpoint to be written or read, leading to extra data being present.

By systematically following these steps, you should be able to identify the root cause of the deserialization error and resolve it.

[thedeepdeepsky]

That's what gpt gave me \

[tpgh24]

GPT's suggestion is actually pretty good, did you just feed it the stack trace, or something more?

This looks like a Python package version issue. When installing the packages, did you use the requirements.txt file as I mentioned in the instructions, or the requirements.in file?

To check package versions, In your virtual environment, do a pip list to list the versions of packages, they should match what's in the requirements.txt file.

[thedeepdeepsky]

Package Version

---

absl-py 1.4.0 array-record 0.4.1 astunparse 1.6.3 Babel 2.13.0 beautifulsoup4 4.12.2 cached-property 1.5.2 cachetools 5.3.1 certifi 2023.7.22 charset-normalizer 3.3.0 chex 0.1.7 click 8.1.7 clu 0.0.7 colorama 0.4.6 commonmark 0.9.1 contextlib2 21.6.0 contourpy 1.1.1 cycler 0.12.1 dm-tree 0.1.8 docstring-parser 0.15 editdistance 0.6.2 etils 1.5.1 filelock 3.12.4 flatbuffers 23.5.26 flax 0.5.3 fonttools 4.43.1 fsspec 2023.9.2 future 0.18.3 gast 0.4.0 gdown 4.7.1 gin 0.1.6 gin-config 0.5.0 google-auth 2.23.3 google-auth-oauthlib 1.0.0 google-pasta 0.2.0 googleapis-common-protos 1.61.0 grpcio 1.59.0 h5py 3.10.0 huggingface-hub 0.17.3 idna 3.4 immutabledict 3.0.0 importlib-resources 6.1.0 jax 0.4.6 jaxlib 0.4.6 joblib 1.3.2 keras 2.13.1 kiwisolver 1.4.5 libclang 16.0.6 lxml 4.9.3 Markdown 3.5 markdown-it-py 3.0.0 MarkupSafe 2.1.3 matplotlib 3.7.0 mdurl 0.1.2 mesh-tensorflow 0.1.21 ml-collections 0.1.1 ml-dtypes 0.3.2 msgpack 1.0.7 namex 0.0.8 nest-asyncio 1.6.0 nltk 3.8.1 numpy 1.23.5 oauthlib 3.2.2 opt-einsum 3.3.0 optax 0.1.7 optree 0.11.0 orbax-checkpoint 0.1.1 packaging 23.2 pandas 2.1.1 Pillow 10.0.1 pip 22.0.2 portalocker 2.8.2 promise 2.3 protobuf 3.20.3 psutil 5.9.5 pyarrow 16.1.0 pyasn1 0.5.0 pyasn1-modules 0.3.0 pyglove 0.4.3 Pygments 2.16.1 pyparsing 3.1.1 PySocks 1.7.1 python-dateutil 2.8.2 pytz 2023.3.post1 PyYAML 6.0.1 regex 2023.10.3 requests 2.31.0 requests-oauthlib 1.3.1 rich 11.2.0 rouge-score 0.1.2 rsa 4.9 sacrebleu 2.3.1 safetensors 0.4.0 scikit-learn 1.3.1 scipy 1.10.0 sentencepiece 0.1.99 seqio 0.0.18 seqio-nightly 0.0.17.dev20231013 setuptools 59.6.0 simple_parsing 0.1.5 six 1.16.0 soupsieve 2.5 t5 0.9.4 tabulate 0.9.0 tensorboard 2.13.0 tensorboard-data-server 0.7.1 tensorflow 2.13.0 tensorflow-datasets 4.9.3 tensorflow-estimator 2.13.0 tensorflow-hub 0.15.0 tensorflow-io-gcs-filesystem 0.34.0 tensorflow-metadata 1.14.0 tensorflow-text 2.13.0 tensorstore 0.1.45 termcolor 2.3.0 tfds-nightly 4.9.2.dev202308090034 threadpoolctl 3.2.0 tk 0.1.0 tokenizers 0.14.1 toml 0.10.2 toolz 0.12.0 tqdm 4.66.1 transformers 4.34.0 typing_extensions 4.5.0 tzdata 2023.3 urllib3 2.0.6 Werkzeug 3.0.0 wheel 0.41.2 wrapt 1.15.0 zipp 3.17.0

[thedeepdeepsky]

But it seems ok thedeepdeepsky@Henry:~/ag4masses/alphageometry$ pip install --require-hashes --no-deps -r requirements.txt Requirement already satisfied: absl-py==1.4.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 7)) (1.4.0) Requirement already satisfied: array-record==0.4.1 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 28)) (0.4.1) Requirement already satisfied: astunparse==1.6.3 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 34)) (1.6.3) Requirement already satisfied: babel==2.13.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 38)) (2.13.0) Requirement already satisfied: beautifulsoup4==4.12.2 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 42)) (4.12.2) Requirement already satisfied: cached-property==1.5.2 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 46)) (1.5.2) Requirement already satisfied: cachetools==5.3.1 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 50)) (5.3.1) Requirement already satisfied: certifi==2023.7.22 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 54)) (2023.7.22) Requirement already satisfied: charset-normalizer==3.3.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 58)) (3.3.0) Requirement already satisfied: chex==0.1.7 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 150)) (0.1.7) Requirement already satisfied: click==8.1.7 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 154)) (8.1.7) Requirement already satisfied: clu==0.0.7 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 161)) (0.0.7) Requirement already satisfied: colorama==0.4.6 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 168)) (0.4.6) Requirement already satisfied: commonmark==0.9.1 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 174)) (0.9.1) Requirement already satisfied: contextlib2==21.6.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 178)) (21.6.0) Requirement already satisfied: contourpy==1.1.1 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 182)) (1.1.1) Requirement already satisfied: cycler==0.12.1 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 236)) (0.12.1) Requirement already satisfied: dm-tree==0.1.8 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 240)) (0.1.8) Requirement already satisfied: docstring-parser==0.15 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 284)) (0.15) Requirement already satisfied: editdistance==0.6.2 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 288)) (0.6.2) Requirement already satisfied: etils[array-types,enp,epath,epy,etqdm,etree]==1.5.1 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 371)) (1.5.1) Requirement already satisfied: filelock==3.12.4 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 379)) (3.12.4) Requirement already satisfied: flatbuffers==23.5.26 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 386)) (23.5.26) Requirement already satisfied: flax==0.5.3 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 390)) (0.5.3) Requirement already satisfied: fonttools==4.43.1 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 396)) (4.43.1) Requirement already satisfied: fsspec==2023.9.2 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 440)) (2023.9.2) Requirement already satisfied: future==0.18.3 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 446)) (0.18.3) Requirement already satisfied: gast==0.4.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 449)) (0.4.0) Requirement already satisfied: gdown==4.7.1 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 453)) (4.7.1) Requirement already satisfied: gin==0.1.6 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 457)) (0.1.6) Requirement already satisfied: gin-config==0.5.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 460)) (0.5.0) Requirement already satisfied: google-auth==2.23.3 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 467)) (2.23.3) Requirement already satisfied: google-auth-oauthlib==1.0.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 473)) (1.0.0) Requirement already satisfied: google-pasta==0.2.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 477)) (0.2.0) Requirement already satisfied: googleapis-common-protos==1.61.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 482)) (1.61.0) Requirement already satisfied: grpcio==1.59.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 486)) (1.59.0) Requirement already satisfied: h5py==3.10.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 544)) (3.10.0) Requirement already satisfied: huggingface-hub==0.17.3 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 571)) (0.17.3) Requirement already satisfied: idna==3.4 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 577)) (3.4) Requirement already satisfied: immutabledict==3.0.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 581)) (3.0.0) Requirement already satisfied: importlib-resources==6.1.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 585)) (6.1.0) Requirement already satisfied: jax==0.4.6 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 589)) (0.4.6) Requirement already satisfied: jaxlib==0.4.6 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 599)) (0.4.6) Requirement already satisfied: joblib==1.3.2 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 619)) (1.3.2) Requirement already satisfied: keras==2.13.1 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 625)) (2.13.1) Requirement already satisfied: kiwisolver==1.4.5 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 629)) (1.4.5) Requirement already satisfied: libclang==16.0.6 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 735)) (16.0.6) Requirement already satisfied: lxml==4.9.3 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 748)) (4.9.3) Requirement already satisfied: markdown==3.5 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 842)) (3.5) Requirement already satisfied: markupsafe==2.1.3 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 846)) (2.1.3) Requirement already satisfied: matplotlib==3.7.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 908)) (3.7.0) Requirement already satisfied: mesh-tensorflow[transformer]==0.1.21 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 953)) (0.1.21) Requirement already satisfied: ml-collections==0.1.1 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 957)) (0.1.1) Requirement already satisfied: msgpack==1.0.7 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 960)) (1.0.7) Requirement already satisfied: nltk==3.8.1 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1018)) (3.8.1) Requirement already satisfied: numpy==1.23.5 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1024)) (1.23.5) Requirement already satisfied: oauthlib==3.2.2 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1081)) (3.2.2) Requirement already satisfied: opt-einsum==3.3.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1085)) (3.3.0) Requirement already satisfied: optax==0.1.7 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1091)) (0.1.7) Requirement already satisfied: packaging==23.2 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1097)) (23.2) Requirement already satisfied: pandas==2.1.1 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1108)) (2.1.1) Requirement already satisfied: pillow==10.0.1 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1135)) (10.0.1) Requirement already satisfied: portalocker==2.8.2 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1191)) (2.8.2) Requirement already satisfied: promise==2.3 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1195)) (2.3) Requirement already satisfied: protobuf==3.20.3 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1200)) (3.20.3) Requirement already satisfied: psutil==5.9.5 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1233)) (5.9.5) Requirement already satisfied: pyasn1==0.5.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1251)) (0.5.0) Requirement already satisfied: pyasn1-modules==0.3.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1257)) (0.3.0) Requirement already satisfied: pyglove==0.4.3 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1261)) (0.4.3) Requirement already satisfied: pygments==2.16.1 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1267)) (2.16.1) Requirement already satisfied: pyparsing==3.1.1 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1271)) (3.1.1) Requirement already satisfied: pysocks==1.7.1 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1275)) (1.7.1) Requirement already satisfied: python-dateutil==2.8.2 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1280)) (2.8.2) Requirement already satisfied: pytz==2023.3.post1 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1286)) (2023.3.post1) Requirement already satisfied: pyyaml==6.0.1 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1290)) (6.0.1) Requirement already satisfied: regex==2023.10.3 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1346)) (2023.10.3) Requirement already satisfied: requests[socks]==2.31.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1439)) (2.31.0) Requirement already satisfied: requests-oauthlib==1.3.1 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1450)) (1.3.1) Requirement already satisfied: rich==11.2.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1454)) (11.2.0) Requirement already satisfied: rouge-score==0.1.2 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1458)) (0.1.2) Requirement already satisfied: rsa==4.9 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1461)) (4.9) Requirement already satisfied: sacrebleu==2.3.1 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1465)) (2.3.1) Requirement already satisfied: safetensors==0.4.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1469)) (0.4.0) Requirement already satisfied: scikit-learn==1.3.1 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1569)) (1.3.1) Requirement already satisfied: scipy==1.10.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1597)) (1.10.0) Requirement already satisfied: sentencepiece==0.1.99 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1626)) (0.1.99) Requirement already satisfied: seqio==0.0.18 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1677)) (0.0.18) Requirement already satisfied: seqio-nightly==0.0.17.dev20231013 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1681)) (0.0.17.dev20231013) Requirement already satisfied: six==1.16.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1685)) (1.16.0) Requirement already satisfied: soupsieve==2.5 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1699)) (2.5) Requirement already satisfied: t5==0.9.4 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1703)) (0.9.4) Requirement already satisfied: tabulate==0.9.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1706)) (0.9.0) Requirement already satisfied: tensorboard==2.13.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1710)) (2.13.0) Requirement already satisfied: tensorboard-data-server==0.7.1 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1713)) (0.7.1) Requirement already satisfied: tensorflow==2.13.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1718)) (2.13.0) Requirement already satisfied: tensorflow-datasets==4.9.3 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1743)) (4.9.3) Requirement already satisfied: tensorflow-estimator==2.13.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1750)) (2.13.0) Requirement already satisfied: tensorflow-hub==0.15.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1753)) (0.15.0) Requirement already satisfied: tensorflow-io-gcs-filesystem==0.34.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1756)) (0.34.0) Requirement already satisfied: tensorflow-metadata==1.14.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1777)) (1.14.0) Requirement already satisfied: tensorflow-text==2.13.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1782)) (2.13.0) Requirement already satisfied: tensorstore==0.1.45 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1792)) (0.1.45) Requirement already satisfied: termcolor==2.3.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1811)) (2.3.0) Requirement already satisfied: tfds-nightly==4.9.2.dev202308090034 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1818)) (4.9.2.dev202308090034) Requirement already satisfied: threadpoolctl==3.2.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1825)) (3.2.0) Requirement already satisfied: tokenizers==0.14.1 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1829)) (0.14.1) Requirement already satisfied: toml==0.10.2 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1929)) (0.10.2) Requirement already satisfied: toolz==0.12.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1935)) (0.12.0) Requirement already satisfied: tqdm==4.66.1 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1939)) (4.66.1) Requirement already satisfied: transformers==4.34.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1950)) (4.34.0) Requirement already satisfied: typing-extensions==4.5.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1954)) (4.5.0) Requirement already satisfied: tzdata==2023.3 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1963)) (2023.3) Requirement already satisfied: urllib3==2.0.6 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1967)) (2.0.6) Requirement already satisfied: werkzeug==3.0.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1971)) (3.0.0) Requirement already satisfied: wheel==0.41.2 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1975)) (0.41.2) Requirement already satisfied: wrapt==1.15.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 1981)) (1.15.0) Requirement already satisfied: zipp==3.17.0 in /home/thedeepdeepsky/lib/python3.10/site-packages (from -r requirements.txt (line 2061)) (3.17.0)

[tpgh24]

package versions look good. Check whether the checkpoint file size is correct, this is what I have:

-rw-rw-r-- 1 peng peng 1217608082 Jan 20 00:15 checkpoint_10999999
-rw-rw-r-- 1 peng peng      14334 Sep 17  2023 geometry.757.model
-rw-rw-r-- 1 peng peng      10025 Sep 17  2023 geometry.757.vocab

[thedeepdeepsky]

It looks all Ok

[thedeepdeepsky]

Let me redownload them

[thedeepdeepsky]

Thanks a lot! There is sth called checkpoint_10999999 Zone.Identifier It needs to be deleted

[tpgh24]

great. Strange where that file came from.