Checkpoint path should be absolute

[bhargav25dave1996]

at time of training i am geeting this error :

code : python -m tevatron.tevax.experimental.mp.train_lora \ --checkpoint_dir retriever-mistral-jax \ --train_file Tevatron/msmarco-passage-aug \ --model_name mistralai/Mistral-7B-v0.1 \ --model_type mistral \ --batch_size 128 \ --num_target_passages 16 \ --learning_rate 1e-4 \ --seed 12345 \ --mesh_shape 1 -1 \ --weight_decay 0.00001 \ --num_epochs 1 \ --max_query_length 64 \ --max_passage_length 128 \ --pooling eos \ --scale_by_dim True \ --grad_cache \ --passage_num_chunks 32 \ --query_num_chunks 4

Error:

Traceback (most recent call last): File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/home/irlab/tevatron/src/tevatron/tevax/experimental/mp/train_lora.py", line 394, in main() File "/home/irlab/tevatron/src/tevatron/tevax/experimental/mp/train_lora.py", line 375, in main checkpoint_manager.save( File "/home/irlab/.local/lib/python3.10/site-packages/orbax/checkpoint/checkpoint_manager.py", line 515, in save self._checkpointers[k].save(item_dir, item, *kwargs) File "/home/irlab/.local/lib/python3.10/site-packages/orbax/checkpoint/async_checkpointer.py", line 281, in save commit_ops = asyncio.run(self._handler.async_save(tmpdir, args=ckpt_args)) File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/usr/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete return future.result() File "/home/irlab/.local/lib/python3.10/site-packages/orbax/checkpoint/pytree_checkpoint_handler.py", line 835, in async_save commit_futures = await asyncio.gather(serialize_ops) File "/home/irlab/.local/lib/python3.10/site-packages/orbax/checkpoint/type_handlers.py", line 1376, in serialize tspec = self._get_json_tspec_write( File "/home/irlab/.local/lib/python3.10/site-packages/orbax/checkpoint/type_handlers.py", line 1273, in _get_json_tspec_write tspec = self._get_json_tspec( File "/home/irlab/.local/lib/python3.10/site-packages/orbax/checkpoint/type_handlers.py", line 1253, in _get_json_tspec tspec: Dict[str, Any] = get_tensorstore_spec( File "/home/irlab/.local/lib/python3.10/site-packages/orbax/checkpoint/type_handlers.py", line 821, in get_tensorstore_spec raise ValueError(f'Checkpoint path should be absolute. Got {directory}') ValueError: Checkpoint path should be absolute. Got retriever-mistral-jax/0.orbax-checkpoint-tmp-1711610071493337/lora.orbax-checkpoint-tmp-1711610103114668

[luyug]

As it notes,

Checkpoint path should be absolute.

This is for compatibility w/ cloud storage.

[bhargav25dave1996]

@luyug I am using Google Cloud TPU V4-8 as suggested. Can you help with the issue?

[MXueguang]

--checkpoint_dir retriever-mistral-jax

change to something like /home/<your user name>/retriever-mistral-jax should work.

[bhargav25dave1996]

@MXueguang Thanks, it resole issue.

I am facing an issue in encoding, I am using the below code to encode msmarco.

python -m tevatron.tevax.experimental.mp.encode \ --model_type mistral \ --model_name_or_path mistralai/Mistral-7B-v0.1 \ --model_config_name_or_path mistralai/Mistral-7B-v0.1 \ --tokenizer_name_or_path mistralai/Mistral-7B-v0.1 \ --dataset_name_or_path Tevatron/msmarco-passage-corpus \ --output_dir /mnt/disk/corpus-embedding \ --batch_size 32 \ --input_type passage \ --max_seq_length 128 \ --mesh_shape 1 -1 \ --lora /mnt/disk/retriever-mistral-jax/lora \ --scale_by_dim

But it does not save embedding at the output path. Please find the screenshot for the same below. @MXueguang @luyug can you help me in this?