search

allenai / spoc-robot-training

SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World
https://spoc-robot.github.io/
Other
58 stars 2 forks source link

evaluation using checkpoint #23

Closed rginjapan closed 1 week ago

rginjapan commented 3 months ago

Loading val: 200it [00:00, 106630.33it/s] WARNING:py.warnings::165: UserWarning: Split train does not exist at path /spoc-robot-training/objaverse_houses/train.jsonl.gz, won't be included.

WARNING:py.warnings::165: UserWarning: Split val does not exist at path /spoc-robot-training/objaverse_houses/val.jsonl.gz, won't be included.

WARNING:py.warnings::165: UserWarning: Split test does not exist at path /spoc-robot-training/objaverse_houses/test.jsonl.gz, won't be included.

out_dataset: DatasetDict = out"load_dataset" File "", line 170, in load_dataset ValueError: No splits found.

Error when I run

python -m training.offline.online_eval --shuffle --eval_subset minival --output_basedir /path/to/save/logs \ --test_augmentation --task_type TASK_TYPE \ --input_sensors raw_navigation_camera raw_manipulation_camera last_actions an_object_is_in_hand \ --house_set objaverse --wandb_logging False --num_workers NUM_WORKERS \ --gpu_devices LIST_OF_GPU_IDS --training_run_id TRAINING_ID --local_checkpoint_dir /path/to/local/checkpoints

KuoHaoZeng commented 3 months ago

Looks like you didn't download the objaverse_houses or the path to the downloaded objaverse_houses is not specified. Please refer this example for downloading the objaverse_houses and this one for assigning the path to the environment variable OBJAVERSE_HOUSES_BASE_DIR.

rginjapan commented 3 months ago

@KuoHaoZeng Thanks for your reply. However, I am sure I have downloaded the objaverse_house and assigned the path correctly. But still the same errors occur, plz give me some guidance. Thanks!

rginjapan commented 3 months ago

I have solved this issue by using bash scripts/evaluate_pretrained_model.sh I have some other questions:

  1. "Evaluating a Model (Online Evaluation)", what is the meaning of online evaluation compared with evaluation with your provided pre-trained model?
  2. "Creating Your Own Model", what does this model mean? robot model? network model?
  3. In the evaluation, there are some following erros occur: FAILED TO TELEPORT AGENT AFTER INITIALIZATION ... /spoc-robot-training/tasks/abstract_task_sampler.py:184: UserWarning: Initial teleport failed in 13859, attempting to fix this initial position. , what do these mean and how to solve it? Thanks in advance.
ehsanik commented 1 week ago
  1. Refers to the same process.
  2. Network Model.
  3. That is just a warning and can be ignored.