about download_data.sh

[fengjie215]

Hello, I am very interested in your paper "PROGrasp" presented at 2024 ICRA. While running the process, I found that the raw data could not be downloaded. Could you please update the links? download_data.sh Thank you!

[gicheonkang]

@fengjie215 Hi, Thank you for letting me know! I just updated the download link for the raw data. Please check the file download_data.sh.

[fengjie215]

Thank you for your response. Your work has been very inspiring to me. I would like to express my gratitude once again.

[fengjie215]

@fengjie215 Hi, Thank you for letting me know! I just updated the download link for the raw data. Please check the file download_data.sh.

Hello, I'm encountering an error while running the prograsp_eval.ipynb program and I'm not quite sure what's causing it. `--------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) Cell In[3], line 2 1 # init module ----> 2 prograsp = PRoGrasp(params) 3 gt = [] 4 pred = []

Cell In[2], line 37 33 #################################################### 34 # questioner build and load 35 #################################################### 36 tasks.register_task('questioner', questionerTask)
---> 37 self.q_models, self.q_cfg, self.q_task = checkpoint_utils.load_model_ensemble_and_task( 38 utils.split_paths(self.params['q_path']), 39 arg_overrides=overrides 40 ) 42 self.q_cfg.common.seed = self.params['seed'] 43 self.q_cfg.generation.beam = self.params['beam_size']

File ~/feng/prograsp/OFA/fairseq/fairseq/checkpoint_utils.py:457, in load_model_ensemble_and_task(filenames, arg_overrides, task, strict, suffix, num_shards, state) 452 model.load_state_dict( 453 consolidated_model_state, strict=strict, model_cfg=cfg.model 454 ) 455 else: ... ---> 80 pkl_file = cPickle.load(open(df_mode,'rb'), **(dict(encoding='latin1') if six.PY3 else {})) 81 self.ref_len = np.log(float(pkl_file['ref_len'])) 82 self.document_frequency = pkl_file['document_frequency']

FileNotFoundError: [Errno 2] No such file or directory: '/data/jhkim/GC/caption_data/cider_cached_tokens/coco-valid-words.p'`

[gicheonkang]

@fengjie215 Hi, Thank you for the comment! A simple answer for this comment is to modify the overrides with overrides={"bpe_dir":"utils/BPE", "eval_cider":False}

"eval_cider" is one of the configurations when training the model. It is nothing to do with the demonstration, so please set eval_cider to False!

[fengjie215]

I am extremely grateful for your response. The program is now running normally. I tested the weights that I trained myself, but the performance is poor. Is this because the training parameters for the pre-trained weights you provided are different?

[fengjie215]

The parameters I used for training were those specified within the three shell scripts located in the 'OFA/run_scripts/prograsp' directory.

[gicheonkang]

@fengjie215 No, we released the exact same script that we used for training the models. For every epoch, the model is tested on the validation data. Please check the validation score when you select the final model. If the score is too low, please set the maximum epoch up to 40!

[fengjie215]

Thank you for your timely and patient response. After troubleshooting, it seems the issue was due to my improper setup of the path for the "./checkpoints/ofa_large.pt" file. Once I correctly reconfigured the path, both the grounding and question modules performed as expected. However, I'm now encountering an error during the training of the answerer module (interestingly, this error did not arise previously when the path for ofa_large.pt was not set correctly).

Traceback (most recent call last): File "./OFA/train.py", line 552, in cli_main() File "./OFA/train.py", line 548, in cli_main distributed_utils.call_main(cfg, main) File "/home/feng/feng/prograsp/OFA/fairseq/fairseq/distributed/utils.py", line 389, in call_main main(cfg, kwargs) File "./OFA/train.py", line 199, in main valid_losses, should_stop = train(cfg, trainer, task, epoch_itr) File "/home/feng/anaconda3/envs/prograsp/lib/python3.8/contextlib.py", line 75, in inner return func(*args, *kwds) File "./OFA/train.py", line 332, in train valid_losses, should_stop = validate_and_save( File "./OFA/train.py", line 419, in validate_and_save valid_losses = validate(cfg, trainer, task, epoch_itr, valid_subsets) File "./OFA/train.py", line 489, in validate trainer.valid_step(sample) File "/home/feng/anaconda3/envs/prograsp/lib/python3.8/contextlib.py", line 75, in inner return func(args, kwds) File "/home/feng/feng/prograsp/OFA/trainer.py", line 1058, in valid_step _loss, sample_size, logging_output = self.task.valid_step( File "/home/feng/feng/prograsp/OFA/tasks/mm_tasks/answerer.py", line 166, in valid_step hyps, refs = self._inference(self.sequence_generator, sample, model) File "/home/feng/feng/prograsp/OFA/tasks/mm_tasks/answerer.py", line 261, in _inference decode_tokens = decode(gen_out[i][0]["tokens"]) IndexError: list index out of range

[gicheonkang]

@fengjie215 Hi, I ran the script to train the answerer module and couldn't find that kind of error. I think the error is related to the inference (i.e., generating text sequences). Can you check the configurations in the scripts for the answerer module?

[fengjie215]

Thank you for your response. I printed out 'i' and 'gen_out' in answerer.py, and here is the answerer.log. I've verified that the parameters match those you uploaded, with the exception of data=./data/preprocessed/train_answerer.tsv, ./data/preprocessed/val_common.tsv since there's no val_answerer.tsv available, thus I had to use val_common.tsv instead. Also, I'm running the program on a single RTX 3090 GPU, could this be the reason? (Additionally, if I want to create a similar dataset, how should I go about annotating it and converting it into a .stv format suitable for fine-tuning?) answerer.log

[gicheonkang]

Hi @fengjie215, Thank you for your interest!

1. I forgot to upload the validation data when training each module and just updated the download_data.sh file. Please download the validation data.
2. I also uploaded the file for preprocessing how you can convert the json files into the tsv files. Please see the preprocess.py