Ram81 / goat-bench

55 stars 6 forks source link

Unable to run ```SenseAct-NN Skill-Chain Baseline``` evaluation. #24

Open francescotaioli opened 1 month ago

francescotaioli commented 1 month ago

I'm unable to run the evaluation of the SenseAct-NN Skill-Chain Baseline (eval code at the end of the README)

Content of data/goat-assets/checkpoints/sense_act_nn_skill_chain is only language_nav_ckpt.pth. See here - HF.

When running the code, I got the following error:

File goat_bench/models/high_level_policy.py", line 124, in __init__
    self.ovon_policy = ovon_policy_cls.from_config(
AttributeError: 'NoneType' object has no attribute 'from_config'

Indeed, ovon_policy_cls should be

ovon_policy_cls = baseline_registry.get_policy(
            "PointNavResnetCLIPPolicy"
        )

but PointNavResnetCLIPPolicy it is not registered. It seems to be refactored to GOATPolicy, correct me if I'm wrong. However, If I replace PointNavResnetCLIPPolicy with GOATPolicy , several model keys seem to be missing. Indeed, I have the following error:

FileNotFoundError: [Errno 2] No such file or directory: 'data/new_checkpoints/ovon/ver/resnetclip_rgb_text/seed_1/ckpt.121.pth'
rtabrizi commented 1 month ago

same issue with me

francescotaioli commented 1 month ago

It also seems that language_nav_ckpt.pth is serialized with the old module name goat

rtabrizi commented 1 month ago

@francescotaioli have you had any luck?

francescotaioli commented 1 month ago

@rtabrizi Nope, unfortunately, I could not run it, and in the meantime, I was focusing on other things. Hope they will release the new checkpoints. @Ram81, do you have any updates on this, by any chance?

rtabrizi commented 1 month ago

@Ram81 are you able to upload the correct checkpoint?

Ram81 commented 1 month ago

@rtabrizi @francescotaioli you are right. The checkpoints are outdated. I'll push the fixed version by the end of this week. Sorry about that

francescotaioli commented 1 month ago

Thanks @Ram81 !

rtabrizi commented 1 month ago

Thank you!

francescotaioli commented 3 weeks ago

Hi @Ram81, sorry to bother you again. Do you have any updates on that?

NormanMises commented 3 weeks ago

want updates, too