search

mees / calvin

CALVIN - A benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks
http://calvin.cs.uni-freiburg.de
MIT License
366 stars 55 forks source link

Error in automatic annotation #39

Open jsw7460 opened 1 year ago

jsw7460 commented 1 year ago

Thank you for supporting great benchmark.

I'm trying to generate language annotation for unlabeled steps(frames).

I used automatic_lang_annotator_mp.py and adjust the eps Hyperparameter here to be 0.5.

I encountered IndexError:

2023-02-01_12-25-36

(Running the same code with eps=0.01 worked fine for me.)

I have two questions

  1. Should I decrease the epsilon to fix the error?
  2. How big should epsilon be to have an annotation on every frame?
lukashermann commented 1 year ago

@jsw7460 sorry for addressing this so late. I just reproduced the error, will get back to you when I find out what causes it. Did you investigate further in the meantime?

lukashermann commented 1 year ago

To your questions, yes a lower eps will prevent it from happening. I successfully annotated it with eps=0.1 Note that not every sampled window can be labeled with a language instruction (because the prerequisite is that a task is detected). I don't know what's the minimum epsilon for which all possible labels are created, but usually setting eps=1 should work (eps=0.5 of course too)