search

google-deepmind / open_x_embodiment

Apache License 2.0
774 stars 55 forks source link

how to set my own task string #2

Closed zhaockab closed 9 months ago

zhaockab commented 10 months ago

Hello, I'm glad you could share this dataset. I want to test the effect of the model locally, but I don't know how to set tasks. It seems that there is no explicit explanation of the tasks to be completed from the examples provided in Jupiter.

datonefaridze commented 10 months ago

@zhaockab Did you figure out how do feed text in there?

kpertsch commented 10 months ago

If you check out the RT-1-X inference colab you can set the natural_language_instruction and natural_language_embedding in the observation dictionary below:

image

The natural_language_embedding is actually the important part that is used by the model. This embedding is produced by encoding the language string with a pre-trained sentence embedding model. For an example how to compute this embedding, see here.

@quanvuong maybe we can add an example for explicitly encoding the task into the Colab instead of loading it from the offline data.

quanvuong commented 9 months ago

I added an example of how to obtain the language embedding to the colab in https://github.com/google-deepmind/open_x_embodiment/commit/1635197c47005351dc099fed2cf8a15423cae843