How do I train across multiple videos

[SamuelSchmidgall]

Hello,

I was wondering how I can train this model across multiple tasks with multiple videos for each task given the code structure. In the metaworld example, I see that there is a single video example with the path structure "datasets/metaworld/metaworld_dataset/assembly/corner/00"

Does this imply that I should have a similar structure for my dataset?: "datasets/MY_DATA/MY_DATA_dataset/TASK/??/VIDEO_NUM"

Will the dataset reader automatically infer task strings from this structure?

[kbkbowo]

TLDR:

1. Use "datasets/MY_DATA/metaworld_dataset/TASK/??/VIDEO_NUM"
2. The position "??" can be any non-empty string you want (without "/")
3. TASK can be "my-task" or "my task"

Hello!

In this case "metaworld_dataset" is an anchor-like thing for the reader to locate the dataset.

So you might want to do "datasets/MY_DATA/metaworld_dataset/TASK/??/VIDEO_NUM" for each of your videos

The position "??" is not used for training, so it can basically be any non-empty string you want (without "/"). It stands for camera names (The model in our metaworld experiments was trained with videos from multiple camera poses). This dataset reader was copied from that used in our experiments and "corner" was the name of one of the cameras we used.

As for the task string, the reader replaces the "-" in TASK with spaces " ", so TASK can be "my-task" or "my task". Both are valid.

Let me know if there's still anything unclear!

edit: The position "??" can be any non-empty string you want (without "/"), I feel anything you want was too ambiguous lol.

[SamuelSchmidgall]

Thank you very much!!!!!

[kbkbowo]

Hi, I realized there were constraints on the dataset from our experiments that might stop you from really using your custom dataset. I've updated the repo to remove these constraints.

You can either re-clone the repo or simply modify datasets.py yourself by commenting out a few lines. See the changes here