How to resume the dataset build process?

kpertsch / rlds_dataset_builder

An example RLDS dataset builder for X-embodiment dataset conversion.

MIT License

62 stars 97 forks source link

How to resume the dataset build process? #4

Open wuyukun-tong opened 3 months ago

wuyukun-tong commented 3 months ago

I had a little error in the middle of generating the data broke, due to the huge amount of data, tensorflow_dataset folder under the 70G of data has been generated (1.0.0.incompletePLFLVH) how to resume the process.

kpertsch commented 3 months ago

Hey, resuming is not straightforward unfortunately. I have implemented a hacky version of this in the past here: https://github.com/JonathanYang0127/r2d2_rlds_dataset_builder/blob/af87e4ea8a2d4bd1707734ed71e35f47454e5b76/r2_d2/r2_d2.py#L45 But I wouldn't recommend and instead just regenerate from the start.