search

rowanz / merlot

MERLOT: Multimodal Neural Script Knowledge Models
MIT License
223 stars 25 forks source link

[Question] Est. disk space to hold the pretraining dataset #12

Closed dxli94 closed 2 years ago

dxli94 commented 2 years ago

Hi,

Congrats on the impressive work. I was just wondering do you have a rough estimation about the disk quota required to host the YT-Temporal-180M dataset? Sorry if I missed this information in the manuscript.

Thanks.

rowanz commented 2 years ago

hi Dongxu! For YT-Temporal-180M, a rough estimate would be around 200TiB including the videos. That said, it might be possible to download the videos at a lower resolution or to omit downloading them with audio. In our followup MERLOT Reserve work, we scaled things to an even larger number of videos (20 million), which is around 600 TiB including audio.

dxli94 commented 2 years ago

hi Dongxu! For YT-Temporal-180M, a rough estimate would be around 200TiB including the videos. That said, it might be possible to download the videos at a lower resolution or to omit downloading them with audio. In our followup MERLOT Reserve work, we scaled things to an even larger number of videos (20 million), which is around 600 TiB including audio.

Thanks for the response and kind pointer. This helped a lot.