Hi,
I followed the tutorial to download the pretrain data step by step. After i downloaded all the .gz files and unzip them, i found there are just 123GB and 58.6GB data for java and python respectively(as to json file number,there are 1020 and 102 respectively), which mismatches the statistics in paper(352GB and 224GB respectively).
Have i missed something?(I saw there is a de-duplicating step in that tutorial, which will reduce the data size, is it the reason?)
Hi, I followed the tutorial to download the pretrain data step by step. After i downloaded all the .gz files and unzip them, i found there are just 123GB and 58.6GB data for java and python respectively(as to json file number,there are 1020 and 102 respectively), which mismatches the statistics in paper(352GB and 224GB respectively). Have i missed something?(I saw there is a de-duplicating step in that tutorial, which will reduce the data size, is it the reason?)