search

mlcommons / peoples-speech

The People’s Speech Dataset
https://mlcommons.org/en/peoples-speech/
Apache License 2.0
96 stars 12 forks source link

Truncating keys to length 100 sometimes causes duplication #47

Open galv opened 3 years ago

galv commented 3 years ago

I found this by diffing utt2spk and spk2utt in kaldi:

Can I write tar files with no maximum file name length?

-Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_87.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_87.flac Talented\
_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_87.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_87.flac Talented_People_w\
ith_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_87.flac                                                                                                                             
-Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_88.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_88.flac Talented\
_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_88.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_88.flac Talented_People_w\
ith_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_88.flac                                                                                                                             
-Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_89.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_89.flac Talented\
_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_89.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_89.flac Talented_People_w\
ith_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_89.flac                                                                                                                             
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_0.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_0.flac            
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_0.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_0.flac            
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_1.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_1.flac            
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_1.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_1.flac            
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_2.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_2.flac            
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_2.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_2.flac            
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_3.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_3.flac            
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_3.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_3.flac            
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_4.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_4.flac            
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_4.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_4.flac            
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_5.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_5.flac            
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_5.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_5.flac            
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_6.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_6.flac            
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_6.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_6.flac            
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_7.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_7.flac            
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_7.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_7.flac            
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_8.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_8.flac            
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_8.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_8.flac            
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_9.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_9.flac            
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_9.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_9.flac            
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_10.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_10.flac          
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_10.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_10.flac          
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_11.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_11.flac          
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_11.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_11.flac          
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_12.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_12.flac          
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_12.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_12.flac          
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_13.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_13.flac          
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_13.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_13.flac          
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_14.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_14.flac          
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_14.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_14.flac          
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_15.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_15.flac          
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_15.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_15.flac          
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_16.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_16.flac          
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_16.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_16.flac          
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_17.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_17.flac          
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_17.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_17.flac          
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_18.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_18.flac          
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_18.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_18.flac          
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_19.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_19.flac          
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_19.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_19.flac          
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_20.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_20.flac          
+Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_20.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_20.flac
galv commented 3 years ago

Solution is to set longFileMode= LONGFILE_POSIX as described here: http://commons.apache.org/proper/commons-compress/tar.html