confusion about LDC datasets

arademaker commented 3 years ago

copied from https://github.com/UniversalDependencies/UD_English-EWT/issues/204

I would like to clarify the relation between https://catalog.ldc.upenn.edu/LDC2013T19 (OntoNotes 5.0) with https://catalog.ldc.upenn.edu/LDC2012T04 (2009 CoNLL Shared Task). The doc of LDC2012T04 says that the texts came from https://catalog.ldc.upenn.edu/LDC95T7. I didn't know that LDC95T7 is part of LDC2013T19 (OntoNotes). I just found ontonotes-release 5.0/data/files/data/english/annotations/nw/wsj/ with the same subdirectories of WSJ that LDC95T7/combined/wsj/!

So LDC95T7 is part of LDC2013T19! My bad! I didn't know that. I also didn't know about https://catalog.ldc.upenn.edu/LDC2015T13, it is a revised version of https://catalog.ldc.upenn.edu/LDC95T7 but sadly, there is no reference to this new version in the https://catalog.ldc.upenn.edu/LDC95T7 page.

Since I am projecting the SRL into UD annotations obtained from LDC95T7, and I have already done that for EWT and Ontonotes, I may already have the data I was trying to obtain from LDC2012T04. I suspect that probably the SRL annotations from LDC2012T04 should also differ from the OntoNotes annotations here, being the last probably the most recent one. Am I right? Maybe @MarthaSPalmer can also have something to add here and can confirm my understanding!

But in OntoNotes, we have 1728 files inside data/files/data/english/annotations/nw/wsj/ subdirectories and in LDC95T7 we have 2312 files. There are 584 missing files in OntoNotes, does anyone knows anything about it? There are the missing ones in OntoNotes:

  [1] "wsj/00/wsj_0006.conllu" "wsj/00/wsj_0013.conllu" "wsj/00/wsj_0015.conllu"
  [4] "wsj/00/wsj_0018.conllu" "wsj/00/wsj_0026.conllu" "wsj/00/wsj_0027.conllu"
  [7] "wsj/00/wsj_0032.conllu" "wsj/00/wsj_0048.conllu" "wsj/00/wsj_0057.conllu"
 [10] "wsj/00/wsj_0063.conllu" "wsj/00/wsj_0067.conllu" "wsj/00/wsj_0068.conllu"
 [13] "wsj/00/wsj_0073.conllu" "wsj/00/wsj_0080.conllu" "wsj/01/wsj_0106.conllu"
 [16] "wsj/01/wsj_0109.conllu" "wsj/01/wsj_0118.conllu" "wsj/01/wsj_0122.conllu"
 [19] "wsj/01/wsj_0124.conllu" "wsj/01/wsj_0125.conllu" "wsj/01/wsj_0127.conllu"
 [22] "wsj/01/wsj_0132.conllu" "wsj/01/wsj_0135.conllu" "wsj/01/wsj_0136.conllu"
 [25] "wsj/01/wsj_0137.conllu" "wsj/01/wsj_0144.conllu" "wsj/01/wsj_0149.conllu"
 [28] "wsj/01/wsj_0150.conllu" "wsj/01/wsj_0151.conllu" "wsj/01/wsj_0152.conllu"
 [31] "wsj/01/wsj_0153.conllu" "wsj/01/wsj_0157.conllu" "wsj/01/wsj_0158.conllu"
 [34] "wsj/01/wsj_0159.conllu" "wsj/01/wsj_0160.conllu" "wsj/01/wsj_0161.conllu"
 [37] "wsj/01/wsj_0162.conllu" "wsj/01/wsj_0165.conllu" "wsj/01/wsj_0166.conllu"
 [40] "wsj/01/wsj_0167.conllu" "wsj/01/wsj_0168.conllu" "wsj/01/wsj_0169.conllu"
 [43] "wsj/01/wsj_0171.conllu" "wsj/01/wsj_0172.conllu" "wsj/01/wsj_0173.conllu"
 [46] "wsj/01/wsj_0175.conllu" "wsj/01/wsj_0176.conllu" "wsj/01/wsj_0177.conllu"
 [49] "wsj/01/wsj_0178.conllu" "wsj/01/wsj_0179.conllu" "wsj/01/wsj_0184.conllu"
 [52] "wsj/01/wsj_0187.conllu" "wsj/01/wsj_0188.conllu" "wsj/01/wsj_0189.conllu"
 [55] "wsj/01/wsj_0194.conllu" "wsj/02/wsj_0224.conllu" "wsj/02/wsj_0231.conllu"
 [58] "wsj/02/wsj_0253.conllu" "wsj/02/wsj_0263.conllu" "wsj/02/wsj_0266.conllu"
 [61] "wsj/02/wsj_0271.conllu" "wsj/02/wsj_0279.conllu" "wsj/02/wsj_0292.conllu"
 [64] "wsj/02/wsj_0295.conllu" "wsj/03/wsj_0308.conllu" "wsj/03/wsj_0313.conllu"
 [67] "wsj/03/wsj_0316.conllu" "wsj/03/wsj_0317.conllu" "wsj/03/wsj_0321.conllu"
 [70] "wsj/03/wsj_0322.conllu" "wsj/03/wsj_0324.conllu" "wsj/03/wsj_0325.conllu"
 [73] "wsj/03/wsj_0327.conllu" "wsj/03/wsj_0329.conllu" "wsj/03/wsj_0331.conllu"
 [76] "wsj/03/wsj_0332.conllu" "wsj/03/wsj_0336.conllu" "wsj/03/wsj_0340.conllu"
 [79] "wsj/03/wsj_0344.conllu" "wsj/03/wsj_0345.conllu" "wsj/03/wsj_0346.conllu"
 [82] "wsj/03/wsj_0348.conllu" "wsj/03/wsj_0350.conllu" "wsj/03/wsj_0356.conllu"
 [85] "wsj/03/wsj_0361.conllu" "wsj/03/wsj_0376.conllu" "wsj/04/wsj_0402.conllu"
 [88] "wsj/04/wsj_0404.conllu" "wsj/04/wsj_0419.conllu" "wsj/04/wsj_0427.conllu"
 [91] "wsj/04/wsj_0430.conllu" "wsj/04/wsj_0438.conllu" "wsj/04/wsj_0448.conllu"
 [94] "wsj/04/wsj_0462.conllu" "wsj/04/wsj_0464.conllu" "wsj/04/wsj_0466.conllu"
 [97] "wsj/04/wsj_0467.conllu" "wsj/04/wsj_0471.conllu" "wsj/04/wsj_0472.conllu"
[100] "wsj/04/wsj_0474.conllu" "wsj/04/wsj_0482.conllu" "wsj/04/wsj_0483.conllu"
[103] "wsj/04/wsj_0487.conllu" "wsj/04/wsj_0491.conllu" "wsj/04/wsj_0493.conllu"
[106] "wsj/04/wsj_0497.conllu" "wsj/04/wsj_0498.conllu" "wsj/05/wsj_0500.conllu"
[109] "wsj/05/wsj_0504.conllu" "wsj/05/wsj_0505.conllu" "wsj/05/wsj_0520.conllu"
[112] "wsj/05/wsj_0527.conllu" "wsj/05/wsj_0533.conllu" "wsj/05/wsj_0534.conllu"
[115] "wsj/05/wsj_0541.conllu" "wsj/05/wsj_0542.conllu" "wsj/05/wsj_0551.conllu"
[118] "wsj/05/wsj_0555.conllu" "wsj/05/wsj_0557.conllu" "wsj/05/wsj_0558.conllu"
[121] "wsj/05/wsj_0568.conllu" "wsj/05/wsj_0570.conllu" "wsj/05/wsj_0575.conllu"
[124] "wsj/05/wsj_0583.conllu" "wsj/05/wsj_0584.conllu" "wsj/05/wsj_0585.conllu"
[127] "wsj/05/wsj_0586.conllu" "wsj/05/wsj_0588.conllu" "wsj/06/wsj_0610.conllu"
[130] "wsj/06/wsj_0612.conllu" "wsj/06/wsj_0632.conllu" "wsj/06/wsj_0637.conllu"
[133] "wsj/06/wsj_0650.conllu" "wsj/06/wsj_0660.conllu" "wsj/06/wsj_0661.conllu"
[136] "wsj/06/wsj_0662.conllu" "wsj/06/wsj_0667.conllu" "wsj/06/wsj_0670.conllu"
[139] "wsj/06/wsj_0674.conllu" "wsj/06/wsj_0675.conllu" "wsj/06/wsj_0679.conllu"
[142] "wsj/06/wsj_0681.conllu" "wsj/06/wsj_0685.conllu" "wsj/06/wsj_0695.conllu"
[145] "wsj/06/wsj_0697.conllu" "wsj/07/wsj_0704.conllu" "wsj/07/wsj_0706.conllu"
[148] "wsj/07/wsj_0709.conllu" "wsj/07/wsj_0713.conllu" "wsj/07/wsj_0736.conllu"
[151] "wsj/07/wsj_0745.conllu" "wsj/07/wsj_0751.conllu" "wsj/07/wsj_0752.conllu"
[154] "wsj/07/wsj_0760.conllu" "wsj/07/wsj_0762.conllu" "wsj/07/wsj_0768.conllu"
[157] "wsj/07/wsj_0778.conllu" "wsj/07/wsj_0781.conllu" "wsj/07/wsj_0786.conllu"
[160] "wsj/07/wsj_0791.conllu" "wsj/07/wsj_0798.conllu" "wsj/08/wsj_0805.conllu"
[163] "wsj/08/wsj_0806.conllu" "wsj/08/wsj_0810.conllu" "wsj/08/wsj_0811.conllu"
[166] "wsj/08/wsj_0815.conllu" "wsj/08/wsj_0816.conllu" "wsj/08/wsj_0818.conllu"
[169] "wsj/09/wsj_0904.conllu" "wsj/09/wsj_0906.conllu" "wsj/09/wsj_0907.conllu"
[172] "wsj/09/wsj_0918.conllu" "wsj/09/wsj_0923.conllu" "wsj/09/wsj_0924.conllu"
[175] "wsj/09/wsj_0927.conllu" "wsj/09/wsj_0928.conllu" "wsj/09/wsj_0938.conllu"
[178] "wsj/09/wsj_0950.conllu" "wsj/09/wsj_0953.conllu" "wsj/09/wsj_0973.conllu"
[181] "wsj/09/wsj_0981.conllu" "wsj/09/wsj_0991.conllu" "wsj/10/wsj_1001.conllu"
[184] "wsj/10/wsj_1002.conllu" "wsj/10/wsj_1003.conllu" "wsj/10/wsj_1006.conllu"
[187] "wsj/10/wsj_1008.conllu" "wsj/10/wsj_1010.conllu" "wsj/10/wsj_1011.conllu"
[190] "wsj/10/wsj_1013.conllu" "wsj/10/wsj_1014.conllu" "wsj/10/wsj_1015.conllu"
[193] "wsj/10/wsj_1020.conllu" "wsj/10/wsj_1021.conllu" "wsj/10/wsj_1022.conllu"
[196] "wsj/10/wsj_1025.conllu" "wsj/10/wsj_1028.conllu" "wsj/10/wsj_1029.conllu"
[199] "wsj/10/wsj_1031.conllu" "wsj/10/wsj_1033.conllu" "wsj/10/wsj_1035.conllu"
[202] "wsj/10/wsj_1036.conllu" "wsj/10/wsj_1037.conllu" "wsj/10/wsj_1038.conllu"
[205] "wsj/10/wsj_1039.conllu" "wsj/10/wsj_1040.conllu" "wsj/10/wsj_1042.conllu"
[208] "wsj/10/wsj_1048.conllu" "wsj/10/wsj_1050.conllu" "wsj/10/wsj_1051.conllu"
[211] "wsj/10/wsj_1052.conllu" "wsj/10/wsj_1059.conllu" "wsj/10/wsj_1069.conllu"
[214] "wsj/10/wsj_1071.conllu" "wsj/10/wsj_1073.conllu" "wsj/10/wsj_1074.conllu"
[217] "wsj/10/wsj_1079.conllu" "wsj/10/wsj_1097.conllu" "wsj/10/wsj_1098.conllu"
[220] "wsj/10/wsj_1099.conllu" "wsj/11/wsj_1105.conllu" "wsj/11/wsj_1109.conllu"
[223] "wsj/11/wsj_1111.conllu" "wsj/11/wsj_1124.conllu" "wsj/11/wsj_1136.conllu"
[226] "wsj/11/wsj_1142.conllu" "wsj/11/wsj_1144.conllu" "wsj/11/wsj_1154.conllu"
[229] "wsj/11/wsj_1155.conllu" "wsj/11/wsj_1156.conllu" "wsj/11/wsj_1157.conllu"
[232] "wsj/11/wsj_1161.conllu" "wsj/11/wsj_1164.conllu" "wsj/11/wsj_1165.conllu"
[235] "wsj/11/wsj_1171.conllu" "wsj/11/wsj_1173.conllu" "wsj/11/wsj_1176.conllu"
[238] "wsj/11/wsj_1177.conllu" "wsj/11/wsj_1179.conllu" "wsj/11/wsj_1182.conllu"
[241] "wsj/11/wsj_1185.conllu" "wsj/11/wsj_1186.conllu" "wsj/11/wsj_1191.conllu"
[244] "wsj/11/wsj_1192.conllu" "wsj/11/wsj_1193.conllu" "wsj/11/wsj_1195.conllu"
[247] "wsj/11/wsj_1196.conllu" "wsj/11/wsj_1197.conllu" "wsj/11/wsj_1199.conllu"
[250] "wsj/12/wsj_1201.conllu" "wsj/12/wsj_1202.conllu" "wsj/12/wsj_1204.conllu"
[253] "wsj/12/wsj_1205.conllu" "wsj/12/wsj_1207.conllu" "wsj/12/wsj_1208.conllu"
[256] "wsj/12/wsj_1209.conllu" "wsj/12/wsj_1210.conllu" "wsj/12/wsj_1217.conllu"
[259] "wsj/12/wsj_1221.conllu" "wsj/12/wsj_1223.conllu" "wsj/12/wsj_1225.conllu"
[262] "wsj/12/wsj_1226.conllu" "wsj/12/wsj_1227.conllu" "wsj/12/wsj_1231.conllu"
[265] "wsj/12/wsj_1247.conllu" "wsj/12/wsj_1248.conllu" "wsj/12/wsj_1257.conllu"
[268] "wsj/12/wsj_1275.conllu" "wsj/12/wsj_1276.conllu" "wsj/12/wsj_1280.conllu"
[271] "wsj/12/wsj_1281.conllu" "wsj/12/wsj_1283.conllu" "wsj/12/wsj_1285.conllu"
[274] "wsj/12/wsj_1289.conllu" "wsj/12/wsj_1295.conllu" "wsj/12/wsj_1297.conllu"
[277] "wsj/13/wsj_1303.conllu" "wsj/13/wsj_1304.conllu" "wsj/13/wsj_1305.conllu"
[280] "wsj/13/wsj_1309.conllu" "wsj/13/wsj_1311.conllu" "wsj/13/wsj_1314.conllu"
[283] "wsj/13/wsj_1319.conllu" "wsj/13/wsj_1325.conllu" "wsj/13/wsj_1329.conllu"
[286] "wsj/13/wsj_1330.conllu" "wsj/13/wsj_1332.conllu" "wsj/13/wsj_1333.conllu"
[289] "wsj/13/wsj_1334.conllu" "wsj/13/wsj_1336.conllu" "wsj/13/wsj_1338.conllu"
[292] "wsj/13/wsj_1340.conllu" "wsj/13/wsj_1344.conllu" "wsj/13/wsj_1345.conllu"
[295] "wsj/13/wsj_1346.conllu" "wsj/13/wsj_1348.conllu" "wsj/13/wsj_1354.conllu"
[298] "wsj/13/wsj_1355.conllu" "wsj/13/wsj_1356.conllu" "wsj/13/wsj_1358.conllu"
[301] "wsj/13/wsj_1360.conllu" "wsj/13/wsj_1361.conllu" "wsj/13/wsj_1364.conllu"
[304] "wsj/13/wsj_1371.conllu" "wsj/13/wsj_1372.conllu" "wsj/13/wsj_1374.conllu"
[307] "wsj/13/wsj_1394.conllu" "wsj/13/wsj_1399.conllu" "wsj/14/wsj_1401.conllu"
[310] "wsj/14/wsj_1410.conllu" "wsj/14/wsj_1411.conllu" "wsj/14/wsj_1419.conllu"
[313] "wsj/14/wsj_1426.conllu" "wsj/14/wsj_1439.conllu" "wsj/14/wsj_1440.conllu"
[316] "wsj/14/wsj_1441.conllu" "wsj/14/wsj_1442.conllu" "wsj/14/wsj_1445.conllu"
[319] "wsj/14/wsj_1449.conllu" "wsj/14/wsj_1451.conllu" "wsj/14/wsj_1456.conllu"
[322] "wsj/14/wsj_1457.conllu" "wsj/14/wsj_1462.conllu" "wsj/14/wsj_1463.conllu"
[325] "wsj/14/wsj_1467.conllu" "wsj/14/wsj_1468.conllu" "wsj/14/wsj_1479.conllu"
[328] "wsj/14/wsj_1480.conllu" "wsj/14/wsj_1481.conllu" "wsj/14/wsj_1484.conllu"
[331] "wsj/14/wsj_1485.conllu" "wsj/14/wsj_1487.conllu" "wsj/14/wsj_1488.conllu"
[334] "wsj/14/wsj_1490.conllu" "wsj/14/wsj_1491.conllu" "wsj/14/wsj_1492.conllu"
[337] "wsj/15/wsj_1501.conllu" "wsj/15/wsj_1507.conllu" "wsj/15/wsj_1511.conllu"
[340] "wsj/15/wsj_1516.conllu" "wsj/15/wsj_1525.conllu" "wsj/15/wsj_1528.conllu"
[343] "wsj/15/wsj_1531.conllu" "wsj/15/wsj_1538.conllu" "wsj/15/wsj_1539.conllu"
[346] "wsj/15/wsj_1540.conllu" "wsj/15/wsj_1541.conllu" "wsj/15/wsj_1543.conllu"
[349] "wsj/15/wsj_1546.conllu" "wsj/15/wsj_1548.conllu" "wsj/15/wsj_1552.conllu"
[352] "wsj/15/wsj_1562.conllu" "wsj/15/wsj_1575.conllu" "wsj/15/wsj_1578.conllu"
[355] "wsj/15/wsj_1594.conllu" "wsj/15/wsj_1596.conllu" "wsj/15/wsj_1597.conllu"
[358] "wsj/15/wsj_1599.conllu" "wsj/16/wsj_1606.conllu" "wsj/16/wsj_1609.conllu"
[361] "wsj/16/wsj_1610.conllu" "wsj/16/wsj_1614.conllu" "wsj/16/wsj_1619.conllu"
[364] "wsj/16/wsj_1621.conllu" "wsj/16/wsj_1625.conllu" "wsj/16/wsj_1626.conllu"
[367] "wsj/16/wsj_1629.conllu" "wsj/16/wsj_1633.conllu" "wsj/16/wsj_1634.conllu"
[370] "wsj/16/wsj_1642.conllu" "wsj/16/wsj_1651.conllu" "wsj/16/wsj_1652.conllu"
[373] "wsj/16/wsj_1656.conllu" "wsj/16/wsj_1658.conllu" "wsj/16/wsj_1660.conllu"
[376] "wsj/16/wsj_1661.conllu" "wsj/16/wsj_1663.conllu" "wsj/16/wsj_1664.conllu"
[379] "wsj/16/wsj_1665.conllu" "wsj/16/wsj_1666.conllu" "wsj/16/wsj_1668.conllu"
[382] "wsj/16/wsj_1670.conllu" "wsj/16/wsj_1672.conllu" "wsj/16/wsj_1674.conllu"
[385] "wsj/16/wsj_1680.conllu" "wsj/16/wsj_1683.conllu" "wsj/16/wsj_1693.conllu"
[388] "wsj/17/wsj_1702.conllu" "wsj/17/wsj_1703.conllu" "wsj/17/wsj_1706.conllu"
[391] "wsj/17/wsj_1708.conllu" "wsj/17/wsj_1709.conllu" "wsj/17/wsj_1710.conllu"
[394] "wsj/17/wsj_1713.conllu" "wsj/17/wsj_1715.conllu" "wsj/17/wsj_1719.conllu"
[397] "wsj/17/wsj_1723.conllu" "wsj/17/wsj_1733.conllu" "wsj/17/wsj_1735.conllu"
[400] "wsj/17/wsj_1736.conllu" "wsj/17/wsj_1739.conllu" "wsj/17/wsj_1740.conllu"
[403] "wsj/17/wsj_1741.conllu" "wsj/17/wsj_1746.conllu" "wsj/17/wsj_1749.conllu"
[406] "wsj/17/wsj_1752.conllu" "wsj/17/wsj_1756.conllu" "wsj/17/wsj_1757.conllu"
[409] "wsj/17/wsj_1765.conllu" "wsj/17/wsj_1768.conllu" "wsj/17/wsj_1769.conllu"
[412] "wsj/17/wsj_1773.conllu" "wsj/17/wsj_1774.conllu" "wsj/17/wsj_1779.conllu"
[415] "wsj/17/wsj_1781.conllu" "wsj/17/wsj_1783.conllu" "wsj/17/wsj_1784.conllu"
[418] "wsj/17/wsj_1787.conllu" "wsj/17/wsj_1791.conllu" "wsj/17/wsj_1792.conllu"
[421] "wsj/17/wsj_1796.conllu" "wsj/17/wsj_1799.conllu" "wsj/18/wsj_1802.conllu"
[424] "wsj/18/wsj_1808.conllu" "wsj/18/wsj_1812.conllu" "wsj/18/wsj_1819.conllu"
[427] "wsj/18/wsj_1827.conllu" "wsj/18/wsj_1828.conllu" "wsj/18/wsj_1832.conllu"
[430] "wsj/18/wsj_1836.conllu" "wsj/18/wsj_1837.conllu" "wsj/18/wsj_1853.conllu"
[433] "wsj/18/wsj_1858.conllu" "wsj/18/wsj_1859.conllu" "wsj/18/wsj_1860.conllu"
[436] "wsj/18/wsj_1861.conllu" "wsj/18/wsj_1865.conllu" "wsj/18/wsj_1867.conllu"
[439] "wsj/18/wsj_1879.conllu" "wsj/18/wsj_1880.conllu" "wsj/18/wsj_1881.conllu"
[442] "wsj/18/wsj_1883.conllu" "wsj/18/wsj_1884.conllu" "wsj/18/wsj_1885.conllu"
[445] "wsj/18/wsj_1888.conllu" "wsj/18/wsj_1889.conllu" "wsj/18/wsj_1890.conllu"
[448] "wsj/18/wsj_1892.conllu" "wsj/18/wsj_1893.conllu" "wsj/18/wsj_1897.conllu"
[451] "wsj/18/wsj_1898.conllu" "wsj/19/wsj_1900.conllu" "wsj/19/wsj_1905.conllu"
[454] "wsj/19/wsj_1907.conllu" "wsj/19/wsj_1909.conllu" "wsj/19/wsj_1910.conllu"
[457] "wsj/19/wsj_1911.conllu" "wsj/19/wsj_1913.conllu" "wsj/19/wsj_1917.conllu"
[460] "wsj/19/wsj_1919.conllu" "wsj/19/wsj_1926.conllu" "wsj/19/wsj_1930.conllu"
[463] "wsj/19/wsj_1938.conllu" "wsj/19/wsj_1944.conllu" "wsj/19/wsj_1950.conllu"
[466] "wsj/19/wsj_1957.conllu" "wsj/19/wsj_1966.conllu" "wsj/19/wsj_1973.conllu"
[469] "wsj/19/wsj_1975.conllu" "wsj/19/wsj_1976.conllu" "wsj/19/wsj_1977.conllu"
[472] "wsj/19/wsj_1987.conllu" "wsj/19/wsj_1991.conllu" "wsj/19/wsj_1992.conllu"
[475] "wsj/19/wsj_1998.conllu" "wsj/19/wsj_1999.conllu" "wsj/20/wsj_2003.conllu"
[478] "wsj/20/wsj_2004.conllu" "wsj/20/wsj_2007.conllu" "wsj/20/wsj_2008.conllu"
[481] "wsj/20/wsj_2014.conllu" "wsj/20/wsj_2019.conllu" "wsj/20/wsj_2022.conllu"
[484] "wsj/20/wsj_2026.conllu" "wsj/20/wsj_2039.conllu" "wsj/20/wsj_2042.conllu"
[487] "wsj/20/wsj_2056.conllu" "wsj/20/wsj_2057.conllu" "wsj/20/wsj_2060.conllu"
[490] "wsj/20/wsj_2061.conllu" "wsj/20/wsj_2062.conllu" "wsj/20/wsj_2064.conllu"
[493] "wsj/20/wsj_2068.conllu" "wsj/20/wsj_2077.conllu" "wsj/20/wsj_2083.conllu"
[496] "wsj/20/wsj_2086.conllu" "wsj/20/wsj_2088.conllu" "wsj/20/wsj_2091.conllu"
[499] "wsj/20/wsj_2093.conllu" "wsj/20/wsj_2095.conllu" "wsj/21/wsj_2101.conllu"
[502] "wsj/21/wsj_2103.conllu" "wsj/21/wsj_2104.conllu" "wsj/21/wsj_2105.conllu"
[505] "wsj/21/wsj_2107.conllu" "wsj/21/wsj_2113.conllu" "wsj/21/wsj_2123.conllu"
[508] "wsj/21/wsj_2127.conllu" "wsj/21/wsj_2133.conllu" "wsj/21/wsj_2138.conllu"
[511] "wsj/21/wsj_2142.conllu" "wsj/21/wsj_2145.conllu" "wsj/21/wsj_2146.conllu"
[514] "wsj/21/wsj_2148.conllu" "wsj/21/wsj_2155.conllu" "wsj/21/wsj_2156.conllu"
[517] "wsj/21/wsj_2163.conllu" "wsj/21/wsj_2165.conllu" "wsj/21/wsj_2172.conllu"
[520] "wsj/22/wsj_2203.conllu" "wsj/22/wsj_2209.conllu" "wsj/22/wsj_2210.conllu"
[523] "wsj/22/wsj_2211.conllu" "wsj/22/wsj_2212.conllu" "wsj/22/wsj_2213.conllu"
[526] "wsj/22/wsj_2216.conllu" "wsj/22/wsj_2218.conllu" "wsj/22/wsj_2219.conllu"
[529] "wsj/22/wsj_2235.conllu" "wsj/22/wsj_2236.conllu" "wsj/22/wsj_2244.conllu"
[532] "wsj/22/wsj_2245.conllu" "wsj/22/wsj_2246.conllu" "wsj/22/wsj_2248.conllu"
[535] "wsj/22/wsj_2249.conllu" "wsj/22/wsj_2251.conllu" "wsj/22/wsj_2256.conllu"
[538] "wsj/22/wsj_2259.conllu" "wsj/22/wsj_2262.conllu" "wsj/22/wsj_2266.conllu"
[541] "wsj/22/wsj_2267.conllu" "wsj/22/wsj_2268.conllu" "wsj/22/wsj_2271.conllu"
[544] "wsj/22/wsj_2272.conllu" "wsj/22/wsj_2273.conllu" "wsj/22/wsj_2277.conllu"
[547] "wsj/22/wsj_2282.conllu" "wsj/23/wsj_2300.conllu" "wsj/23/wsj_2307.conllu"
[550] "wsj/23/wsj_2308.conllu" "wsj/23/wsj_2310.conllu" "wsj/23/wsj_2314.conllu"
[553] "wsj/23/wsj_2317.conllu" "wsj/23/wsj_2322.conllu" "wsj/23/wsj_2327.conllu"
[556] "wsj/23/wsj_2332.conllu" "wsj/23/wsj_2333.conllu" "wsj/23/wsj_2337.conllu"
[559] "wsj/23/wsj_2339.conllu" "wsj/23/wsj_2341.conllu" "wsj/23/wsj_2342.conllu"
[562] "wsj/23/wsj_2345.conllu" "wsj/23/wsj_2359.conllu" "wsj/23/wsj_2360.conllu"
[565] "wsj/23/wsj_2364.conllu" "wsj/23/wsj_2365.conllu" "wsj/23/wsj_2379.conllu"
[568] "wsj/23/wsj_2385.conllu" "wsj/23/wsj_2386.conllu" "wsj/23/wsj_2387.conllu"
[571] "wsj/23/wsj_2391.conllu" "wsj/23/wsj_2396.conllu" "wsj/23/wsj_2398.conllu"
[574] "wsj/24/wsj_2415.conllu" "wsj/24/wsj_2416.conllu" "wsj/24/wsj_2419.conllu"
[577] "wsj/24/wsj_2420.conllu" "wsj/24/wsj_2421.conllu" "wsj/24/wsj_2433.conllu"
[580] "wsj/24/wsj_2438.conllu" "wsj/24/wsj_2442.conllu" "wsj/24/wsj_2443.conllu"
[583] "wsj/24/wsj_2450.conllu" "wsj/24/wsj_2453.conllu"

arademaker commented 3 years ago

Among those files, 83 files don't matter. They were not used in LDC2012T04 to SRL according to CoNLL-2009-ST/LDC2012T04/docs/English-README.TXT:

CoNLL2009-ST-English-trial.txt - the trial corpus. It contains the first 506 lines of the development file.
CoNLL2009-ST-English-train.txt - the training corpus. It matches sections 2 through 21 of Treebank. Note that a few sentences from the original Treebank were removed due to merging problems between the input corpora.
CoNLL2009-ST-English-development.txt - the development corpus. It matches section 24 of Treebank.

But what about the other 500 files?

  [1] "wsj/00/wsj_0006.conllu" "wsj/00/wsj_0013.conllu" "wsj/00/wsj_0015.conllu"
  [4] "wsj/00/wsj_0018.conllu" "wsj/00/wsj_0026.conllu" "wsj/00/wsj_0027.conllu"
  [7] "wsj/00/wsj_0032.conllu" "wsj/00/wsj_0048.conllu" "wsj/00/wsj_0057.conllu"
 [10] "wsj/00/wsj_0063.conllu" "wsj/00/wsj_0067.conllu" "wsj/00/wsj_0068.conllu"
 [13] "wsj/00/wsj_0073.conllu" "wsj/00/wsj_0080.conllu" "wsj/01/wsj_0106.conllu"
 [16] "wsj/01/wsj_0109.conllu" "wsj/01/wsj_0118.conllu" "wsj/01/wsj_0122.conllu"
 [19] "wsj/01/wsj_0124.conllu" "wsj/01/wsj_0125.conllu" "wsj/01/wsj_0127.conllu"
 [22] "wsj/01/wsj_0132.conllu" "wsj/01/wsj_0135.conllu" "wsj/01/wsj_0136.conllu"
 [25] "wsj/01/wsj_0137.conllu" "wsj/01/wsj_0144.conllu" "wsj/01/wsj_0149.conllu"
 [28] "wsj/01/wsj_0150.conllu" "wsj/01/wsj_0151.conllu" "wsj/01/wsj_0152.conllu"
 [31] "wsj/01/wsj_0153.conllu" "wsj/01/wsj_0157.conllu" "wsj/01/wsj_0158.conllu"
 [34] "wsj/01/wsj_0159.conllu" "wsj/01/wsj_0160.conllu" "wsj/01/wsj_0161.conllu"
 [37] "wsj/01/wsj_0162.conllu" "wsj/01/wsj_0165.conllu" "wsj/01/wsj_0166.conllu"
 [40] "wsj/01/wsj_0167.conllu" "wsj/01/wsj_0168.conllu" "wsj/01/wsj_0169.conllu"
 [43] "wsj/01/wsj_0171.conllu" "wsj/01/wsj_0172.conllu" "wsj/01/wsj_0173.conllu"
 [46] "wsj/01/wsj_0175.conllu" "wsj/01/wsj_0176.conllu" "wsj/01/wsj_0177.conllu"
 [49] "wsj/01/wsj_0178.conllu" "wsj/01/wsj_0179.conllu" "wsj/01/wsj_0184.conllu"
 [52] "wsj/01/wsj_0187.conllu" "wsj/01/wsj_0188.conllu" "wsj/01/wsj_0189.conllu"
 [55] "wsj/01/wsj_0194.conllu"
..
[520] "wsj/22/wsj_2203.conllu" "wsj/22/wsj_2209.conllu" "wsj/22/wsj_2210.conllu"
[523] "wsj/22/wsj_2211.conllu" "wsj/22/wsj_2212.conllu" "wsj/22/wsj_2213.conllu"
[526] "wsj/22/wsj_2216.conllu" "wsj/22/wsj_2218.conllu" "wsj/22/wsj_2219.conllu"
[529] "wsj/22/wsj_2235.conllu" "wsj/22/wsj_2236.conllu" "wsj/22/wsj_2244.conllu"
[532] "wsj/22/wsj_2245.conllu" "wsj/22/wsj_2246.conllu" "wsj/22/wsj_2248.conllu"
[535] "wsj/22/wsj_2249.conllu" "wsj/22/wsj_2251.conllu" "wsj/22/wsj_2256.conllu"
[538] "wsj/22/wsj_2259.conllu" "wsj/22/wsj_2262.conllu" "wsj/22/wsj_2266.conllu"
[541] "wsj/22/wsj_2267.conllu" "wsj/22/wsj_2268.conllu" "wsj/22/wsj_2271.conllu"
[544] "wsj/22/wsj_2272.conllu" "wsj/22/wsj_2273.conllu" "wsj/22/wsj_2277.conllu"
[547] "wsj/22/wsj_2282.conllu"

timjogorman commented 3 years ago

This is what I've heard referred to as the "financial subset" of WSJ. There was a bunch of files which were judged to be extremely repetitive formulaic financial text (largely a list of stocks going up and down) and those putting together OntoNotes decided to remove them from the release. So as far as I've been told, it was intentional that it was omitted from OntoNotes, even though some of them already had SRL annotations.

Updating these wasn't really in scope when we did the unification, so I don't think any unified form exists (it requires updating the old SRL data to match OntoNotes tokenization, and some manual judgments for the small set of frames that changed since PB1)

MarthaSPalmer commented 3 years ago

Most importantly, these files were not included in the treebank revision that was aimed at improving synchronization with PropBank. So at this point the original TB/PB annotation on those files is pretty archaic. I was personally very unhappy with that decision because those files also had WSD and I believe coref as well, so it seemed wasteful but it wasn’t up to me.

Martha

On Sep 2, 2021, at 7:46 AM, timjogorman @.***> wrote:

This is what I've heard referred to as the "financial subset" of WSJ. There was a bunch of files which were judged to be extremely repetitive formulaic financial text (largely a list of stocks going up and down) and those putting together OntoNotes decided to remove them from the release. So as far as I've been told, it was intentional that it was omitted from OntoNotes, even though some of them already had SRL annotations.

Updating these wasn't really in scope when we did the unification, so I don't think any unified form exists (it requires updating the old SRL data to match OntoNotes tokenization, and some manual judgments for the small set of frames that changed since PB1)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/propbank/propbank-release/issues/14#issuecomment-911695576, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABB327U56RRJS2USHITHPPLT755DFANCNFSM5CRBO7LQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

propbank / propbank-release

confusion about LDC datasets #14