Chunks generated from Gov2 seem to have lots of duplicate DocId values. For instance, index-273-100-150 (all 273 gov2 directories, retaining only documents with 100-150 terms) contains 3,870,096 documents, but only 123,544 unique DocIds. When ingesting the documents in Gov2 directory order (GX000, GX001, GX002, ...), the first duplicate is DocId=24 in GX001.
Chunks generated from Gov2 seem to have lots of duplicate DocId values. For instance, index-273-100-150 (all 273 gov2 directories, retaining only documents with 100-150 terms) contains 3,870,096 documents, but only 123,544 unique DocIds. When ingesting the documents in Gov2 directory order (GX000, GX001, GX002, ...), the first duplicate is DocId=24 in GX001.