Discrepancy between OSTI Data Explorer and DataSpace records

[astrochun]

I noticed the following when running Scraper:

Pulled 177 records from OSTI.
countItems:  164
all_items:  164
Pulled 164 records from DSpace.
8 unpublished records were found in the PPPL dataspace community that have not been registered with OSTI.

The difference would suggest 13 new records; however only 8 were identified. This means that there might be 5 records in OSTI that is not included in DataSpace. We should investigate what those records are. 

[astrochun]

So upon investigation it seems that this problem is reproducible and comes from DataExplorer API.

The code is here: https://github.com/pulibrary/pdc-osti/blob/7a1a587fc3d601b339309c12dc9770d56af25d94/src/pdc_osti/scraper.py#L143-L158

After pulling the list, here are just a few checks:

len(existing_datasets)
>> 210

osti_ids = [i['osti_id'] for i in existing_datasets]
len(set(osti_ids))
>> 190

In short, there are 20 records duplicated.

Here's a sorted list:

1354837, 1356364, 1366453, 1366462, 1366473, 1366473, 1366479, 1366479, 1366480, 1366495, 1366498, 1366719, 1366720, 1366720, 1366721, 1366722, 1366742, 1366746, 1366750, 1366751, 1366752, 1366755, 1366984, 1366984, 1366985, 1366985, 1366987, 1367078, 1367081, 1367081, 1367082, 1367156, 1367157, 1367163, 1367164, 1367164, 1367166, 1367167, 1367170, 1367547, 1367548, 1367550, 1367550, 1367551, 1367551, 1367552, 1367554, 1367555, 1367557, 1367557, 1367618, 1367870, 1367870, 1367873, 1367878, 1367878, 1367890, 1561996, 1561998, 1561999, 1562000, 1562001, 1562006, 1562007, 1562008, 1562009, 1562010, 1562012, 1562013, 1562019, 1562020, 1562021, 1562022, 1562023, 1562024, 1562025, 1562025, 1562027, 1562028, 1562031, 1562032, 1562033, 1562038, 1562041, 1562041, 1562042, 1562044, 1562044, 1562045, 1562046, 1562048, 1562048, 1562052, 1562053, 1562053, 1562054, 1562055, 1562055, 1562056, 1562056, 1562057, 1562057, 1562061, 1562064, 1562065, 1562067, 1562069, 1562070, 1562073, 1562074, 1562076, 1562077, 1562080, 1562082, 1562083, 1562086, 1562088, 1562089, 1562090, 1562092, 1562094, 1562095, 1562096, 1562097, 1562098, 1562099, 1562100, 1562101, 1562102, 1562103, 1562104, 1562105, 1562106, 1562107, 1608254, 1608293, 1608300, 1608302, 1608306, 1608312, 1608313, 1661166, 1661167, 1661168, 1661169, 1661170, 1661171, 1661172, 1814932, 1814933, 1814934, 1814935, 1814936, 1814937, 1814938, 1814939, 1814940, 1814941, 1814942, 1814943, 1814944, 1814945, 1814946, 1814947, 1814948, 1814949, 1814950, 1814951, 1814952, 1814953, 1814954, 1814955, 1814956, 1814957, 1814958, 1814959, 1814960, 1888253, 1888254, 1888255, 1888256, 1888257, 1888258, 1888259, 1888260, 1888261, 1888262, 1888263, 1888264, 1888265, 1888266, 1888267, 1888268, 1888269, 1888270, 1888271, 1888272, 1888273, 1888274, 1888275, 1888276, 1888277, 1888278, 1888279, 1888280, 1888281, 1891332, 1897992, 1897994, 1897995