[DATA] Queue Cleanup - DataCite

[adambuttrick]

Data Task Type

Queue Cleanup - DataCite

Describe the data task needed

As described in #723, we need to clean up the DataCite queue to address the successful status changed from "S" to "O", processing missed records from #394 #371.

Task list

• [ ] 1. Review records with status='S' created before "2023-06-29 05:08:25" (Thursday 5am)/seq=7993833
• [ ] 2. Update records with status='S' to status='O'
• [ ] 3. Process and delete over 7m records that had status change from 'S' to 'O'
• [ ] 4. Process and delete over 4m records with success (status='O') or ignore (status='I') status missed as a result of a bug in proc-cleanup-async-queues.py.

Expected outcome

DataCite queue cleaned of unnecessary records, including old 'S' status records and missed 'O' and 'I' status records.

[jsjiang]

DataCite queue status stats (2024-10-14):

"datacite": { "F": 1565, "O": 5192961, "S": 7992232 }

• Total records with status='S': 7,992,232

  select COUNT(seq) from ezidapp_datacitequeue where status='S';

• Latest record with status='S': seq= 7993832

  select max(seq) from ezidapp_datacitequeue where status='S';

• Timestamp: 2023-06-29 05:03:43

  select *, from_unixtime(enqueuetime), from_unixtime(submittime) from ezidapp_datacitequeue where seq=7993832;

• Verified that there are no records with status='S' since then

  select * , from_unixtime(enqueuetime), from_unixtime(submittime) from ezidapp_datacitequeue where seq>7993832 and status='S';