Deploy pqqateway.py, XSL and createmarc.py changes to production

[elopatin-uc3]

https://github.com/CDLUC3/uc3-etds/pull/5

Tasks

• [x] CDLSYS ticket to take snapshot of ETDs prod box on Thursday at 9:30AM
• [x] Ask Mark to pause crons on Thursday at 9:30AM
• [x] Merge stage changes to master and/or new production branch
• [x] Add EL address to recipient entries for campuses that receive records via email in campus.yml – needed for UCR (no actual marc deliveries), UCM, UCSC, UCSD and UCSF
• [x] Deploy to ETD production service host: 5/26 at 4PM
• [x] Run pqgateway 5/27 at 10AM
• [x] Runcreatemarc 5/27 at 10:15AM
• [x] Review .mrc files generated with campuses
• [x] If records OK, re-enable crons on 6/1/21
• [ ] Introduce all .unx files and capture ISBN number and title mismatches. <-- Moving to new ticket
• [ ] Correct titles and re-run. <-- Moving to new ticket

[elopatin-uc3]

pqgateway run complete:

etds@uc3-etdx2-prd:~/apps/uc3-etds/scripts$ ./pqgateway 
------------------------------------------------------------
Starting pqgateway at Thu May 27 10:01:19 PDT 2021
/usr/lib64/python2.7/urllib.py:1306: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  return ''.join(map(quoter, s))
Finished pqgateway at Thu May 27 10:10:20 PDT 2021

[elopatin-uc3]

createmarc run complete:

------------------------------------------------------------
Starting createmarc at Thu May 27 10:13:28 PDT 2021
Finished createmarc at Thu May 27 10:15:43 PDT 2021

[elopatin-uc3]

New errors in etds.log – but believe these zips were not deleted because they haven't been harvested to eScholarship. Once they are, createmarc.py can use the eSchol feeds to obtain eSchol URLs for their records. Then they will be removed:

2021-05-27 10:14:56,278 ERROR: File /apps/etds/apps/uc3-etds/zipfiles/20210527/19634.zip not deleted
2021-05-27 10:14:58,327 ERROR: File /apps/etds/apps/uc3-etds/zipfiles/20210527/19662.zip not deleted
2021-05-27 10:15:00,585 ERROR: File /apps/etds/apps/uc3-etds/zipfiles/20210527/19655.zip not deleted
2021-05-27 10:15:02,935 ERROR: File /apps/etds/apps/uc3-etds/zipfiles/20210527/19647.zip not deleted
2021-05-27 10:15:05,082 ERROR: File /apps/etds/apps/uc3-etds/zipfiles/20210527/19674.zip not deleted
2021-05-27 10:15:07,134 ERROR: File /apps/etds/apps/uc3-etds/zipfiles/20210527/19659.zip not deleted
2021-05-27 10:15:09,532 ERROR: File /apps/etds/apps/uc3-etds/zipfiles/20210527/19672.zip not deleted
2021-05-27 10:15:11,926 ERROR: File /apps/etds/apps/uc3-etds/zipfiles/20210527/19657.zip not deleted
2021-05-27 10:15:14,192 ERROR: File /apps/etds/apps/uc3-etds/zipfiles/20210527/19648.zip not deleted
2021-05-27 10:15:16,439 ERROR: File /apps/etds/apps/uc3-etds/zipfiles/20210527/19613.zip not deleted
2021-05-27 10:15:18,625 ERROR: File /apps/etds/apps/uc3-etds/zipfiles/20210527/19640.zip not deleted
2021-05-27 10:15:20,847 ERROR: File /apps/etds/apps/uc3-etds/zipfiles/20210527/19661.zip not deleted

[elopatin-uc3]

Summary after deploy and script runs on 5/27:

• No marc records sent.
• No ETD error emails sent.
• No fatal errors in pqgateway run (though the log was not augmented - assume this the case because the script was manually run)

I will reach out to campus contacts and let them know we'll resume regular script execution on Tuesday. Also, I presume new marc records will be generated for the most recent UCLA ingests (all errors above related to UCLA submissions) when eSchol links become available.

[elopatin-uc3]

UCLA has re-enabled their automated process for picking up new records from SFTP. I've also confirmed that new ProQuest links are being added to the .csv files delivered with (or without) records. These links can be seen in the most recent .csv files delivered to UCSF, UCLA, UCSD and UCSC.

Given these results, I'm going to move this ticket to Deployed/Done. And I will spin off a new ticket with the remaining two tasks that are related to processing .unx files from ProQuest.