Nuxeo Management System

[terrywbrady]

Project Goals

• 31 feeds are currently scheduled over a 12 hour period in a crontab
  • smooth out the processing schedule by using some form of a queue (zookeeper)
  • ensure that 2 instances of the same feed can never be run simultaneously (currently managed with a lock file)
  • enable a simple process to pause/unpause feed processing during software deployments or other operational exceptions
• Simplify operations
  • disassociate the UI rails server from nuxeo processing
  • disassociate UI hosts from nuxeo processing
  • eliminate the need for the private repo (or use puppet)
  • make nuxeo credentials available via SSM
• Optimize processing
  • utilize merritt batch manifests to simplify processing
  • optimize batch size for good performance
  • streamline end user notifications
• Future Enhancements
  • create a model that could be replicated for other feed types such as tind

Notes

• Mark, Eric and Terry met on 10/18 to discuss the effort
• When Mark is ready to move to his next project, he will work on a design document to define an optimal design for this part of the system

[mreyescdl]

Here are the current steps in creating a Nuxeo process

• Create an entry in create script and run to create feed specific script https://github.com/cdlib/mrt-dashboard-config/blob/main/atom/bin/create_atom_scripts_production.csh
• Manually run script and adjust throttling, if needed (See Fujimoto collection for example)
• After submission, schedule weekly with Puppet

[mreyescdl]

Meeting notes

• Create stand alone Ruby script to process an atom feed
• House this on the Batch server (currently a Rake task on UI worker)
• Ruby script will not submit to Merritt but will populate the Ingest ZK queue (possible? - How much processing does the Ingest queue handlers perform?)
• Input that drives process will be a list of feeds in YAML file. Is the feed enough data or do we need additional info?

Additional thoughts

• Populating ingest queue directly bypasses authentication. Safe?

Future work

• Populate queue with Ingest batches, not single objects.

[elopatin-uc3]

Consider approach of editing a yaml file in a repo as a means for establishing a new harvest into an existing (or new) Merritt collection.