Develop new publishing system

[sumerjabri]

Duplicates

• [X] I have searched the existing issues

Is your feature request related to a problem? Please describe.

The existing publishing system needs a refresh; see tickets:

* https://github.com/craftercms/craftercms/issues/4293 * https://github.com/craftercms/craftercms/issues/5148 * https://github.com/craftercms/craftercms/issues/5244 * https://github.com/craftercms/craftercms/issues/5594 * https://github.com/craftercms/craftercms/issues/5832 * https://github.com/craftercms/craftercms/issues/5833 * https://github.com/craftercms/craftercms/issues/5921 * https://github.com/craftercms/craftercms/issues/5932 * https://github.com/craftercms/craftercms/issues/5950 * https://github.com/craftercms/craftercms/issues/5976 * https://github.com/craftercms/craftercms/issues/6063 * https://github.com/craftercms/craftercms/issues/6191 * https://github.com/craftercms/craftercms/issues/6410 * https://github.com/craftercms/craftercms/issues/5583 * https://github.com/craftercms/craftercms/issues/6618

Describe the solution you'd like

• [x] Remove old APIs from specs:
  • /api/2/workflow/packages/reject
  • /api/2/workflow/packages/approve
  • /api/2/workflow/packages/{id}
  • /api/1/services/api/1/publish/publish-items.json
  • /api/2/workflow/packages
• [x] Deprecate obsolete publish-related APIs
  • /api/2/workflow/request_publish
  • /api/2/workflow/publish
  • /api/1/services/api/1/deployment/bulk-golive.json
  • /api/1/services/api/1/publish/commits.json
• [ ] Refactor DB to support new publishing system
• [ ] Update existing APIs and services to work with the new DB
• [ ] Update duplicate sql queries to copy data from new tables
• [ ] Update delete site sql queries to clean up new tables
• [ ] Make published a bare repository
• [x] Update dependency calculation to work with the new DB schema
• [x] Remove dependency calculation from publisher queue processing (keep in package calculation)
  • User will indicate soft dependencies on package creation
  • The backend will add hard dependencies
• [ ] Provide a report for every item in the publishing queue
  • Log every item as DEBUG
  • Keep individual and package level status in DB
• [ ] Create a single commit per publishing package
• [ ] Make Publishing and Pulling from remote mutually exclusive
• [ ] Update AwsUtils.copyObjects to support error and progress report
• [ ] Add events to notify UI about publishing status/progress Per the design document, add the events:
  • Publishing started
  • Publishing progress %
  • Publishing successful
  • Publishing successful with issues
  • Publishing failed All publishing events will carry information about the publishing package (name, etc.)
• [ ] Add new Workflow events (submit, approve, reject, cancel)
  • [x] Submit/schedule
  • [x] Approve/direct publish
  • [ ] Reject
  • [ ] Cancel
• [x] Make Publish API reject empty packages
• [x] Track Publish All requests in the publishing queue, so the UI can display it in the dashboard/history
• [ ] ~Bulk publish "/" should be equivalent to Publish ALL~ The UI would warn the user if trying to bulk publish everything and suggest publish-all
• [ ] Publish All and Bulk Publish should clear the publish queue
  • Publish all or publish / should mark all packages as published if already in the queue
• [x] Event-based publishing
  • Use events to trigger publishing
  • Trigger a request publish event for each site on startup (on 'stand-alone' instances or cluster primary nodes)
  • Run a clock every minute for scheduled items (if possible, use a single clock for all sites). Clock triggers the request publish event if needed
  • Add lock to prevent multiple packages from being published simultaneously for the same site
  • Events are not processed when the publisher is stopped
• [x] Add option to include soft dependencies in bulk publish
  • Add new ~Tweak the current~ API to accept folder paths and retrieve their children. Once user review, request publish of path + soft deps
• [x] Include a dependency scan in Publish by Commit ID operations
  • Add a new ~Tweak the current~ API to accept a commit ID and retrieve affected files. Once user review, request publish of paths + soft deps
• [ ] Delayed publishing of renamed content can lead to content deletion in published repo
  • [x] Create a item_target table to hold the old_path by target
  • [ ] Update write/rename/move/cut-paste operations to update item_target records
  • [ ] Update publisher to consider new item_target table
• [ ] When trying to publish an item and it fails, the entire package must be rolled back (including blobs)
• [x] Clean up the database to Remove unused columns and references to them:
  • site.publishing_lock_owner
  • site.publishing_lock_heartbeat
• [ ] Sync-from-repo task should cancel packages containing affected ingested paths

Considerations

• User requests to publish a number of items
• The system computes the soft and hard deps as it does today
• The selected files/deps end up in a single package
• Workflow works as it does today except it operates against the whole package and doesn't allow for partial - approval/rejection, the whole package goes or doesn't
• Updating an item in a package alerts the user that they're about to remove the entire package from the publishing process and it would have to be resubmitted
• Publishing will create a single commit ID for the entire package, such that looking at the history of the published repo shows a single commit per package

High level tasks

• [x] Update API spec
• [ ] Refactor DB schema
• [ ] Refactor repository creation
• [ ] Refactor existing APIs
• [ ] Implement new APIs
• [ ] Implement new publish item processing
• [ ] Upgrade scripts for repositories
• [ ] Upgrade scripts for database