digdag can't delete project

[frsyuki]

Deleting a project should do this:

• make the project invisible by default.
• all running session attempts associated to the project are canceled, or impossible to delete a project if there are running session attempts.
• keeps session records and task logs so that we can check past information.

Implementation limitations are:

• deleting sessions associated to the project may take long time. Therefore, it needs to be divided into multiple DB transactions scheduled by a background thread.
• it needs a long transaction to check whether there is a running session attempt or not in a project.

[frsyuki]

Digdag should have a background procedure that deletes old sessions periodically. it's not implemented but expected by design.

[frsyuki]

implementation idea 1:

• add state_flags and obsolete_name to projects table.
• make projects.name nullable.
• when user deletes a project, digdag does following works in a single transaction:
  • copies name to obsolete_name.
  • set NULL to name.
  • set ProjectStateFlags.DELETED to state_flags.
  • delete all schedules associated to the project.
• adding a new revision should be rejected if state_flags is DELETED (at ProjectControl.insert* methods because having ProjectControl ensures that the record is locked with FOR UPDATE).

This architecture makes it easy to have UNIQUE INDEX on name column. Looking up a project by name doesn't return logically deleted projects but looking up by id returns one. This is a naturally expected behavior.

Future plan 1:

• add a background thread that does following work:
  • delete old sessions and session_attempts older than timestamp T at most N records
  • if number of deleted records is less than N, there're no sessions or session_attempts older than timestamp T. in this case, start deleting projects whose state_flags is DELETED and updated_at is older than T.
  • deleting project consists of following work:
  • delete task log files from storage
  • delete revision archive files from storage
  • delete workflow_configs using multiple transactions
  • delete workflow_definitions using multiple transactions
  • delete revisions using multiple transactions
  • delete project