[Core feature] Enable project deletions

[seunggs]

Motivation: Why do you think this is important?

Currently, you can only archive a project, but it leaves a lot of hanging resources when you create unused/test projects.

Goal: What should the final outcome look like, ideally?

There should be a flyte API endpoint for deleting a project and all related resources completely. This allows for safe and clean removal of unused resources from flyte cluster

Describe alternatives you've considered

Manually deleting resources seems unsafe since most users (including myself) do not completely understand flyte internals and which resources or database entries need to be manually removed.

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

• [X] Yes

Have you read the Code of Conduct?

• [X] Yes

[welcome[bot]]

Thank you for opening your first issue here! 🛠

[rahul-theorem]

Would love to have this capability for other entities (like workflows/executions/launch plans/etc) as well

[eapolinario]

Could we maybe get some more input from the community? we’re not quite sure how to prioritize this change since no one else has asked for this. This will have to be done very carefully due to:

1. this is the first time we’re giving explicit delete permissions to Flyte.
2. this code has to be written and vetted very carefully since we don’t want to leave admin in an inconsistent state.
3. depending on the size of the tables and affected rows, this may take the database down for some time.

Before releasing this we’ll need to do some benchmarking to see how this affects some of the larger users. We’ll have to make sure any future tables we add also get included in the deletion logic.

Could you explain more the use case here? For example, why is tombstoning the project insufficient? Are you seeing a performance hit? We've had big Flyte deployments that never had any perf issues, so I'm curious to learn more about it.

[flixr]

Just to throw in my 2 cents here: I also would have like to have this, mostly to completely remove the test projects/workflows again... Instead I completely wiped the whole flyte db/storage and started from scratch. Not sure how relevant this is once you actually get to prod though... Main usecase I can think of is to remove projects/tasks/workflows where someone (accidentally?) added secrets (as I did for testing on purpose) and they should be completely removed...

[andrusha]

Flyte also create k8s artifacts, eg namespace, related to the job given project. Removing them manually without cleaning up the table rows feels like it could accidentally break something.

[kumare3]

What if there are workflows that reference copies of workflows from deleted projects. The current idea of archiving is like tombstones. I do not know if delete is safe

[github-actions[bot]]

Hello 👋, this issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will engage on it to decide if it is still applicable. Thank you for your contribution and understanding! 🙏