search

pulp / pulpcore

Pulp 3 pulpcore package https://pypi.org/project/pulpcore/
GNU General Public License v2.0
301 stars 116 forks source link

[EPIC] Audit sync deletion calls in all viewsets #4374

Open mdellweg opened 1 year ago

mdellweg commented 1 year ago

Sometimes deleting an object cascades to deleting multiple thousands of objects. This can stall or even timeout the gunicorn worker. In such cases, pulp tasks should be used. However switching to asynchronous deletion is changing the api slightly.

What we identified so far is that Publications, Exports and Uploads may be problematic.

One Idea that would help and make the codebase cleaner is to turn Upload chunks into Artifacts.

ipanova commented 1 year ago

Also Importer is syncronously removed, we should check it out too

mdellweg commented 1 year ago

To clarify: The number of upload chunks related to an upload is probably not problematic, but the fact that each chunk needs to delete a file from storage is. Instead just producing orphaned artifacts would make thinks way easier.

ggainey commented 1 year ago

RE import/export - deleting an Importer/Exporter removes its associated Import/Export objects, which have lists of repo-versions-exported or publications-exported. This is a "small" number of entities, making the chance of the delete-timeout-problem very small.

I think the more important consideration is, "what happens if you delete an Importer/Exporter while it's in use. If the delete was async, the user would be protected by the standard reserved-resources processing. For the moment, I'd propose this is a "don't do that" problem, rather than changing the existing API.