nerc-project / operations

Issues related to the operation of the NERC OpenShift environment
2 stars 0 forks source link

Account for container images in billing and quotas #698

Open larsks opened 2 months ago

larsks commented 2 months ago

People are able to create container images in OpenShift via BuildConfig resources or by directly pushing images into the internal registry.

Container images can potentially consume a large chunk of storage. We should account for this in our billing and quotas. This will become particularly relevant as the amount of AI work on the clusters increases, since there are plans afoot to use container images as the mechanism for distributing AI models.

larsks commented 2 months ago

Here is the existing documentation on resource limits for images. It looks like we can limit:

But that's not the same thing as limiting the total amount of space consumed by images. I'll play with that a bit tomorrow to see if things work as expected.

hpdempsey commented 2 months ago

The decision about whether we want to bill for container image space is separate from investigating how much space we expect to need, how we can manage the space, and whether quotas are needed. It is possible that we choose only track the image space usage, rather than implementing all the mechanisms for billing and quotas, depending on the tradeoff between new development costs and the (very small) cost of storage.