Track down and disable containers still running from Projects that are archived

[joachimweyl]

Motivation

We have a few projects that are archived but the containers are still running. This means we end up with a project in the invoice with usage but no PI associated with it. One of the projects is using at least 2 GPUs.

Completion Criteria

no remove all containers associated with projects that are archived.

Description

• [x] review list of projects with no PI, confirm they are archived
• [x] if not then find out why they have no PI
• [x] stop containers
• [x] review all archived projects for still running containers

List of Projects with no PI

cost last month	Project ID
$8259.34	d53dfc39548a48edb024b6e58d8716e6
$29.02	df4b64c4366742ebb8b8f4b9966319f7
$19.34	0b406404755945ecbab2109d32bce661
$19.34	c3d11da447b1473bb77d74cc5b40cdd5
$9.67	05cca114ea014b8ea588016e9d2aff04

Completion dates

Desired - 2024-04-30 Required - TBD

[joachimweyl]

Can this be automated?

[joachimweyl]

@Milstein have these projects been manually shut down?

[Milstein]

Not found Projects:

| 0b406404755945ecbab2109d32bce661 | <Not Found>            |
| c3d11da447b1473bb77d74cc5b40cdd5 | <Not Found>            |

---

I found these projects detail:

| d53dfc39548a48edb024b6e58d8716e6 | nerc-admin                     |

--> I added this to our non_billed_project list.

| df4b64c4366742ebb8b8f4b9966319f7 | Testing NERC for BU-fff6a31    |

$ ospurge --verbose --purge-project "Testing NERC for BU-fff6a31" --os-cloud=nerc --> purged!

| 05cca114ea014b8ea588016e9d2aff04 | Testing NERC for BU-f778c4d        |

$ ospurge --verbose --purge-project "Testing NERC for BU-f778c4d" --os-cloud=nerc --> purged!

---

When we change allocation status from Revoked --> Active This creates a new Allocation Project ID and removes the older Project details!

| a7e1529b37464e328c46d84dd9365735 | Testing NERC for BU-d79ec8 |

$ ospurge --verbose --purge-project "Testing NERC for BU-d79ec8" --os-cloud=nerc --> purged!

| 5e47421927c443dabe934f4e6147ce67  | Testing NERC for BU-9719ac  | 

--> Just decommissioned with zero storage quota so will not get charged afterward.

[Milstein]

Manually cleaning up using os-purge except the nerc-admin that is our main internal project.

[joachimweyl]

@Milstein please close this issue once you clear those 4 items.

[Milstein]

@joachimweyl : can you see if in invoicing we can see more details about:

| 0b406404755945ecbab2109d32bce661 | <Not Found>            |
| c3d11da447b1473bb77d74cc5b40cdd5 | <Not Found>            |

[joachimweyl]

661 used 1488 OpenStack CPU SU in March, it was associated with my project Migrating from MOC to NERC-f04a7e9 before it was expired dd5 used 1488 OpenStack CPU SU in only March, not sure what this was.

[Milstein]

For your project "Migrating from MOC to NERC", I can see two allocations:

Migrating from MOC to NERC-fa886c1 | 5587ca2a20c74d219ad20381dfcf5d98
Migrating from MOC to NERC-feeace3 | 11317353d42e41d4be2ee82f5b0038ec

So this is strange behavior, how the Project ID has changed!

[joachimweyl]

What are next steps for closing this out?

[Milstein]

I am adding two more BU projects to this list:

Allocated Project ID    68543e14d86a42e59e6542b8c414f4c5
Allocated Project Name  MSoM Platform-1e370c

Allocated Project ID    3080a54ffbec45659fbc125fccbbac8e
Allocated Project Name  Fitness activity recognition from video-fc77646

[joachimweyl]

@Milstein were you able to clean up the two BU projects?

[joachimweyl]

Blocked until we can confirm they do not show up in the invoice.

[joachimweyl]

They showed up in May data but that is not unexpected as they were turned off during May, we will have to wait until June data to be sure they are cleared.

[joachimweyl]

all gone.