Regularly find and delete vestigial objects in .NET Engineering Services Subscriptions

[dotnet-bot]

While we have S360 alerts for various things, we need a way to know if unwanted objects exist. More often than not these are non-maliciously created objects that happen to have fallen through the cracks on the way to being deleted Any such automation would want to run infrequently (less than daily) to avoid API throttling.

Epic Goals:

• Provide functionality to know what we have in our Azure Subscriptions and draw attention to any unexpected objects
• Save money by deleting unused objects with ongoing cost, e.g. old VHDs, Image galleries, and unused "play" VMs
• Investigate origin of non-vestigial objects we don't expect to look for security or other issues
• Provide an extensibility point where additional checks of our objects can be stuck as need arises.

Epic completion criteria:

• [x] .NET Core Engineering team has no known leakage of objects that can continuously grow (images, storage size, etc)
• [ ] The DncEng team can run a tool and document all "interesting" (billable) objects in their subscriptions in a simple, reliable manner.
• [ ] Process documentation exists and has been handed off to vendor for periodic evaluation and reporting.

Subscriptions we need to monitor:

Definitely:

• HelixProd (68672ab8-de0c-40f1-8d1b-ffb20bd62c0f)
• dncenghelix-02 (a6ad62fb-177e-40ef-af51-5c342911ebf5)
• dncenghelix-01 (f8c1f536-2a9b-41ba-9868-811cc982bb25)
• dnceng-internaltooling (84a65c9a-787d-45da-b10a-3a1cefce8060)
• Dotnet Engineering services (a4fc5514-21a9-4296-bfaf-5c7ee7fa35d1)

Priority 0: Manual cleanup of known cruft: (Complete)

Take a first couple of passes at obviously dead objects in our Azure subscriptions and delete them.

Deliverable	Owner	Completion	Status
Clean up unused build pools and document the rest	@MattGal	22 March 2022	Complete
Clean image factory base images in helix-image-factory-westus2	@MattGal	29 March 2022	Complete
Deprecate Helix build pools and pool provider	@MattGal	29 March 2022	Complete
Automate cleanup of any Helix agent log files older than 30 days and offline agent files for non-heartbeating agents	@MattGal	15 April 2022	Complete
Automate cleanup of Helix offline blob storage blobs	@MattGal	15 April 2022	Complete

• [x] dotnet/core-eng#13422
• [x] dotnet/core-eng#14345
• [x] dotnet/core-eng#14348
• [x] dotnet/arcade#8663
• [x] dotnet/arcade#8696

Priority 1: Inventory Automation and process documentation

The known places above (log storage, VHD storage, blobs for cancellation / online/offline) have been already cleaned up, but another source for spending money unnecessarily is objects created within the subscription that continue to incur cost for things that we're not currently using

Deliverable	Owner	Completion	Status
Create inventory tool to flag unexpected objects found in DncEng subscriptions for review	TBD	TBD	Not Started
Create vendor process docs for DncEng inventory automation monitoring	TBD	TBD	Not Started

Recently Triaged Issues

All issues in this section should be triaged by the v-team into one of their business objectives or features.

• [x] https://github.com/dotnet/core-eng/issues/13422
• [x] https://github.com/dotnet/core-eng/issues/14345
• [x] https://github.com/dotnet/core-eng/issues/14348
• [x] https://github.com/dotnet/core-eng/issues/15159
• [x] https://github.com/dotnet/arcade/issues/8663

[MattGal]

I have merged the changes for helix offline blobs / cancellation blobs, and agent logs, and will monitor these until they roll out.

However, in the interest of focusing on having more than one person per epic, I am putting this one back onto the backlog after filling out some Epic table-y stuff for it.