Add Lifecycle rule to cleanup S3 bucket files in Venue

[galenatjpl]

The S3 bucket needs to be cleaned up in terms of older files. Start with a 7-day rule, per the Unity requirements. Make this configurable. Use AWS lifecycle rule, and bake this into the Go code that creates and configures the S3 bucket as part of the bootstrap procedure.

[jdrodjpl]

How do we envision this being configured?

[galenatjpl]

@jdrodjpl I was thinking that the https://github.com/unity-sds/unity-management-console/blob/main/backend/internal/aws/s3.go file would pass some sort of parameters/config to the CreateBucket call. This would need to be researched, but this is code that chatgpt gave me:

import (
    "context"
    "log"
    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/s3"
    "github.com/aws/aws-sdk-go-v2/service/s3/types"
)

func main() {
    cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithRegion("us-west-2"))
    if err != nil {
        log.Fatalf("unable to load SDK config, %v", err)
    }

    svc := s3.NewFromConfig(cfg)

    lifecycleConfiguration := &types.BucketLifecycleConfiguration{
        Rules: []types.LifecycleRule{
            {
                Status: types.ExpirationStatusEnabled,
                Filter: &types.LifecycleRuleFilter{
                    Prefix: aws.String(""),
                },
                Transitions: []types.Transition{
                    {
                        Days:         30,
                        StorageClass: types.TransitionStorageClassGlacier,
                    },
                },
                Expiration: &types.LifecycleExpiration{
                    Days: 365,
                },
                ID: aws.String("MyRule"),
            },
        },
    }

    _, err = svc.PutBucketLifecycleConfiguration(context.TODO(), &s3.PutBucketLifecycleConfigurationInput{
        Bucket:                 aws.String("your-bucket-name"),
        LifecycleConfiguration: lifecycleConfiguration,
    })
    if err != nil {
        log.Fatalf("unable to put bucket lifecycle configuration, %v", err)
    }

    log.Println("Successfully applied lifecycle configuration to the bucket")
}

[galenatjpl]

that's example code, but you get the idea..

[jdrodjpl]

Got it. I guess I was also asking -- is this going to be just a hard-coded value, or is there some kind of on-bootstrap configuration parameter we want to introduce to set this lifecycle rule time?

[galenatjpl]

@jdrodjpl Yeah, perhaps we can throw it as a parameter to the run.sh / deploy.sh script pathway, that gets passed to the MC in this section? https://github.com/unity-sds/unity-cs-infra/blob/8d232401a30e5edaa07a4fb92c4821c908e12d10/nightly_tests/deploy.sh#L67-L83 Then it would end up being a variable accessible by the MC, on the same level as project, venue, etc..

[galenatjpl]

This works now with new deployments.