S3 BucketLifecycleConfigurationV2RuleArgs Expiration is not deleted when set to nil

cooloncoolon commented 2 days ago

Describe what happened

we are implementing S3 Lifecycle policy for our S3 buckets and noticed that all lifecycle policy configuration works as expected except "Expiration" when 'nil' is passed to this field. We are expecting that the Expiration configuration is removed from the rule configuration but the preview shows no objects to be changed. At the same time the "Expiration" works when creating or updating(ex. changing number of days).

we have the same configuration for the "NoncurrentVersionExpiration" but in this case the configuration block works as expected - created, updated and removed.

moreover the "Expiration" configuration is removed if there are another rule changes at the same time. but it is removed silently, preview does not shows any diff for the "Expiration" configuration.

Sample program

type LifecyclePolicyRulesArgs struct {
    RuleIdSuffix                                 string
    RuleStatus                                   string
    ObjectsFilterOptions                         ObjectsFilterOptionsArgs
    ObjectsIncompleteMultipartUploadAbortOptions *ObjectsIncompleteMultipartUploadAbortOptionsArgs
    ObjectsTransitionRulesOptions                ObjectsTransitionRulesOptionsArgs
    ObjectsExpirationRuleOptions                 *ObjectsExpirationRuleOptionsArgs
    NonCurrentVersionsTransitionRulesOptions     NonCurrentVersionsTransitionRulesOptionsArgs
    NonCurrentVersionsExpirationRuleOptions      *NonCurrentVersionsExpirationRuleOptionsArgs
}

// S3 Bucket Lifecycle Rules configuration - Objects expiration
type ObjectsExpirationRuleOptionsArgs struct {
    AfterDays                  int
    AfterDate                  string
    IsExpireObjectDeleteMarker bool
}

// S3 Bucket Lifecycle Rules configuration - Noncurrent Versions expiration
type NonCurrentVersionsExpirationRuleOptionsArgs struct {
    AfterDays          int
    RetainNonCurVerQty string
}

func (args *BucketArgs) createS3LifecycleConfig(ctx *pulumi.Context) error {

    // Convert S3 Lyfecycle Rules configuration to s3.BucketLifecycleRule format
    var s3LifecyclePolicyRules s3.BucketLifecycleConfigurationV2RuleArray
    for _, lifecycleRules := range args.LifecyclePolicyOptions.Rules {

        // Parse and convert S3 Objects Expiration Rules configuration
        var s3ObjectsExpirations *s3.BucketLifecycleConfigurationV2RuleExpirationArgs
        if lifecycleRules.ObjectsExpirationRuleOptions != nil {
            s3ObjectsExpirations = &s3.BucketLifecycleConfigurationV2RuleExpirationArgs{
                Days:                      pulumi.Int(lifecycleRules.ObjectsExpirationRuleOptions.AfterDays),
                ExpiredObjectDeleteMarker: pulumi.Bool(lifecycleRules.ObjectsExpirationRuleOptions.IsExpireObjectDeleteMarker),
            }
        }

        // Build S3 Lifecycle Policy Rule
        s3LifecyclePolicyRules = append(s3LifecyclePolicyRules, s3.BucketLifecycleConfigurationV2RuleArgs{
            Status:                         pulumi.String(lifecycleRules.RuleStatus),
            Id:                             pulumi.String("s3-lifecycle-rule-" + args.BucketName + "_" + lifecycleRules.RuleIdSuffix),
            Filter:                         s3ObjectsFilters,
            AbortIncompleteMultipartUpload: s3ObjectsIncompleteMultipartUploadAbort,
            Transitions:                    s3ObjectsTransitions,
            NoncurrentVersionTransitions:   s3NonCurVerTransitions,
            NoncurrentVersionExpiration:    s3NonCurVerExpirations,
            Expiration:                     s3ObjectsExpirations,
        })
    }
    // Create S3 Lifecycle Policy Rule
    _, err := s3.NewBucketLifecycleConfigurationV2(ctx, "s3-lifecycle-policy-"+args.BucketName, &s3.BucketLifecycleConfigurationV2Args{
        Bucket: args.internal.Bucket.ID(),
        Rules:  s3LifecyclePolicyRules,
    })

    if err != nil {
        return err
    }
    return nil
}

Log output

No response

Affected Resource(s)

s3.NewBucketLifecycleConfigurationV2 Expiration

Output of `pulumi about`

CLI
Version 3.132.0 Go Version go1.23.1 Go Compiler gc

Plugins KIND NAME VERSION resource aws 5.24.0 language go 3.132.0 resource kubernetes 3.30.2

Host
OS darwin Version 14.6.1 Arch arm64

This project is written in go: executable='/opt/homebrew/bin/go' version='go version go1.23.1 darwin/arm64'

Dependencies: NAME VERSION github.com/lib/pq v1.10.9 github.com/pulumi/pulumi-aws/sdk/v5 v5.24.0 github.com/pulumi/pulumi-kubernetes/sdk/v3 v3.30.2 github.com/pulumi/pulumi/sdk/v3 v3.81.0 github.com/stretchr/testify v1.8.3

Additional context

lifecycle policy configuration declared as follows:

testing rule array

  lifecyclePolicyOptions:
    enabled: true
    rules:
    #rule №1
    #- ruleStatus: Enabled
      #ruleIdSuffix: "lifecycle_for_IncompleteMultipartUploadAbort"
      #objectsIncompleteMultipartUploadAbortOptions:
        #afterDays: 15
    #rule №2
    #- ruleStatus: Enabled
      #ruleIdSuffix: "lifecycle_for_ExpireObjectDeleteMarker"
      #objectsExpirationRuleOptions:
        #isExpireObjectDeleteMarker: true
    #rule №3
    #- ruleStatus: Enabled
      #ruleIdSuffix: "lifecycle_for_expire_tag_logs"
      #objectsFilterOptions:
        #tags:
          #rule: "log"
          #autoclean: "true"
      #objectsExpirationRuleOptions:
        #afterDays: 90
    #rule №4
    #- ruleStatus: Enabled
      #ruleIdSuffix: "lifecycle_for_transition"
      #objectsFilterOptions:
        #prefix: "log/"
        #sizeGreaterThan: 132000
        #sizeLessThan: 133000
      #objectsTransitionRulesOptions:
        #rules:
        #- afterDays: 30 
          #afterDate: "2025-01-01T00:00:00Z"
          #toStorageClass: "STANDARD_IA"
        #- afterDays: 60
          #afterDate: "2026-01-02T00:00:00Z"
          #toStorageClass: "GLACIER"
    #rule №5
    #- ruleStatus: Enabled
      #ruleIdSuffix: "lifecycle_for_noncurrentversions"
      #nonCurrentVersionsTransitionRulesOptions:
        #rules:
        #- afterDays: 30 
          #toStorageClass: "STANDARD_IA"
          #retainNonCurVerQtyAfterTransition: "7"
        #- afterDays: 60
          #toStorageClass: "GLACIER"
          #retainNonCurVerQtyAfterTransition: "7"
      #nonCurrentVersionsExpirationRuleOptions:
          #retainNonCurVerQty: "7"
          #afterDays: 90    
    #rule №6
    - ruleStatus: Enabled
      ruleIdSuffix: "lifecycle_for_transition"
      #objectsIncompleteMultipartUploadAbortOptions:
        #afterDays: 15
      #objectsFilterOptions:
        #prefix: "log/"
        #sizeGreaterThan: 132000
        #sizeLessThan: 133000
        #tags:
          #rule_1: "log_1"
          #autoclean_1: "true_1"
        #tag:
          #key: "log_1"
          #value: "true_1"
      objectsTransitionRulesOptions:
        rules:
        - afterDays: 30 
          #afterDate: "2025-01-02T00:00:00Z"
          toStorageClass: "STANDARD_IA"
        - afterDays: 60
          #afterDate: "2026-01-03T00:00:00Z"
          toStorageClass: "GLACIER"
      #objectsExpirationRuleOptions:
        #afterDays: 90
      #nonCurrentVersionsTransitionRulesOptions:
        #rules:
        #- afterDays: 30 
          #toStorageClass: "STANDARD_IA"
          #retainNonCurVerQtyAfterTransition: "7"
        #- afterDays: 60
          #toStorageClass: "GLACIER"
          #retainNonCurVerQtyAfterTransition: "7"
      #nonCurrentVersionsExpirationRuleOptions:
        #retainNonCurVerQty: "7"
        #afterDays: 90

     testing rule array

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

corymhall commented 1 day ago

@cooloncoolon it looks like from the about info you are on a previous major version v5. Can you try upgrading to v6 and see if you still have the same issue?

cooloncoolon commented 1 day ago

hi @corymhall , ill give a try v6 and get back with the results

pulumi / pulumi-aws