Multiple DMS replication tasks with terraform

[teseo24]

Hi

I am Martin from Peru. I want to create many replication tasks in DMS with terraform. All replication tasks have the same source and target.

I use terraform version 0.12 and provider.aws: version = "~> 2.42"

My issues are:

1. Cant put Cloudwatch log stream for each replication task
2. Cant reuse the same ARN for source and target for all these replication tasks. I got this message

Initializing modules... There are some problems with the configuration, described below.

The Terraform configuration must be valid before initialization so that Terraform can determine which modules and providers need to be installed.

Error: Duplicate variable declaration

on ../../modules/product/aws-dms-dev-xxx1.tf line 4: 4: variable "dms-replication-instance_arn" {}

A variable named "dms-replication-instance_arn" was already declared at ../../modules/product/aws-dms-dev-xxx1.tf:4,1-40. Variable names must be unique within a module.

I am using a module

main file

provider "aws" { region = "us-east-1" }

modules

module "product" {

source = "../../modules/product"

}

variables file

variable "dms-replication-instance_arn" { default = "arn:aws:dms:us-east-1:XXXXXXXX:rep:C4BXXXXXXXXYT2BNFN64" }

variable "dms_replication_source_endpoint_arn" {
  default = "arn:aws:dms:us-east-1:XXXXXX:endpoint:3WOR4GXXXXXXXOCHTN7Y"
} 

variable "dms_replication_target_endpoint_arn" {
  default = "arn:aws:dms:us-east-1:XXXXXXXX:endpoint:LZ7O4AXXXXXDIZSXICXXA"
} 

task 1 file

Create a task for admindb

variable "dms-replication-instance_arn" {}

variable "dms_replication_source_endpoint_arn" {}

variable "dms_replication_target_endpoint_arn" {}

variable "dms_replication_task_CloudWatchLogGroup" {}

variable "dms_replication_task_CloudWatchLogStream_affiliatedb" {}

resource "aws_dms_replication_task" "replication_admindb" {
migration_type = "full-load-and-cdc" replication_instance_arn = var.dms-replication-instance_arn replication_task_id = "dev-dms-admindb" source_endpoint_arn = var.dms_replication_source_endpoint_arn target_endpoint_arn = var.dms_replication_target_endpoint_arn replication_task_settings = jsonencode(
{ ChangeProcessingDdlHandlingPolicy = { HandleSourceTableAltered = true HandleSourceTableDropped = true HandleSourceTableTruncated = true } ChangeProcessingTuning = { BatchApplyMemoryLimit = 500 BatchApplyPreserveTransaction = true BatchApplyTimeoutMax = 30 BatchApplyTimeoutMin = 1 BatchSplitSize = 0 CommitTimeout = 1 MemoryKeepTime = 60 MemoryLimitTotal = 1024 MinTransactionSize = 1000 StatementCacheSize = 50 } CharacterSetSettings = null ControlTablesSettings = { ControlSchema = "" HistoryTableEnabled = false HistoryTimeslotInMinutes = 5 StatusTableEnabled = false SuspendedTablesTableEnabled = false historyTimeslotInMinutes = 5 } ErrorBehavior = { ApplyErrorDeletePolicy = "IGNORE_RECORD" ApplyErrorEscalationCount = 0 ApplyErrorEscalationPolicy = "LOG_ERROR" ApplyErrorFailOnTruncationDdl = false ApplyErrorInsertPolicy = "LOG_ERROR" ApplyErrorUpdatePolicy = "LOG_ERROR" DataErrorEscalationCount = 0 DataErrorEscalationPolicy = "SUSPEND_TABLE" DataErrorPolicy = "LOG_ERROR" DataTruncationErrorPolicy = "LOG_ERROR" FailOnNoTablesCaptured = false FailOnTransactionConsistencyBreached = false FullLoadIgnoreConflicts = true RecoverableErrorCount = -1 RecoverableErrorInterval = 5 RecoverableErrorThrottling = true RecoverableErrorThrottlingMax = 1800 TableErrorEscalationCount = 0 TableErrorEscalationPolicy = "STOP_TASK" TableErrorPolicy = "SUSPEND_TABLE" } FullLoadSettings = { CommitRate = 10000 CreatePkAfterFullLoad = false MaxFullLoadSubTasks = 8 StopTaskCachedChangesApplied = false StopTaskCachedChangesNotApplied = false TargetTablePrepMode = "DO_NOTHING" TransactionConsistencyTimeout = 600 } Logging = { EnableLogging = true

CloudWatchLogGroup = var.dms_replication_task_CloudWatchLogGroup

            #CloudWatchLogStream = var.dms_replication_task_CloudWatchLogStream_affiliatedb
            LogComponents = [
            {
                Id       = "SOURCE_UNLOAD"
                Severity = "LOGGER_SEVERITY_DEFAULT"
            },
            {
                Id       = "TARGET_LOAD"
                Severity = "LOGGER_SEVERITY_DEFAULT"
            },
            {
                Id       = "SOURCE_CAPTURE"
                Severity = "LOGGER_SEVERITY_DEFAULT"
            },
            {
                Id       = "TARGET_APPLY"
                Severity = "LOGGER_SEVERITY_DEFAULT"
            },
            {
                Id       = "TASK_MANAGER"
                Severity = "LOGGER_SEVERITY_DEFAULT"
            },
            ]
        }
        PostProcessingRules = null
        StreamBufferSettings = {
            CtrlStreamBufferSizeInMB = 5
            StreamBufferCount        = 3
            StreamBufferSizeInMB     = 8
        }
        TargetMetadata = {
            BatchApplyEnabled        = false
            FullLobMode              = false
            InlineLobMaxSize         = 0
            LimitedSizeLobMode       = false
            LoadMaxFileSize          = 0
            LobChunkSize             = 0
            LobMaxSize               = 0
            ParallelLoadBufferSize   = 0
            ParallelLoadThreads      = 0
            SupportLobs              = false
            TargetSchema             = ""
            TaskRecoveryTableEnabled = false
        }
    }
)
table_mappings = jsonencode(
    {
        rules = [
            {
                rule-id     = "1"
                rule-name   = "schema-admindb"
                rule-type   = "selection"
                rule-action = "include"
                object-locator = {
                    schema-name = "admindb"
                    table-name  = "%"
                }
                filters = []
            }
        ]    
    }
)
tags = {
    Name = "Task replication schema admindb"
}

}

task 2 file

Create a task for affiliatedb

variable "dms-replication-instance_arn" {}

variable "dms_replication_source_endpoint_arn" {}

variable "dms_replication_target_endpoint_arn" {}

variable "dms_replication_task_CloudWatchLogGroup" {}

variable "dms_replication_task_CloudWatchLogStream_affiliatedb" {}

resource "aws_dms_replication_task" "replication_affiliatedb" {
migration_type = "full-load-and-cdc" replication_instance_arn = var.dms-replication-instance_arn replication_task_id = "dev-dms-affiliatedb" source_endpoint_arn = var.dms_replication_source_endpoint_arn target_endpoint_arn = var.dms_replication_target_endpoint_arn replication_task_settings = jsonencode(
{ ChangeProcessingDdlHandlingPolicy = { HandleSourceTableAltered = true HandleSourceTableDropped = true HandleSourceTableTruncated = true } ChangeProcessingTuning = { BatchApplyMemoryLimit = 500 BatchApplyPreserveTransaction = true BatchApplyTimeoutMax = 30 BatchApplyTimeoutMin = 1 BatchSplitSize = 0 CommitTimeout = 1 MemoryKeepTime = 60 MemoryLimitTotal = 1024 MinTransactionSize = 1000 StatementCacheSize = 50 } CharacterSetSettings = null ControlTablesSettings = { ControlSchema = "" HistoryTableEnabled = false HistoryTimeslotInMinutes = 5 StatusTableEnabled = false SuspendedTablesTableEnabled = false historyTimeslotInMinutes = 5 } ErrorBehavior = { ApplyErrorDeletePolicy = "IGNORE_RECORD" ApplyErrorEscalationCount = 0 ApplyErrorEscalationPolicy = "LOG_ERROR" ApplyErrorFailOnTruncationDdl = false ApplyErrorInsertPolicy = "LOG_ERROR" ApplyErrorUpdatePolicy = "LOG_ERROR" DataErrorEscalationCount = 0 DataErrorEscalationPolicy = "SUSPEND_TABLE" DataErrorPolicy = "LOG_ERROR" DataTruncationErrorPolicy = "LOG_ERROR" FailOnNoTablesCaptured = false FailOnTransactionConsistencyBreached = false FullLoadIgnoreConflicts = true RecoverableErrorCount = -1 RecoverableErrorInterval = 5 RecoverableErrorThrottling = true RecoverableErrorThrottlingMax = 1800 TableErrorEscalationCount = 0 TableErrorEscalationPolicy = "STOP_TASK" TableErrorPolicy = "SUSPEND_TABLE" } FullLoadSettings = { CommitRate = 10000 CreatePkAfterFullLoad = false MaxFullLoadSubTasks = 8 StopTaskCachedChangesApplied = false StopTaskCachedChangesNotApplied = false TargetTablePrepMode = "DO_NOTHING" TransactionConsistencyTimeout = 600 } Logging = { EnableLogging = true

CloudWatchLogGroup = var.dms_replication_task_CloudWatchLogGroup

            #CloudWatchLogStream = var.dms_replication_task_CloudWatchLogStream_affiliatedb
            LogComponents = [
            {
                Id       = "SOURCE_UNLOAD"
                Severity = "LOGGER_SEVERITY_DEFAULT"
            },
            {
                Id       = "TARGET_LOAD"
                Severity = "LOGGER_SEVERITY_DEFAULT"
            },
            {
                Id       = "SOURCE_CAPTURE"
                Severity = "LOGGER_SEVERITY_DEFAULT"
            },
            {
                Id       = "TARGET_APPLY"
                Severity = "LOGGER_SEVERITY_DEFAULT"
            },
            {
                Id       = "TASK_MANAGER"
                Severity = "LOGGER_SEVERITY_DEFAULT"
            },
            ]
        }
        PostProcessingRules = null
        StreamBufferSettings = {
            CtrlStreamBufferSizeInMB = 5
            StreamBufferCount        = 3
            StreamBufferSizeInMB     = 8
        }
        TargetMetadata = {
            BatchApplyEnabled        = false
            FullLobMode              = false
            InlineLobMaxSize         = 0
            LimitedSizeLobMode       = false
            LoadMaxFileSize          = 0
            LobChunkSize             = 0
            LobMaxSize               = 0
            ParallelLoadBufferSize   = 0
            ParallelLoadThreads      = 0
            SupportLobs              = false
            TargetSchema             = ""
            TaskRecoveryTableEnabled = false
        }
    }
)
table_mappings = jsonencode(
    {
        rules = [
            {
                rule-id     = "1"
                rule-name   = "schema-affiliatedb"
                rule-type   = "selection"
                rule-action = "include"
                object-locator = {
                    schema-name = "affiliatedb"
                    table-name  = "%"
                }
                filters = []
            }
        ]    
    }
)
tags = {
    Name = "Task replication schema affiliatedb"
}

}

Thanks in advance

Martin Contreras

[SunriseLong]

If I'm understanding this correctly: All replication tasks have the same source and target.

Since all tasks have the same source and target why not create one replication tasks with all the replication rules in table_mappings

The syntax is here https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TableMapping.SelectionTransformation.Selections.html

[justinretzolk]

Hey @teseo24 👋 Thank you for taking the time to file this issue! Given the information provided above, I wanted to follow up here. Can you confirm whether you're still having troubles, or whether you were able to get past the blocker?

[byronmamamoney]

It's better to split the tasks out, else one tasks gets to big and if the replication breaks then everything would be impacted. Also perhaps even splitting the tasks out over different replication instances will ensure that the EC2 behind the scene is not over burdened with to many tasks. I'm currently stuck on the same thing. My setup is different though. I've implemented a DMS module and I'm using the "for_each" meta argument to iterate over a map object and pass the inputs into the module. If I get it working and I remember I'll post my findings here....

[github-actions[bot]]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!