search

enGMzizo / copy-dynamodb-table

Copy Dynamodb table to another in the same or different zone , It is 100% safe. Speed depends on your destination table user-defined write provisioned throughput
130 stars 39 forks source link

Add support for ondemand dynamodb tables #20

Closed shomeprasanjit closed 4 years ago

shomeprasanjit commented 5 years ago

Hey Guys, I am getting errors while trying to copy Ondemand tables but works fine with provisioned tables.

{ ValidationException: 2 validation errors detected: Value '0' at 'provisionedThroughput.writeCapacityUnits' failed to satisfy constraint: Member must have value greater than or equal to 1; Value '0' at 'provisionedThroughput.readCapacityUnits' failed to satisfy constraint: Member must have value greater than or equal to 1 at Request.extractError (/Users/prasanjit_shome/node_modules/aws-sdk/lib/protocol/json.js:51:27) at Request.callListeners (/Users/prasanjit_shome/node_modules/aws-sdk/lib/sequential_executor.js:106:20) at Request.emit (/Users/prasanjit_shome/node_modules/aws-sdk/lib/sequential_executor.js:78:10) at Request.emit (/Users/prasanjit_shome/node_modules/aws-sdk/lib/request.js:683:14) at Request.transition (/Users/prasanjit_shome/node_modules/aws-sdk/lib/request.js:22:10) at AcceptorStateMachine.runTo (/Users/prasanjit_shome/node_modules/aws-sdk/lib/state_machine.js:14:12) at /Users/prasanjit_shome/node_modules/aws-sdk/lib/state_machine.js:26:10 at Request. (/Users/prasanjit_shome/node_modules/aws-sdk/lib/request.js:38:9) at Request. (/Users/prasanjit_shome/node_modules/aws-sdk/lib/request.js:685:12) at Request.callListeners (/Users/prasanjit_shome/node_modules/aws-sdk/lib/sequential_executor.js:116:18) message: '2 validation errors detected: Value \'0\' at \'provisionedThroughput.writeCapacityUnits\' failed to satisfy constraint: Member must have value greater than or equal to 1; Value \'0\' at \'provisionedThroughput.readCapacityUnits\' failed to satisfy constraint: Member must have value greater than or equal to 1', code: 'ValidationException', time: 2019-04-25T19:54:24.873Z, requestId: 'L9MVKUQ57HSNAU189FOUGCIJDBVV4KQNSO5AEMVJF66Q9ASUAAJG', statusCode: 400, retryable: false, retryDelay: 32.77870743752317 } { Table: { AttributeDefinitions: [ [Object], [Object] ], TableName: 'XXXXXXXXXXXXXXXXX, KeySchema: [ [Object], [Object] ], ProvisionedThroughput: { ReadCapacityUnits: 0, WriteCapacityUnits: 0 }, StreamSpecification: { StreamEnabled: true, StreamViewType: 'NEW_IMAGE' }, SSESpecification: { Enabled: true } } }

najimou commented 5 years ago

a workaround is to set a capacity and if needed select onDemand from console once it created. function clearTableSchema(table) { .... provisionedThroughput(table); return table }

WHERE :

function provisionedThroughput(table){

console.log('provisionedThroughput');
const defaultValue=5;

if (table.ProvisionedThroughput.ReadCapacityUnits === 0) {
    table.ProvisionedThroughput.ReadCapacityUnits = defaultValue;
}

if (table.ProvisionedThroughput.WriteCapacityUnits === 0) {
    table.ProvisionedThroughput.WriteCapacityUnits = defaultValue;
}

for(var index= 0; index < table.GlobalSecondaryIndexes.length; index++)
{
    if (table.GlobalSecondaryIndexes[index].ProvisionedThroughput.ReadCapacityUnits === 0) {
        table.GlobalSecondaryIndexes[index].ProvisionedThroughput.ReadCapacityUnits = defaultValue;
    }

    if (table.GlobalSecondaryIndexes[index].ProvisionedThroughput.WriteCapacityUnits === 0) {
        table.GlobalSecondaryIndexes[index].ProvisionedThroughput.WriteCapacityUnits = defaultValue;
    }

}

}

shomeprasanjit commented 5 years ago

was still getting an error message as shown below.

/Users/prasanjit_shome/node_modules/aws-sdk/lib/request.js:31 throw err; ^

TypeError: Cannot read property 'length' of undefined at clearTableSchema (/Users/prasanjit_shome/prog/common_cloud/copy-dynamodb-table/index.js:157:58) at Response. (/Users/prasanjit_shome/prog/common_cloud/copy-dynamodb-table/index.js:53:48) at Request. (/Users/prasanjit_shome/node_modules/aws-sdk/lib/request.js:364:18) at Request.callListeners (/Users/prasanjit_shome/node_modules/aws-sdk/lib/sequential_executor.js:106:20) at Request.emit (/Users/prasanjit_shome/node_modules/aws-sdk/lib/sequential_executor.js:78:10) at Request.emit (/Users/prasanjit_shome/node_modules/aws-sdk/lib/request.js:683:14) at Request.transition (/Users/prasanjit_shome/node_modules/aws-sdk/lib/request.js:22:10) at AcceptorStateMachine.runTo (/Users/prasanjit_shome/node_modules/aws-sdk/lib/state_machine.js:14:12) at /Users/prasanjit_shome/node_modules/aws-sdk/lib/state_machine.js:26:10 at Request. (/Users/prasanjit_shome/node_modules/aws-sdk/lib/request.js:38:9)

BUT got it working by adding if statement in "provisionedThroughput" function.

if(table.GlobalSecondaryIndexes && table.GlobalSecondaryIndexes.length > 0){ for(var index= 0; index < table.GlobalSecondaryIndexes.length; index++) { if (table.GlobalSecondaryIndexes[index].ProvisionedThroughput.ReadCapacityUnits === 0) { table.GlobalSecondaryIndexes[index].ProvisionedThroughput.ReadCapacityUnits = defaultValue; } if (table.GlobalSecondaryIndexes[index].ProvisionedThroughput.WriteCapacityUnits === 0) { table.GlobalSecondaryIndexes[index].ProvisionedThroughput.WriteCapacityUnits = defaultValue; } } }

Note about OnDemand tables - while copying, it creates provisioned destination table, we need to change it back to OnDemand from console once the table is created.

shomeprasanjit commented 5 years ago

najimou: should we add it to master?

najimou commented 5 years ago

shomeprasanjit good catch. Concerning add it to master, I think it is only a workaround. const defaultValue=5; is arbitrary value, depending the numbers of items on the table. If the value provisioned is not high enough there is a risk of having errors to reach the maximum threshold... feel free to propose a PR :)

TheFeelTrain commented 5 years ago

Adding both of your changes worked for me. It would probably be better to add the default value as a config option under source and destination.

pragati-l commented 5 years ago

As Source table has 'on-demand' capacity, it will be better to keep the same for the destination table.

As per aws documentation,

image

In order to do that, index.js -> clearTableSchema function code can be changed as below,

if (table.BillingModeSummary) {
    table.BillingMode = table.BillingModeSummary.BillingMode;
    if(table.BillingMode === "PAY_PER_REQUEST")
       {
           delete table.ProvisionedThroughput
       }
     delete table.BillingModeSummary
  }

  if(table.LocalSecondaryIndexes && table.LocalSecondaryIndexes.length > 0){
    for(var i = 0 ; i < table.LocalSecondaryIndexes.length ; i++){
        delete table.LocalSecondaryIndexes[i].IndexStatus
       if(table.BillingMode === "PAY_PER_REQUEST")
       {
        delete table.LocalSecondaryIndexes[i].ProvisionedThroughput
       }else{
        delete table.LocalSecondaryIndexes[i].ProvisionedThroughput.LastIncreaseDateTime
        delete table.LocalSecondaryIndexes[i].ProvisionedThroughput.LastDecreaseDateTime
        delete table.LocalSecondaryIndexes[i].ProvisionedThroughput.NumberOfDecreasesToday
       }
        delete table.LocalSecondaryIndexes[i].IndexSizeBytes
        delete table.LocalSecondaryIndexes[i].ItemCount
        delete table.LocalSecondaryIndexes[i].IndexArn
        delete table.LocalSecondaryIndexes[i].LatestStreamLabel
        delete table.LocalSecondaryIndexes[i].LatestStreamArn
    }
  }
  if(table.GlobalSecondaryIndexes && table.GlobalSecondaryIndexes.length > 0){
    for(var j = 0 ; j < table.GlobalSecondaryIndexes.length ; j++){
        delete table.GlobalSecondaryIndexes[j].IndexStatus
        if(table.BillingMode === "PAY_PER_REQUEST")
        {
          delete table.GlobalSecondaryIndexes[j].ProvisionedThroughput
        }else{
          delete table.GlobalSecondaryIndexes[j].ProvisionedThroughput.LastIncreaseDateTime
          delete table.GlobalSecondaryIndexes[j].ProvisionedThroughput.LastDecreaseDateTime
          delete table.GlobalSecondaryIndexes[j].ProvisionedThroughput.NumberOfDecreasesToday
        }

So in short, instead of keeping default 'BillingMode', we can copy mode from the source table...

jforge commented 4 years ago

Is there a chance to get this change in a new release?

enGMzizo commented 4 years ago

Added to latest release