How does incremental backup work? How do I setup the lambda function to trigger?

[camhart]

How does incremental backup work? How do I setup the lambda function to trigger?

Thanks for this! Looks great, just trying to figure out how best to integrate it. Can I simply trigger the aws lambda as a cron event once a day and it will backup any new records? Or is there some dynamodb stream stuff going on there?

[nathaniel-holder]

I have the same question...

[thoean]

• The DynamoDB stream is setup for "live-backup".
• If you want to run a full or incremental backup, you'll need to run that from your desktop, following the guidelines in the readme.

[nathaniel-holder]

Thanks, good to know. I was trying to setup backups to be run from lambda, on a schedule. Below is the lambda function that accomplishes this. It does a full backup of every table, and assumes this lambda function, dynamodb, and the s3 bucket are all in the same region.

'use strict';

console.log('[DynamoDBBackupRestore starting]');

const aws = require('aws-sdk');
const async = require('async');
const DBBackup = require('dynamodb-backup-restore').Backup;
const moment = require('moment-timezone');

// Triggered by CloudWatch Rule lambda scheduled backup
exports.handler = function DynamoDBBackupRestore(event, context, callback) {
    if (!event) return; // if called by build test

    console.log('Received event:', JSON.stringify(event, null, 2));
    console.log('Received context:', JSON.stringify(context, null, 2));

    const dynamodb = new aws.DynamoDB({ region: event.region });

    let configFullS3Prefix = 'DynamoDBFullBackups/' + moment().tz('America/Los_Angeles').format().replace(/\:/g, '');
    let config = {
        S3Bucket: 'STRING_VALUE', /* required */
        S3Prefix: configFullS3Prefix + '/' + '<tableName>', // Dynamically set for each table
        S3Region: event.region,
        DbRegion: event.region
    };
    console.log('config: %j', config);

    // Backup each table
    let dynamodBackupRestore;
    dynamodb.listTables({}, function (err, tables) {
        if (err) return callback(err);

        async.eachSeries(tables.TableNames,
            function (tableName, eachCallback) {
                console.log('Table: %s', tableName);
                config.DbTable = tableName;

                // Full backup
                console.log('Full backup of %s', tableName);
                config.S3Prefix = configFullS3Prefix + '/' + tableName;
                dynamodBackupRestore = new DBBackup(config);
                dynamodBackupRestore.full()
                    .then(() => eachCallback())
                    .catch(err => eachCallback(err));
            },
            function (err, results) {
                if (err) return callback(err);

                return callback();
            }
        );
    });
};

[thoean]

Thanks for sharing this. However note that Lambda has a maximum execution time of 5min, which might not work depending on your database size.

That said, I recommend setting up the dynamoDB stream if you want ongoing backups, and run a full backup only once locally (or on a rare occasion to yet another S3 bucket).

[nathaniel-holder]

Thanks for the heads up on lambda limitations. With dynamoDB stream backups, how would I restore to a specific point in time? Would be great to see an example of a dynamoDB stream backup and restore scenario. Thanks!

[thoean]

The backup isn't once, but continuous. Just run the restore with the right date to restore, following the documentation.

Technically, it uses a versioned S3 bucket, and uses that timestamp to recover. Let's say you have a single document stored at 09:00 the first time, then changed at 10:00 and again at 11:00. If you want to restore to 10:15, it'd apply the last entry before 10:15, which is the one at 10:00.

It's like a full backup, but continuously applied.

[nathaniel-holder]

Awesome, this is great! I missed the "--RestoreTime" parameter for gulp because it's not in the documentation (only in the documentation for code), but now I see it's in the gulpfile. Thanks!