search

DanielHindi / aws-s3-zipper

takes an amazon s3 bucket folder and zips it for streaming or serializes to a file
119 stars 74 forks source link

readme

Amazon S3 Zipping tool (aws-s3-zipper)

What does it do?

1. Zips S3 files

Takes an amazon s3 bucket folder and zips it to a:

2. Differential zipping

It also allows you to do differential zips. You can save the key of the last file you zipped and then zip files that have been uploaded after the last zip.

3. Fragmented Zips

If a zip file has the potential of getting too big. You can provide limits to breakup the compression into multiple zip files. You can limit based on file count or total size (pre zip)

4. Filter Files to zip

You can filter out files you dont want zipped based on any criteria you need

How do i use it?

Setup

var S3Zipper = require ('aws-s3-zipper');

var config ={
    accessKeyId: "XXXX",
    secretAccessKey: "XXXX",
    region: "us-west-2",
    bucket: 'XXX'
};
var zipper = new S3Zipper(config);

Filter out Files

zipper.filterOutFiles= function(file){
    if(file.Key.indexOf('.tmp') >= 0) // filter out temp files
        return null;
    else 
      return file;
};

Zip to local file

zipper.zipToFile ({
        s3FolderName: 'myBucketFolderName'
        , startKey: 'keyOfLastFileIZipped' // could keep null
        , zipFileName: './myLocalFile.zip'
        , recursive: true
    }
    ,function(err,result){
        if(err)
            console.error(err);
        else{
            var lastFile = result.zippedFiles[result.zippedFiles.length-1];
            if(lastFile)
                console.log('last key ', lastFile.Key); // next time start from here
        }
});

Pipe zip data to stream (using Express.js)

app.all('/', function (request, response) {
    response.set('content-type', 'application/zip') // optional
    zipper.streamZipDataTo({
        pipe: response
        , folderName: 'myBucketFolderName'
        , startKey: 'keyOfLastFileIZipped' // could keep null
        , recursive: true
        }
        ,function (err, result) {
            if(err)
                console.error(err);
            else{
                console.log(result)
            }
        })
})

Zip fragments to local file system with the filename pattern with a maximum file count

zipper.zipToFileFragments ({
        s3FolderName:'myBucketFolderName'
        ,startKey: null
        ,zipFileName './myLocalFile.zip'
        ,maxFileCount: 5
        ,maxFileSize: 1024*1024
    }, function(err,results){
        if(err)
               console.error(err);
           else{
               if(results.length > 0) {
                   var result = results[results.length - 1];
                   var lastFile = result.zippedFiles[result.zippedFiles.length - 1];
                   if (lastFile)
                       console.log('last key ', lastFile.Key); // next time start from here
               }
           }
   });

Zip to S3 file

/// if no path is given to S3 zip file then it will be placed in the same folder
zipper.zipToS3File ({
        s3FolderName: 'myBucketFolderName'
        , startKey: 'keyOfLastFileIZipped' // optional
        , s3ZipFileName: 'myS3File.zip'
        , tmpDir: "/tmp" // optional, defaults to node_modules/aws-s3-zipper
    },function(err,result){
        if(err)
            console.error(err);
        else{
            var lastFile = result.zippedFiles[result.zippedFiles.length-1];
            if(lastFile)
                console.log('last key ', lastFile.Key); // next time start from here
        }
});

Zip fragments to S3

zipper.zipToS3FileFragments({
    s3FolderName: 'myBucketFolderName'
    , startKey: 'keyOfLastFileIZipped' // optional
    , s3ZipFileName: 'myS3File.zip'
    , maxFileCount: 5
    , maxFileSize: 1024*1024
    , tmpDir: "/tmp" // optional, defaults to node_modules/aws-s3-zipper
    },function(err, results){
    if(err)
        console.error(err);
    else    if(results.length > 0) {
        var result = results[results.length - 1];
        var lastFile = result.zippedFiles[result.zippedFiles.length - 1];
        if (lastFile)
            console.log('last key ', lastFile.Key); // next time start from here
    }

});

The Details

init

Either from the constructor or from the init(config) function you can pass along the AWS config object

{
    accessKeyId: [Your access id],
    secretAccessKey: [your access key],
    region: [the region of your S3 bucket],
    bucket: [your bucket name],
    endpoint: [optional, for use with S3-compatible services]
}

If using temporary credentials

{
    accessKeyId: [Your access id],
    secretAccessKey: [your access key],
    sessionToken: [your session token],
    region: [the region of your S3 bucket],
    bucket: [your bucket name],
    endpoint: [optional, for use with S3-compatible services]
}

filterOutFiles(file)

Override this function when you want to filter out certain files. The file param that is passed to you is the format of the aws file

getFiles: function(params,callback)

Get a list of files in the bucket folder

streamZipDataTo: function (params, callback)

If you want to get a stream to pipe raw data to. For example if you wanted to stream the zip file directly to an http response

zipToS3File: function (params ,callback)

Zip files in an s3 folder and place the zip file back on s3

zipToS3FileFragments: function (params , callback)

zipToFile: function (params ,callback)

Zip files to a local zip file.

zipToFileFragments: function (params,callback)