How to keep subdirectories when uploading?

[schmkr]

Hi,

Is it possible to copy a directory-structure from src to dst?

So let's say you have:

src/scripts/main.js
src/scripts/vendor/require.js
src/scripts/vendor/jquery.js

But after running an upload with:

upload: [
          {
            src: 'src/scripts/**/*.js',
            dest: '/scripts/'
          }
]

the require.js and jquery.js files are in the /scripts/ dir, instead of /scripts/vendor/ dir.

Cheers, Alwin

[pifantastic]

This is a common request, and is theoretically fixed in master. I've delayed releasing it because it needs to be tested a bit more before I'm comfortable with it. The reason it didn't work that way to begin with is that allowing wildcards in the src attribute made it ambiguous as to where they should be uploaded.

What you'll want to do is add a rel attribute that contains the relative base path you want to use when uploading the files.

In your case:

upload: [
  {
    src: 'src/scripts/**/*.js',
    dest: '/scripts/',
    rel: 'src/scripts'
  }
]

This will take all the files from src/scripts and upload them, preserving their tree, to /scripts/.

It's a touch confusing, and if I can come up with a clearer API, I will.

[dlupu]

This works only when the source folder is under the current folder. It does not work in the following case

   s3: {
        options: {
            key: '<%=aws.key%>',
            secret: '<%=aws.secret%>',
            bucket: '<%=aws.bucket%>',
            access: '<%=aws.access%>',
            debug: false
        },
        dev : {
            upload: [
                {
                    src: '../../build/<%=pkg.name%>/**/*',
                    dest: 'grunt-test/',
                    rel: 'build/<%=pkg.name%>'
                }
            ]
        }

The output is :

     Running "s3:dev" (s3) task
     Running in debug mode, no transfers will be made
     Warning: Arguments to path.resolve must be strings Use --force to continue.
     Aborted due to warnings.

Any ideas on how to fix this ?

[heff]

Just want to say thank you for adding this. And you should really release this. Works for me, and needed to publish a site to s3.

[pifantastic]

@heff it should be released to npm as version 0.2.0-alpha.1.

[heff]

Thanks, yeah I more meant released as in listed in the docs.

On May 7, 2013, at 4:23 PM, Aaron Forsander notifications@github.com wrote:

@heff it should be released to npm as version 0.2.0-alpha.1.

— Reply to this email directly or view it on GitHub.

[pifantastic]

Oh, haha, sorry. Yes, you're absolutely right. I'll put it on my TODO list for today!

[nmec]

Thanks for adding this, it's just what I was looking for.

Is there a reason you can't use the dynamic file object handling baked into Grunt?

[NickHeiner]

:+1:

[pifantastic]

@nmec I didn't actually know about it until recently. It didn't exist when I originally built this. The problem with Grunt's file handling is that it expects a local src and dest. In our case, one or the other isn't actually a local file. So if we use it, it will need to be a hybrid approach. I'm currently experimenting with rewriting grunt-s3 completely, taking advantage of new grunt stuff and learning from past mistakes. My hope is that I have enough time to finish it. In my rewrite, I am indeed using a hybrid approach for file handling.

[derryl]

Is it possible (with the current version) to perform recursive uploads without hard-coding every subdirectory?

I'd try your method, but I'd have to keep updating the config whenever I write a new post in Jekyll.

It'd be nice if we could just do something like:

upload: {
    cwd: 'dist',
    src: '**',
    dest: '' // remote destination, of course
}

By the way, a HUGE thanks for all of your work on this project thus far. It's become a crucial part of my Grunt workflow, and I am very grateful.

[tukutela]

:+1:

[jm3]

So is it correct that Grunt-S3 can't upload a directory without explicitly naming each path level in the Gruntfile? I.e. to push build/vendor/jquery/ui/css/custom-theme/images to s3 (a real example from a current project), it would be necessary to write:

upload: {
    src: 'build/**/**/**/**/**/**/*',
    dest: '/'
}

or something similar? I guess a concise way to say it might be, wildcards seem not to be supported in directories, because build/** just ignores the directories and sprays all the files into the root. Is not handling directory structures a general Grunt limitation?

[johnstark]

Hi @pifantastic , I ran into the same issue that @jm3 described and was able to get upload mirroring sub directories using this plugin: https://github.com/MathieuLoutre/grunt-aws-s3 . You may want to check with that project owner to see if they have suggestions.

[SimplGy]

I can't tell if this feature is supposed to work or not. To use this for deployment I need to maintain directory structure. Is there a way to do this?

[derryl]

@johnstark I also switched to grunt-aws-s3 to work around this problem.

However, that library doesn't let you compress your text files before uploading... so I had to add an extra task using grunt-contrib-compress

---

@jm3 The config you specified (src: 'build/**/**/**/**/**/**/*') will have some wonky behavior.... You'll end up with a single directory that has no name, containing ALL the contents of build in a single level...

[marcbachmann]

Hello everybody, It looks like this module still hasn't a good solution to allow subdirectories, that's why I made a small wrapper to support it. Here is a small task named recursiveS3 and a sample config.

grunt.initConfig
  recursiveS3:
    public:
      options:
        key: 'yourKey'
        secret: 'yourSecret'
        bucket: 'yourBucket'
        access: 'public-read'
      files: [
        expand: true
        cwd: './public/'
        src: ['**/*']
        dest: ''
      ]

grunt.registerMultiTask 'recursiveS3', 'upload entire directory with subdirectories to s3', ->
  uploads = []
  for files in @files
    for file in files.src
      if grunt.file.isFile file
        newFile = {}
        newFile.src = file
        newFile.dest = files.dest
        uploads.push newFile

  options = @options()
  grunt.config('s3',
    key: options.key,
    secret: options.secret,
    bucket: options.bucket,
    access: options.access,
    upload: uploads
  )

  grunt.task.run('s3')

Kind regards

[nitinhayaran]

+1

[SimplGy]

grunt-aws-s3 maintains directory structure when you upload, and even does a diff on upload to only upload files that aren't already there. It's pretty awesome, and I don't work there!

[nitinhayaran]

I am using "grunt-s3": "~0.2.0-alpha.3" with following settings and things seems to be working perfectly fine.

Looks like this issue is fixed.

s3: {
    options: {
        key: '<%= aws.key %>',
        secret: '<%= aws.secret %>',
        bucket: '<%= aws.bucket %>',
        access: 'public-read',
        region: 'ap-southeast-1'
    },
    prod: {
        // These options override the defaults
        options: {
            maxOperations: 10
        },
        // Files to be uploaded.
        sync: [{
            src: 'dist/**/*.*',
            dest: 'static/',
            rel: 'dist',
            options: {
                gzip: true,
                gzipExclude: ['.jpg', '.jpeg', '.png', '.gif', '.woff', '.wav', '.webp']
            }
        }]
    }

}

[jcwilson]

it seems the "rel" option must be provided in order to maintain the directory structure

[nickjanssen]

Thanks @nitinhayaran, now it works for me as well keeping the original folder structure.

        sync: [{
            src: 'dist/prod/**/*',
            dest: '/',
            rel: 'dist/prod'
        }]

[rainabba]

This has got me migrating over to grunt-aws-s3 also.

[elado]

:+1: https://github.com/MathieuLoutre/grunt-aws-s3 works without a problem. This task uploads nested files as dir%2Ffile.js instead of dir/file.js.