appspec.yml pattern and exclude file globs

[ndench]

It would be great if the pattern and exclude keys accepted bash file globs and worked properly across sub directories. For example, I have a Symfony4 app that looks like:

bin/
    console
config/
    services.yaml
public/
    index.php
src/
    MyController.php
templates/
    my_template.html.twig

I want all directories to have mode 750 and all files to have mode 640. Except for the bin/console file which needs mode 750.

This is the config that I've tried in appspec.yml:

version: 0.0
os: linux 

files:
  - source: /
    destination: /srv/www/app/

permissions:
  - object: /srv/www/app
    owner: ubuntu
    group: www-data
    mode: 750
    type:
      - directory
  - object: /srv/www/app
    pattern: '**'
    except:
      - bin/console # also tried 'console' and 'bin/*' and '/srv/www/app/bin/console'
    owner: ubuntu
    group: www-data
    mode: 640
    type:
      - file
  - object: /srv/www/app/bin
    owner: ubuntu
    group: www-data
    mode: 750
    type:
      - file

But I get the following error:

The deployment failed because the permissions setting for (/srv/www/app/bin/console) is specified more than once in the application specification file. Update the files section of the AppSpec file, and then try again.

I can't seem to find a value which I can put in the except key that successfully excludes the bin/console file. If it supported bash file globs you could easily do something like bin/* because all the files in the bin directory are going to need different permissions.

[tomtomau]

Facing the same issue!

[rohkat-aws]

@tomtomau not sure if this is a bug seems more like a feature request

[andrispraulitis]

Yeah at the moment it seems impossible to exclude (using except) one nested nested folder.

[sanju784]

Facing same issue!! Even after an year haven't they provided any solution yet???

[expertcoder]

I'm experiencing the same issue. Been trying different things cant get it to work.

[arick]

Internally, they've acknowledged that this a bug, and that it is on their bug tracker. But, they have no estimate on when, or if, they will ever address it. When you correlate this bug, with the bug on Debian/Ubuntu distributions, where AWS CodeDeploy erases the time/date stamp information on all the files, It seems pretty obvious that either the AWS architects don't understand the criticality of file metadata, or they don't care. Either way, as a consequence, the whole AWS CodePipeline, AWS CodeDeploy etc. is:

1. More of a proof of concept, than a reliable, enterprise service that you can really depend on, and that
2. Amazon doesn't consider it to be a strategic asset.

[philstrong]

The request makes sense. I think we should consider adding a warning and not an error when there is overlap. I believe the original author was trying to stop weird behavior from happening where the last one wins. It's possible that some customers will have dozens of entries and so wouldn't be as easy to identify that you've already set perms on a directory.

Team will triage to understand LOE and dig up the history of why original rule was set.

[philstrong]

AWS CodeDeploy erases the time/date stamp information on all the files

Link?

[philstrong]

My recommendation wouldn't be to exclude but rather to allow the last one to win. Meaning if you get more specific later e.g. next subfolder down it will set that last.

It will still warn that you have already set permissions but will still allow you to do so.

Will this work? Not sure I understand what end permissions you want or the entire file struct from example above

[ppostma1]

After the directive for the permissions of directory: /srv/www/app, I can no long append directives for that directory /srv/www/app, pattern: '**' nor any of its children /srv/www/app/bin I deleted these directives, and move the commands to the shell file application_start.sh with chown user:group /files and chmod 640 /files. Removing these declarations allowed it to deploy without error and the chown/chmod commands completed the required objectives.

[gh-andre]

Five years later and this bug is still there. I spent a bunch of time trying different combinations of the object path and the except value and nothing works.

If Amazon isn't going to fix this bug, can you at least provide some guidance on how the value in except should be specified with regards to the object and pattern values in its section?

[gh-andre]

After wasting more time on various tests on this, I ended up looking at the source and it turns out that except value matching functionality will silently fail on for any value with a path separator. Here it is, right at the top:

          def matches_simple_glob(name, pattern)
            if name.include?(File::SEPARATOR)
              return false                          <--
            end
            options = expand(pattern.chars.entries)

The calling code in matches_except? takes the object value, and subtracts it from the actual path and the result is passed into this method. If the resulting path has a slash, it will fail to match and will be treated as if there's no match, even though it didn't even try. So, it looks like this:

/abc/def/           <-- object
/abc/def/xyz/123/   <-- actual path
xyz/123/            <-- passed to `matches_simple_glob` and will not match

The docs for this functionality are terrible and don't specify any of this.

Given this behavior, the code should fail appspec.yml validation if except values contain path separators, so these failures aren't equated to not matching the except patterns.

Anyway, in order to make all of this work one needs to carefully manage applying permissions to the deployment tree with series of *, ** and specific file/directory name patterns to ensure each entry in the tree is being affected only once. It's tedious and error-prone, but it works. If only computers could automate these tasks.

[marcguyer]

Adding my 2 cents here, essentially echoing @arick , that AWS seems to not care about this at all. The workaround is to do this in a hook script. After running into this prob and reading this thread the permissions: section of the appspec is dead to me 👻.

[gh-andre]

@marcguyer

Basically, you need to operate at individual levels for many of these settings, being careful what's applied within each level and what propagates. Here's an example.

Say, you stage deployment for multiple apps under /home/some_user/aws_staging, so CodeDeploy copies stuff from under /opt/codedeploy-agent/deployment-root/ to /home/some_user/aws_staging/app_1 for app_1, to /home/some_user/aws_staging/app_2 for app_2, etc. Then permissions would look like this.

permissions:
    # permissions are not applied to the `object` directory - set it explicitly from above
  - object: '/home/some_user/aws_staging'
    pattern: 'app_1'
    owner: some_user
    group: some_user
    mode: 775
    type:
    - directory

    # set file permissions only at this level
  - object: '/home/some_user/aws_staging/app_1'
    pattern: '*'
    owner: some_user
    group: some_user
    mode: 664
    type:
    - file

    # allow r/w for directories for user/group and traversal for all
  - object: '/home/some_user/aws_staging/app_1'
    pattern: '**'
    owner: some_user
    group: some_user
    mode: 775
    type:
    - directory

    # set the execute bit for scripts
  - object: '/home/some_user/aws_staging/app_1/some_script'
    owner: some_user
    group: some_user
    mode: 774
    type:
    - file

If you want to add an exception, it cannot contain path separators or it will silently fail. So, if you wanted to avoid setting permissions on a file called special, it would look like this:

  - object: '/home/some_user/aws_staging/app_1'
    pattern: '*'
    except:
    - 'special'
    owner: some_user
    group: some_user
    mode: 664
    type:
    - file

In my case, I have tar archives in each app's staging directory, which are extracted into the actual app directory with tar'ed permissions, so I don't need to set permissions for files recursively above, but it would look similar to the above. It's been a while and I don't recall nuances for ** used with files, though.

[marcguyer]

Yeah, @gh-andre , I vaguely inferred as much but decided not to go through the brain damage when I realized there doesn't seem to be any value in using the permissions: section over a traditional shell script on the AfterInstall hook.

Did you find it necessary to do this in permissions: for some reason I'm missing?

[gh-andre]

@marcguyer My app is running with non-root permissions, so if I don't set permissions in appspec.yml, the default root is propagated into the actual app deployment directory and messes up the app in not being able to access some files because they are root-owned.

You can set permissions in event hook scripts, but that splits where things are changed during deployment and can make it more complicated. Having said that, appspec.yml lacks variables, so one advantage of having using scripts is to be able to reuse some values, such as there's no way to use a variable for the staging directory name, etc, so search-and-replace is the unfortunate alternative.

I hear you. It made my brain hurt as well. Very time consuming to figure it out and maintain.

[ricardomagamobi]

+1 to this feature