eb deploy is bundling useless files

auspicacious commented 4 years ago

eb deploy is controlled by the .ebignore file, but certain directories that should be ignored, such as .direnv, are being bundled into the deployment packages regardless.

matschaffer commented 4 years ago

Check out the package.py we use in api. Pretty sure it’ll fix that. Seem to recall eb deploy ignores ebignore if it sees a gitignore or something silly like that.

auspicacious commented 4 years ago

It's the other way around -- eb deploy uses .gitignore unless a .ebignore is present, and from what I've seen my .ebignore is working for some files, just not others, which suggests to me its format is not actually the same as .gitignore's.

That being said, adapting package.py is probably the right solution so that we can fetch the install package, etc.

charlesgreen commented 4 years ago

"If .ebignore isn't present, but .gitignore is, the EB CLI ignores files specified in .gitignore. If .ebignore is present, the EB CLI doesn't read .gitignore."

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb-cli3-configuration.html#eb-cli3-ebignore

charlesgreen commented 4 years ago

Found an old thread that mentions the .ebignore file uses pathspec for rule parsing.

It also looks like many people have had the same problem with it not ignoring directories due to not including the required trailing slash on the directory name.

For example .direnv would need to be .direnv/.

auspicacious commented 4 years ago

Thanks @charlesgreen, I will update the repository with those new pathspecs and hopefully next time we deploy we'll see it work properly.

auspicacious commented 4 years ago

Hmm. I already had a trailing slash, I'm wondering if the root slash is confusing it (although, again, seems valid according to pathspec's docs). In any case, we are leaning towards a more custom solution as mentioned above and discussed in the channel, I think.

matschaffer commented 4 years ago

I don't think this has been linked directly, but https://github.com/Safecast/safecastapi/blob/master/.elasticbeanstalk/package.py is what the other apps use for packaging.

The reasoning here is that it provides the same zip that eb deploy does, but doesn't actually deploy it.

The flow for api.safecast.org is:

ci builds, then runs package.sh - https://github.com/Safecast/safecastapi/blob/master/.circleci/config.yml#L68
package.sh calls package.py to create the zip - https://github.com/Safecast/safecastapi/blob/master/.elasticbeanstalk/package.sh#L33
package.sh pushes the zip to s3 and registers it as an app version - https://github.com/Safecast/safecastapi/blob/master/.elasticbeanstalk/package.sh#L35-L44

I've found it to work pretty well.

Since that way you can get a deployable bundle from any CI build, but CI doesn't need to have permissions to deploy (thereby giving the world permissions to deploy, anyone can open a PR).

The icing on the cake would be something that triggers autodeploy from circleci master builds (probably some sort of lambda webhook) since master code can only be changed by authorized contributors.

auspicacious commented 4 years ago

Thanks @matschaffer, @charlesgreen is going to tackle the first part of this by automating all the manual stuff I was doing to package the installer in #9

matschaffer commented 4 years ago

Ah nice. I probably should have commented there. Anyway @charlesgreen - let me know on either issue if you have any questions on the above

Safecast / covid19map

eb deploy is bundling useless files #1