Closed labisso closed 4 years ago
Can you confirm which Python versions are coming to beta AL2 environments on EB? Python 3.6 and/or 3.8?
As PSF accepted the annual release cycle PEP, will you support every new Python version in future? (AWS Lambda supports Python 2.7/3.6/3.7/3.8 currently)
Thanks a lot.
Edit: post the GeoDjango support in a new ticket: https://github.com/aws/elastic-beanstalk-roadmap/issues/28
hi @cheng-liven for AL2 platforms we plan to start with Python 3.7 only, with 3.8 (and future versions) to follow. Do you have a specific need for 3.6?
Please can share timelines that would be great. I am waiting for quite sometime considering I am already using AL2 across the board but EB is still on AL.
hi @cheng-liven for AL2 platforms we plan to start with Python 3.7 only, with 3.8 (and future versions) to follow. Do you have a specific need for 3.6?
No, I don't have a specific need for Python 3.6, it's just the latest Python 3.x version supported on EB AL 2018.3 platform, so it might be easier for others to migrate to AL2 platforms. It is actually great that you can spend more time assessing #28 other than Python 3.6 support consider how fundamental GIS is in the mobile Internet era.
Have you thought about switching to use or supporting Pipenv instead of pip & virtualenv by default?
@mattjmorrison Yes! We're planning to use Pipenv in our upcoming Python on Amazon Linux 2 platform.
AL EOL is 12/31/2020. When do you think we can start using AL2 EB since it will be close-cut between AL EB vs AL2 EB from the consumer perspective.
Is there a way to access the original aws:elasticbeanstalk:container:python
values from within the Procfile
, e.g. for determining the process count to use?
Also, the new docs mention that the hooks have "…access to all environment properties that you've defined in application options". Are there any examples on how to do this from within a platform script? Thanks!
The new docs mention some limitations with the beta:
Amazon Linux 2 platforms are missing some features in the beta versions. We're working on adding support for these features. Here's a list of features that aren't supported at this time.
- Serving static files
- Custom platforms
When will these features be available? I am wanting to migrate a Django application.
Python 3.7 on AL2 platform is now generally available. For more information see What's New post and Release Notes
Thank you for the migration Amazon Linux 2 using Python 3.7.
I am wondering if there will be a way to add hooks for config deployment like the ones available on Amazon Linux in future updates since the default config deployment breaks our current deployment since we modify certain configuration settings.
@saranshsingh1 The Platform hooks feature on AL2 platforms is described in this blog post
Yes, but they only work when you deploy the application.
Say, you make some configuration changes like adding environment variables or change the ec2 instance type, etc., that triggers [INFO] Engine command: (config-deploy)
which does not use any of the platform hooks.
So, I was wondering if you guys plan to add that feature as well just like how AL1 had appdeploy
, configdeploy
, and restartappserver
. It would help to have configdeploy
alongside the current predeploy
and postdeploy
hooks.
I can share the cloudwatch logs which show the entire deployment process failing because the hooks in predeploy
and postdeploy
hooks are not executed, rightfully so.
@saranshsingh1 Thanks for the feedback. The configdeploy
is currently not available in AL2 platforms and we will evaluate this request.
Thank you. I appreciate the response and hopefully this feature does get added.
Hi, just wondering if anyone else is having problems with deployment on EB AL2?
Was fighting with it all day yesterday and cannot get apps to deploy properly. Here's what i've tried:
If i switch back to the sample it works again. I just cant figure this out. web.service failed to start.
Help! :)
@N3Cr0 Look at the logs of eb-engine.log
. That should give you an idea of what went wrong.
@N3Cr0 Look at the logs of
eb-engine.log
. That should give you an idea of what went wrong.
Thanks @saranshsingh1 . It's pretty non-descript on this issue.
But that said, I think I figured out what I was doing wrong (maybe you can confirm).
I didn't realise that so much changed between AL1 and AL2 for Python. And following tutorials was actually to my detriment. I was still using Pip and 'requirements.txt' where I should have been using Pipfile. Maybe some extras config required too.
The AWS sample App I downloaded originally was the one foe AL1. So it didn't work either. Am I correct?
The AWS sample App I downloaded originally was the one foe AL1. So it didn't work either. Am I correct?
@N3Cr0 Sorry, I never downloaded the sample application, so I cannot confirm that.
I was still using Pip and 'requirements.txt' where I should have been using Pipfile. Maybe some extras config required too.
You can use either requirements.txt or Pipfile to specify dependencies. Note the order mentioned in the last paragraph. Either use requirements.txt or Pipfile (based on your preference).
There are various other changes like wsgi and static file naming convention, using a Procfile to specify the gunicorn command, etc.
I will say that I found the migration guide a little useful.
I cannot get my Django app to deploy on the new Python platform. Running the command python manage.py collectstatic --noinput
fails, both in the old config file style and in the new .platform/hooks/predeploy/collectstatic.sh
setup.
Either way Django is not being found as a Python package, as if the virtual environment has not beed activated. Do we now have to activate the virtual environment first?
Here's my shell script
#!/bin/bash
sudo source staging-LQM1lest/bin/activate
python manage.py collectstatic --noinput
And the output from the logs
2020/04/26 15:43:40.142452 [INFO] Running command .platform/hooks/predeploy/collect_static.sh
2020/04/26 15:43:40.162781 [ERROR] sudo: source: command not found
File "manage.py", line 25
) from exc
^
SyntaxError: invalid syntax
I have checked on the instance, and Django was successfully install from my requirements.txt
file, with django-admin
in the bin folder and all the requirements in the site-packages folder for the virtual environment.
Can anyone help me to work out how to get this app deployed? Thanks.
EDIT: sudo source
was wrong, removing sudo
worked to activate the env.
I checked within the /var/app/staging
folder to see what was going on. If I set up a Python prompt I get this:
[ec2-user@ip-xxx-xx-xx-xxx ~]$ cd /var/app/
[ec2-user@ip-xxx-xx-xx-xx app]$ ls
current staging venv
[ec2-user@ip-xxx-xx-xx-xx app]$ cd staging/
[ec2-user@ip-xxx-xx-xx-xx staging]$ python
Python 2.7.16 (default, Dec 12 2019, 23:58:22)
[GCC 7.3.1 20180712 (Red Hat 7.3.1-6)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
That shows the virtualenv is not being used, so no wonder Django could not be found. Is the build system on these images missing steps to make sure the venv is being used at all times?
@digicase The way beanstalk sets up the virtual environment using pipenv
does not help in accessing the virtual environment so we you cannot launch it using pipenv shell
.
I cannot get my Django app to deploy on the new Python platform. Running the command python manage.py collectstatic --noinput fails, both in the old config file style and in the new .platform/hooks/predeploy/collectstatic.sh setup
To collect the static files, you have to be in the app directory. If you're running the shell script from predeploy
hooks directory, it is the staging
directory or current
directory for postdeploy hooks.
Try something along the lines of:
#!/bin/bash
source /var/app/venv/*/bin/activate
cd /var/app/staging
python manage.py collectstatic --noinput
You do not need to add sudo since all the hooks are run as root.
For deployment:
The workaround I am using is getting the PATH
environment variable from /opt/elasticbeanstalk/deployment/env
, storing it in /var/app/current/path_file
and using it as EnvironmentFile=/var/app/current/path_file
in the systemd
service block.
I have replaced the default web.service
file (which beanstalk creates) as below:
[Unit]
Description=Gunicorn daemon for application server
After=network.target
[Service]
# systemd.exec ? Execution environment configuration
User=webapp
Group=nginx
Type=notify
WorkingDirectory=/var/app/current/
ExecStart=/bin/sh -c "gunicorn -c gunicorn_config.py app_name.wsgi:application"
ExecStartPost=/bin/sh -c "systemctl show -p MainPID web.service | cut -d= -f2 > /var/pids/web.pid"
ExecStopPost=/bin/sh -c "rm -f /var/pids/web.pid"
Restart=always
EnvironmentFile=/var/app/current/path_file
# https://www.freedesktop.org/software/systemd/man/systemd.exec.html#SyslogIdentifier=
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=web
KillMode=mixed
[Install]
WantedBy=multi-user.target
Feel free to use the above as a reference, works perfectly for me. The above setup of getting the PATH
from the env file ensures I am not hardcoding the virtual environment path.
Thanks @saranshsingh1 I'll try this.
Is there any way to get the leader_only
functionality from container commands using platform hooks?
Is there any way to get the
leader_only
functionality from container commands using platform hooks?
Found a way to do that?
Is there any way to get the
leader_only
functionality from container commands using platform hooks?Found a way to do that?
Nope, would have been useful for django migration. Although inefficient, my workaround is to just migrate on every ec2 deployment. This works for me.
.platform/hooks/predeploy/01_django.sh
#!/bin/bash
source /var/app/venv/*/bin/activate
python manage.py migrate
python manage.py collectstatic --noinput
Interesting. I assume this only works because ELB deploys one instance at a time?
The predeploy hook runs on every new instance that ELB deploys. Doesn't necessarily have to be one at a time (depends on if you're doing rolling deployments or all at once). Should work in either case.
@saranshsingh1 what would be the nginx configuration files look like in order to add the static files location, and various extra settings:
Trying to achieve the following settings, but it seems rather impossible without fully re-writing the nginx config file:
@cristianoccazinsp As the documentation for Reverse configuration proxy
suggests, you can add/extend the Nginx configuration as you wish.
For application related settings like:
static files path (django) add cache headers to static files http to https redirect if forwarded-for proto pass forwarded-for-proto header to the proxy Add security headers (X-Frame-Options, HSTS, etc..) to both static and proxied requests setup GZIP to both static and proxied requests
you can add the properties in a *.conf
file in the .platform/hooks/nginx/conf.d
directory. I have different files for gzip, proxy headers (like x-forwarded-for, etc), security headers (like x-frame-options, etc), and application (like static file path, redirects, etc). All these files are placed in the conf.d
directory.
You can have a similar approach to disable server_tokens
or other properties of http
block by adding these properties using include
keyword of nginx but I found it easier to just replace the complete nginx.conf
file.
It's unclear from the docs if these extra settings are added below the server { ... } group or under the location / { ... } group.
How would you add settings that are supposed to be below server and others below location?
@cristianoccazinsp Here's the /etc/nginx/nginx.conf
file I found.
#Elastic Beanstalk Nginx Configuration File
user nginx;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
worker_processes auto;
worker_rlimit_nofile 66485;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
include conf.d/*.conf;
map $http_upgrade $connection_upgrade {
default "upgrade";
}
server {
listen 80 default_server;
access_log /var/log/nginx/access.log main;
client_header_timeout 60;
client_body_timeout 60;
keepalive_timeout 60;
gzip off;
gzip_comp_level 4;
gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml appl$
# Include the Elastic Beanstalk generated locations
include conf.d/elasticbeanstalk/*.conf;
}
}
@labisso it would be really helpful to get the AWS EB docs updated to include more detail on setting up a Python app in EB on the AL2 platform. The things that have been brought up in the comments in this thread, especially leader_only
functionality for platform hooks. Activating the Python virtual environment and using environment vars in platform hooks. Documenting the default Nginx config. Thanks.
Regarding the leader_only
in scripts, it is possible to do something like this in the EB config:
container_commands:
001-command:
command: echo "ELASTICBEANSTALK_CMD_LEADER=true" > .ebextensions/.elastic-beanstalk-cmd-leader
leader_only: true
And then just check for if that file exists or source it.
On the old platform, I've also used this snippet:
if [[ "$EB_IS_COMMAND_LEADER" == "true" ]]; then
# leader
else
# not a leader
fi
The code was in the /opt/elasticbeanstalk/bin/leader-test.sh
and the EB_IS_COMMAND_LEADER
variable was set during deployment.
I can still see this script on the new platform, in the /opt/elasticbeanstalk/config/private/containercommandsupport/leader-test.sh
, so this method might also work on the new platform, I didn't have time to check it yet.
Thanks for the nginx file, but that still doesn't really tell me how can I add various settings under various places of the config file.
For example, the default configuration for the python reverse proxy only adds X-Forwarded-For but not X-Forwarded-For-Proto, so I need to add an extra line under that location / {...}
section.
I’ll have to leave it to those with Nginx experience to advise on that. I hope Amazon can update their docs with more info and specific examples. For the previous Linux platforms there is a Github repository of proxy server config examples.
The Nginx configuration really depends on how you want the different configurations to work with each other.
I have my main config file which includes all other options. Below is my main file.
/etc/nginx/nginx.conf
.....
http {
charset utf-8;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
server_tokens off;
log_not_found off;
keepalive_timeout 65;
types_hash_max_size 2048;
client_max_body_size 100M;
# logging
access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log warn;
# load configs
include conf.d/*.conf; # this is the reference location
# MIME
include /etc/nginx/mime.types;
....
}
Then all my server
blocks for different URLs are placed inside the conf.d
directory like conf.d/default.conf
, conf.d/app1.conf
, conf.d/app2.conf
, etc. Below is one such file for /etc/nginx/conf.d/app1.conf
:
# http -> https server block
server {
listen 80;
listen [::]:80;
server_name app_url.com;
return 301 https://app_url.com$request_uri;
}
# Subdomains redirect
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name *.app_url.com;
# Include the SSL certificates
include conf.d/common/ssl_cert.conf;
return 301 https://app_url.com$request_uri;
}
# main web config
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name app_url.com;
# Include the Elastic Beanstalk generated locations
include conf.d/common/*.conf;
}
With the above setup, you can add whatever properties you want. I use the above setup following the DRY principle.
Now inside the conf.d/common/
directory, I place all the relevant attributes I want all my server
blocks to reference. Eg: headers.conf
, gzip.conf
, application.conf
, etc.
You'll need to learn how nginx
uses these files. Whenever nginx is reloaded or started, all the files added in the include
directive are pulled in; thus all the references to any number of files make it look as if its one big file with all the http
, server
, and location
blocks added in their respective directives.
It's just like breaking one big function into smaller pieces of code which can be called from the main function. That's Nginx specific and nothing to do with AWS or Beanstalk.
Adding the rest of the files for completeness the answer.
/etc/nginx/conf.d/common/application.conf
# Set base variable to application directory for reference
set $base /var/app/current;
# Django static
location /static/ {
alias $base/static/;
}
# Django media
location /media/ {
alias $base/media/;
}
# Django reverse proxy
# These values are passed to app server
location / {
proxy_http_version 1.1;
proxy_cache_bypass $http_upgrade;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Request-ID $request_id; # Pass to app server --- Maybe add this
proxy_redirect off;
proxy_pass http://unix:/run/my_app/gunicorn.sock;
}
# additional config
favicon.ico
location = /favicon.ico {
log_not_found off;
access_log off;
}
# robots.txt
location = /robots.txt {
log_not_found off;
access_log off;
}
/etc/nginx/conf.d/common/headers.conf
# Security Headers
add_header X-Request-ID $request_id;
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header Referrer-Policy "no-referrer-when-downgrade" always;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; " always;
# add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always;
# add_header X-Content-Type-Options "nosniff" always; # nginx already adds; avoid duplication
/etc/nginx/conf.d/common/gzip.conf
# gzip
gzip on;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_types text/plain text/css text/xml application/json application/javascript application/rss+xml application/atom+xml image/svg+xml;
@saranshsingh1 That's awesome, thanks!
attempting to run .platform/hooks/predeploy/01_django.sh
and getting permission denied
in the logs. file is below:
#!/bin/bash source /var/app/venv/*/bin/activate cd /var/app/staging/ python manage.py migrate python manage.py collectstatic --noinput
I can run the commands manually as root when I ssh into the instance, but not with the predeploy hook.
Silly that ELB doesn't do this automatically, but you can fix that by adding the following to your .ebextensions:
container_commands:
001-chmd:
command: chmod -R 777 .platform/
Silly that ELB doesn't do this automatically, but you can fix that by adding the following to your .ebextensions:
container_commands: 001-chmd: command: chmod -R 777 .platform/
That's the wrong move. The documentation clearly says you have to add execute rights to the shell script via:
chmod +x .platform/hooks/predeploy/01_django.sh
Here's something that might be helpful for some. I ran into an issue where Django wouldn't see any of the environment variables that were set on the ElasticBeanstalk configuration when running management commands or cronjobs. With Amazon Linux you could source /opt/python/current/env
and that would export all the environment variables. On Amazon Linux 2 that file doesn't exist.
I created a post deploy script in .platform/hooks/postdeploy/01_env.sh
that recreates the old file on deploy.
Put these contents in the shell script:
#!/bin/bash
input="/opt/elasticbeanstalk/deployment/env"
output="/var/app/env"
> $output
while IFS= read -r line
do
key=`echo $line | cut -d= -f1`
value=`echo $line | cut -d= -f2`
echo "export $key=\"$value\"" >> $output
done < "$input"
chmod 444 $output
You can use tighter security if you want, but just remember that root will create this file and when you ssh into the server you are the ec2-user.
Don't forget to chmod +x .platform/hooks/postdeploy/01_env.sh
or the build system will not have rights to execute the script.
Here's something that might be helpful for some. I ran into an issue where Django wouldn't see any of the environment variables that were set on the ElasticBeanstalk configuration when running management commands or cronjobs. With Amazon Linux you could
source /opt/python/current/env
and that would export all the environment variables. On Amazon Linux 2 that file doesn't exist.I created a post deploy script in
.platform/hooks/postdeploy/01_env.sh
that recreates the old file on deploy.Put these contents in the shell script:
#!/bin/bash input="/opt/elasticbeanstalk/deployment/env" output="/var/app/env" > $output while IFS= read -r line do key=`echo $line | cut -d= -f1` value=`echo $line | cut -d= -f2` echo "export $key=\"$value\"" >> $output done < "$input" chmod 444 $output
You can use tighter security if you want, but just remember that root will create this file and when you ssh into the server you are the ec2-user.
Don't forget to
chmod +x .platform/hooks/postdeploy/01_env.sh
or the build system will not have rights to execute the script.
It don't work for me, because script .platform/hooks/postdeploy/01_env.sh
run before save variables to /opt/elasticbeanstalk/deployment/env
.
I observe this when I add a new environment properties through the Beanstalk web console.
Has anyone experienced permission denied on the .sh file even after running chmod +x ? I ran the script locally to confirm it could be executed. I also ssh'd into the EC2 instance and ran all of the lines manually so I know they work. But when I deploy the app, it still fails and says that permission is denied for the .sh file.
@labisso it would be really helpful to get the AWS EB docs updated to include more detail on setting up a Python app in EB on the AL2 platform. The things that have been brought up in the comments in this thread, especially
leader_only
functionality for platform hooks. Activating the Python virtual environment and using environment vars in platform hooks. Documenting the default Nginx config. Thanks.
Is there a timeframe for this? The comment was made months ago and even little tidbit improvements would be massively helpful in avoiding wasted time. I was unfamiliar with AL1 & AL2 and have gone through a week of pain due to outdated documentation and major differences I didn't know I had to look out for as I'm fleshing out an eb deployment hitting snag after snag. Even just documenting which log files contain what would be so helpful - I finally realized after opening random log files on the machine that /var/log/cfn-init-cmd.log
is the one I wanted for my last issue.
I found out from this log my command failing is due to the python environment not loading how it used to. How do I execute a command that uses the correct python environment in AL2? Is using a command no longer the recommended way to do a migrate in Django? It would be very helpful to know how the python environment is intended to be used in AL2 - especially for such a widely used framework such as Django.
Since no one has mentioned it yet here, the docs for installing CloudWatch on Beanstalk are super outdated and should get updated:
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-cw.html
Also I created gists for how I get Celery and nginx working:
https://gist.github.com/Alex3917/9b09e67069c88361d4e0187e1968bc64 https://gist.github.com/Alex3917/de07c7e181b61133662af5c76d3ea869
Hello,
I try to execute a run_collectstatic.sh file in .platform/confighooks/predeploy during configuration deployment (not application deployment, Configuration deployment hooks)
My files: .platform/confighooks/predeploy/run_collectstatics.sh:
#!/usr/bin/env bash
source /var/app/venv/*/bin/activate
python manage.py collectstatic --noinput
.ebextensions/01_permission.config:
container_commands:
01_change_mode_of_hooks_file:
command: chmod +x .platform/hooks/predeploy/run_collectstatics.sh
02_change_mode_of_confighooks_file:
command: chmod +x .platform/confighooks/predeploy/run_collectstatics.sh
But the configuration deployment fail with the following error (the application deployment is OK):
Error: Command .platform/confighooks/predeploy/run_collectstatics.sh failed with error fork/exec .platform/confighooks/predeploy/run_collectstatics.sh: permission denied
It seems that container_commands is not executed during configuration deployment. How can I change the chmod of the file for configuration deployment ? Any alternative ?
@appli-intramuros : As the .platform
folder is part of your artifact, you should add the executable flag before you upload or pack your zip-file. So run the chmod command locally before you create a new version into your EBS environment.
A Python platform based on Amazon Linux 2 is now available to all customers as a BETA. We'd love it if you could try out the platform and provide any feedback you have. For details about what is different in this platform, check out the documentation page Migrating your Elastic Beanstalk Linux application to Amazon Linux 2.
Feel free to leave comments or complaints in the comments of this issue.