search

villasv / aws-airflow-stack

Turbine: the bare metals that gets you Airflow
https://victor.villas/aws-airflow-stack/
MIT License
377 stars 69 forks source link

webserver fails to load on successful stack creation #170

Closed ztyree42 closed 4 years ago

ztyree42 commented 4 years ago

CloudFormation finishes with all stacks created but cannot access the Web UI on port 8080.

villasv commented 4 years ago

Thanks for testing the stack so thoroughly @ztyree42, best stream of bug reports I've ever had :wink: I'll take a look this weekend.

villasv commented 4 years ago

I could not reproduce this issue on the branch develop (which is basically master but using airflow 1.10.10) on us-east-1.

ztyree42 commented 4 years ago

Should I expect to immediately be able to access 8080? If I go into SSM session on webserver instance I don't see webserver running. Should it be running after stack creation but before first deployment?

On Sat, Apr 25, 2020, 6:45 PM Victor Villas notifications@github.com wrote:

I could not reproduce this issue on the branch develop (which is basically master but using airflow 1.10.10) on us-east-1.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/villasv/aws-airflow-stack/issues/170#issuecomment-619465679, or unsubscribe https://github.com/notifications/unsubscribe-auth/AECDZPMBQNVTY3AMEUOZV5LROOG2DANCNFSM4MQQJXBQ .

villasv commented 4 years ago

Should it be running after stack creation but before first deployment?

Yes, a minute or two after the entire stack goes green, you should be able to visit the webserver on its public address (e.g. ec2-18-215-126-107.compute-1.amazonaws.com:8080) and see it there.

Can you give me more details about your deployment? Like if you deployed the master or the cluster stack; the region; the parameters (if any differs from the defaults); the contents of /var/log/cfn-init.log on the webserver; the outputs of journalctl -u airflow-weserver.service;

ztyree42 commented 4 years ago

I deployed the cluster into an existing VPC. All the networking/routes seem to be setup fine, if I start a session on the webserver instance, run cfn-init manually there, export the environment variables, then start the webserver w/ "airflow webserver" I can hit the webserver on

:8080. Pardon my ignorance but from which instance do you want to see the contents of cfn-init? webserver or scheduler? Similarly, should I make a session to run journalctl commands or is there a script around that that I'm missing? On Sat, Apr 25, 2020 at 9:53 PM Victor Villas wrote: > Should it be running after stack creation but before first deployment? > > Yes, a minute or two after the entire stack goes green, you should be able > to visit the webserver on its public address (e.g. > ec2-18-215-126-107.compute-1.amazonaws.com:8080) and see it there. > > Can you give me more details about your deployment? Like if you deployed > the master or the cluster stack; the region; the contents of > /var/log/cfn-init.log on the webserver; the outputs of journalctl -u > airflow-weserver.service; > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > , > or unsubscribe > > . >
villasv commented 4 years ago

Hmm. Existing VPC, cool. You shouldn't have to manyally run the cfn-init, it runs by itself when the stack deploys. If you were able to run the webserver manually, then it probably wasn't running before, since the port would be busy.

Sorry for the lack of details. Yes, in the webserver instance. And yes, run journalctl in a session.

ztyree42 commented 4 years ago

Here is the contents of cfn-inti.log

2020-04-25 00:44:06,852 [DEBUG] CloudFormation client initialized with
endpoint https://cloudformation.us-west-1.amazonaws.com
2020-04-25 00:44:06,852 [DEBUG] Describing resource LaunchConfiguration in
stack csds-turbine-WebserverStack-1XG5Y3FID3WN4
2020-04-25 00:44:06,966 [INFO] -----------------------Starting
build-----------------------
2020-04-25 00:44:06,967 [DEBUG] Not setting a reboot trigger as scheduling
support is not available
2020-04-25 00:44:06,967 [INFO] Running configSets: default
2020-04-25 00:44:06,968 [INFO] Running configSet default
2020-04-25 00:44:06,969 [INFO] Running config config
2020-04-25 00:44:06,969 [DEBUG] No packages specified
2020-04-25 00:44:06,969 [DEBUG] No groups specified
2020-04-25 00:44:06,969 [DEBUG] No users specified
2020-04-25 00:44:06,969 [DEBUG] No sources specified
2020-04-25 00:44:06,969 [DEBUG] No files specified
2020-04-25 00:44:06,969 [DEBUG] Running command setup
2020-04-25 00:44:06,970 [DEBUG] No test for command setup
2020-04-25 00:46:28,330 [ERROR] Command setup (export
AWS_STACK_NAME="csds-turbine-WebserverStack-1XG5Y3FID3WN4"
export LOGS_BUCKET="csds-turbine-logsbucket-125it1ytphgjk"
export QUEUE_NAME="csds-turbine-TaskQueue-85AQUQBWCIQD"
export
DB_SECRETS_ARN="arn:aws:secretsmanager:us-west-1:698886251042:secret:Secret-VSoBwBLEzg8K-1ZdBrc"
export LOAD_EXAMPLES="False"
export LOAD_DEFAULTS="False"
export WEB_SERVER_PORT="8080"
aws s3 sync s3://csds-misc/turbine/scripts /opt/turbine
chmod +x /opt/turbine/webserver.setup.sh
/opt/turbine/webserver.setup.sh
) failed
download: s3://csds-misc/turbine/scripts/systemd/airflow-heartbeat.timer to
opt/turbine/systemd/airflow-heartbeat.timerg
download: s3://csds-misc/turbine/scripts/airflow-terminate.sh to
opt/turbine/airflow-terminate.sh
download:
s3://csds-misc/turbine/scripts/systemd/airflow-confapply-agent.path to
opt/turbine/systemd/airflow-confapply-agent.path
download: s3://csds-misc/turbine/scripts/systemd/airflow.conf to
opt/turbine/systemd/airflow.conf
download: s3://csds-misc/turbine/scripts/airflow-heartbeat.sh to
opt/turbine/airflow-heartbeat.sh
download: s3://csds-misc/turbine/scripts/systemd/airflow.env to
opt/turbine/systemd/airflow.env
download: s3://csds-misc/turbine/scripts/systemd/airflow-terminate.service
to opt/turbine/systemd/airflow-terminate.service
download:
s3://csds-misc/turbine/scripts/systemd/airflow-confapply-agent.service to
opt/turbine/systemd/airflow-confapply-agent.service
download: s3://csds-misc/turbine/scripts/airflow-confapply.sh to
opt/turbine/airflow-confapply.sh
download: s3://csds-misc/turbine/scripts/systemd/airflow-heartbeat.service
to opt/turbine/systemd/airflow-heartbeat.service
download: s3://csds-misc/turbine/scripts/systemd/airflow-workerset.service
to opt/turbine/systemd/airflow-workerset.service
download: s3://csds-misc/turbine/scripts/systemd/cfn-hup.service to
opt/turbine/systemd/cfn-hup.service
download: s3://csds-misc/turbine/scripts/systemd/airflow-terminate.timer to
opt/turbine/systemd/airflow-terminate.timer
download: s3://csds-misc/turbine/scripts/webserver.setup.sh to opt/turbine/
webserver.setup.sh
download: s3://csds-misc/turbine/scripts/commons.setup.sh to opt/turbine/
commons.setup.sh
download: s3://csds-misc/turbine/scripts/workerset.setup.sh to opt/turbine/
workerset.setup.sh
download: s3://csds-misc/turbine/scripts/systemd/cfn-auto-reloader.conf to
opt/turbine/systemd/cfn-auto-reloader.conf
download: s3://csds-misc/turbine/scripts/systemd/airflow-webserver.service
to opt/turbine/systemd/airflow-webserver.service
download: s3://csds-misc/turbine/scripts/systemd/cfn-hup.conf to
opt/turbine/systemd/cfn-hup.conf
download: s3://csds-misc/turbine/scripts/systemd/airflow-scheduler.service
to opt/turbine/systemd/airflow-scheduler.service
download: s3://csds-misc/turbine/scripts/scheduler.setup.sh to opt/turbine/
scheduler.setup.sh
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
Existing lock /var/run/yum.pid: another copy is running as pid 2644.
Another app is currently holding the yum lock; waiting for it to exit...
  The other application is: yum
    Memory : 110 M RSS (327 MB VSZ)
    Started: Sat Apr 25 00:44:05 2020 - 00:03 ago
    State  : Running, pid: 2644
Resolving Dependencies
--> Running transaction check
---> Package jq.x86_64 0:1.5-1.amzn2.0.2 will be installed
--> Processing Dependency: libonig.so.2()(64bit) for package:
jq-1.5-1.amzn2.0.2.x86_64
--> Running transaction check
---> Package oniguruma.x86_64 0:5.9.6-1.amzn2.0.3 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package         Arch         Version                    Repository
 Size
================================================================================
Installing:
 jq              x86_64       1.5-1.amzn2.0.2            amzn2-core
154 k
Installing for dependencies:
 oniguruma       x86_64       5.9.6-1.amzn2.0.3          amzn2-core
127 k

Transaction Summary
================================================================================
Install  1 Package (+1 Dependent package)

Total download size: 281 k
Installed size: 890 k
Downloading packages:
--------------------------------------------------------------------------------
Total                                              3.0 MB/s | 281 kB  00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : oniguruma-5.9.6-1.amzn2.0.3.x86_64
1/2
  Installing : jq-1.5-1.amzn2.0.2.x86_64
 2/2
  Verifying  : oniguruma-5.9.6-1.amzn2.0.3.x86_64
1/2
  Verifying  : jq-1.5-1.amzn2.0.2.x86_64
 2/2

Installed:
  jq.x86_64 0:1.5-1.amzn2.0.2

Dependency Installed:
  oniguruma.x86_64 0:5.9.6-1.amzn2.0.3

Complete!
  % Total    % Received % Xferd  Average Speed   Time    Time     Time
 Current
                                 Dload  Upload   Total   Spent    Left
 Speed
100     3  100     3    0     0   3000      0 --:--:-- --:--:-- --:--:--
 3000
  % Total    % Received % Xferd  Average Speed   Time    Time     Time
 Current
                                 Dload  Upload   Total   Spent    Left
 Speed
100    63  100    63    0     0  63000      0 --:--:-- --:--:-- --:--:--
63000
  % Total    % Received % Xferd  Average Speed   Time    Time     Time
 Current
                                 Dload  Upload   Total   Spent    Left
 Speed
100  1310  100  1310    0     0  1279k      0 --:--:-- --:--:-- --:--:--
1279k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time
 Current
                                 Dload  Upload   Total   Spent    Left
 Speed
100   476  100   476    0     0  29750      0 --:--:-- --:--:-- --:--:--
29750
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
Resolving Dependencies
--> Running transaction check
---> Package python3.x86_64 0:3.7.6-1.amzn2.0.1 will be installed
--> Processing Dependency: python3-libs(x86-64) = 3.7.6-1.amzn2.0.1 for
package: python3-3.7.6-1.amzn2.0.1.x86_64
--> Processing Dependency: python3-setuptools for package:
python3-3.7.6-1.amzn2.0.1.x86_64
--> Processing Dependency: libpython3.7m.so.1.0()(64bit) for package:
python3-3.7.6-1.amzn2.0.1.x86_64
---> Package python3-devel.x86_64 0:3.7.6-1.amzn2.0.1 will be installed
--> Processing Dependency: system-rpm-config for package:
python3-devel-3.7.6-1.amzn2.0.1.x86_64
--> Processing Dependency: python3-rpm-macros for package:
python3-devel-3.7.6-1.amzn2.0.1.x86_64
---> Package python3-pip.noarch 0:9.0.3-1.amzn2.0.2 will be installed
---> Package python3-wheel.noarch 0:0.30.0a0-9.amzn2.0.3 will be installed
--> Running transaction check
---> Package python3-libs.x86_64 0:3.7.6-1.amzn2.0.1 will be installed
---> Package python3-rpm-macros.noarch 0:3-23.amzn2 will be installed
---> Package python3-setuptools.noarch 0:38.4.0-3.amzn2.0.6 will be
installed
---> Package system-rpm-config.noarch 0:9.1.0-76.amzn2.0.8 will be installed
--> Processing Dependency: dwz >= 0.4 for package:
system-rpm-config-9.1.0-76.amzn2.0.8.noarch
--> Processing Dependency: perl-srpm-macros for package:
system-rpm-config-9.1.0-76.amzn2.0.8.noarch
--> Running transaction check
---> Package dwz.x86_64 0:0.11-3.amzn2.0.2 will be installed
---> Package perl-srpm-macros.noarch 0:1-8.amzn2.0.1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package               Arch      Version                    Repository
Size
================================================================================
Installing:
 python3               x86_64    3.7.6-1.amzn2.0.1          amzn2-core
71 k
 python3-devel         x86_64    3.7.6-1.amzn2.0.1          amzn2-core
 242 k
 python3-pip           noarch    9.0.3-1.amzn2.0.2          amzn2-core
 1.9 M
 python3-wheel         noarch    0.30.0a0-9.amzn2.0.3       amzn2-core
84 k
Installing for dependencies:
 dwz                   x86_64    0.11-3.amzn2.0.2           amzn2-core
99 k
 perl-srpm-macros      noarch    1-8.amzn2.0.1              amzn2-core
 4.7 k
 python3-libs          x86_64    3.7.6-1.amzn2.0.1          amzn2-core
 9.1 M
 python3-rpm-macros    noarch    3-23.amzn2                 amzn2-core
 6.4 k
 python3-setuptools    noarch    38.4.0-3.amzn2.0.6         amzn2-core
 617 k
 system-rpm-config     noarch    9.1.0-76.amzn2.0.8         amzn2-core
80 k

Transaction Summary
================================================================================
Install  4 Packages (+6 Dependent packages)

Total download size: 12 M
Installed size: 51 M
Downloading packages:
--------------------------------------------------------------------------------
Total                                               47 MB/s |  12 MB  00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : python3-setuptools-38.4.0-3.amzn2.0.6.noarch
 1/10
  Installing : python3-pip-9.0.3-1.amzn2.0.2.noarch
 2/10
  Installing : python3-3.7.6-1.amzn2.0.1.x86_64
 3/10
  Installing : python3-libs-3.7.6-1.amzn2.0.1.x86_64
4/10
  Installing : perl-srpm-macros-1-8.amzn2.0.1.noarch
5/10
  Installing : dwz-0.11-3.amzn2.0.2.x86_64
6/10
  Installing : system-rpm-config-9.1.0-76.amzn2.0.8.noarch
7/10
  Installing : python3-rpm-macros-3-23.amzn2.noarch
 8/10
  Installing : python3-devel-3.7.6-1.amzn2.0.1.x86_64
 9/10
  Installing : python3-wheel-0.30.0a0-9.amzn2.0.3.noarch
 10/10
  Verifying  : python3-rpm-macros-3-23.amzn2.noarch
 1/10
  Verifying  : python3-libs-3.7.6-1.amzn2.0.1.x86_64
2/10
  Verifying  : python3-setuptools-38.4.0-3.amzn2.0.6.noarch
 3/10
  Verifying  : dwz-0.11-3.amzn2.0.2.x86_64
4/10
  Verifying  : python3-devel-3.7.6-1.amzn2.0.1.x86_64
 5/10
  Verifying  : python3-wheel-0.30.0a0-9.amzn2.0.3.noarch
6/10
  Verifying  : system-rpm-config-9.1.0-76.amzn2.0.8.noarch
7/10
  Verifying  : python3-pip-9.0.3-1.amzn2.0.2.noarch
 8/10
  Verifying  : python3-3.7.6-1.amzn2.0.1.x86_64
 9/10
  Verifying  : perl-srpm-macros-1-8.amzn2.0.1.noarch
 10/10

Installed:
  python3.x86_64 0:3.7.6-1.amzn2.0.1
  python3-devel.x86_64 0:3.7.6-1.amzn2.0.1
  python3-pip.noarch 0:9.0.3-1.amzn2.0.2
  python3-wheel.noarch 0:0.30.0a0-9.amzn2.0.3

Dependency Installed:
  dwz.x86_64 0:0.11-3.amzn2.0.2
  perl-srpm-macros.noarch 0:1-8.amzn2.0.1
  python3-libs.x86_64 0:3.7.6-1.amzn2.0.1
  python3-rpm-macros.noarch 0:3-23.amzn2
  python3-setuptools.noarch 0:38.4.0-3.amzn2.0.6
  system-rpm-config.noarch 0:9.1.0-76.amzn2.0.8

Complete!
WARNING: Running pip install with root privileges is generally not a good
idea. Try `pip3 install --user` instead.
Collecting awscurl
  Downloading
https://files.pythonhosted.org/packages/00/f1/bd9f704234da2e4686aaa93861483adcaff0fd8681d81fbb405379fedd5f/awscurl-0.20-py3-none-any.whl
Collecting configargparse (from awscurl)
  Downloading
https://files.pythonhosted.org/packages/bb/79/3045743bb26ca2e44a1d317c37395462bfed82dbbd38e69a3280b63696ce/ConfigArgParse-1.2.3.tar.gz
(42kB)
Collecting urllib3[secure] (from awscurl)
  Downloading
https://files.pythonhosted.org/packages/e1/e5/df302e8017440f111c11cc41a6b432838672f5a70aa29227bf58149dc72f/urllib3-1.25.9-py2.py3-none-any.whl
(126kB)
Collecting configparser (from awscurl)
  Downloading
https://files.pythonhosted.org/packages/4b/6b/01baa293090240cf0562cc5eccb69c6f5006282127f2b846fad011305c79/configparser-5.0.0-py3-none-any.whl
Collecting requests (from awscurl)
  Downloading
https://files.pythonhosted.org/packages/1a/70/1935c770cb3be6e3a8b78ced23d7e0f3b187f5cbfab4749523ed65d7c9b1/requests-2.23.0-py2.py3-none-any.whl
(58kB)
Collecting idna>=2.0.0; extra == "secure" (from urllib3[secure]->awscurl)
  Downloading
https://files.pythonhosted.org/packages/89/e3/afebe61c546d18fb1709a61bee788254b40e736cff7271c7de5de2dc4128/idna-2.9-py2.py3-none-any.whl
(58kB)
Collecting pyOpenSSL>=0.14; extra == "secure" (from
urllib3[secure]->awscurl)
  Downloading
https://files.pythonhosted.org/packages/9e/de/f8342b68fa9e981d348039954657bdf681b2ab93de27443be51865ffa310/pyOpenSSL-19.1.0-py2.py3-none-any.whl
(53kB)
Collecting cryptography>=1.3.4; extra == "secure" (from
urllib3[secure]->awscurl)
  Downloading
https://files.pythonhosted.org/packages/58/95/f1282ca55649b60afcf617e1e2ca384a2a3e7a5cf91f724cf83c8fbe76a1/cryptography-2.9.2-cp35-abi3-manylinux1_x86_64.whl
(2.7MB)
Collecting certifi; extra == "secure" (from urllib3[secure]->awscurl)
  Downloading
https://files.pythonhosted.org/packages/57/2b/26e37a4b034800c960a00c4e1b3d9ca5d7014e983e6e729e33ea2f36426c/certifi-2020.4.5.1-py2.py3-none-any.whl
(157kB)
Collecting chardet<4,>=3.0.2 (from requests->awscurl)
  Downloading
https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
(133kB)
Collecting six>=1.5.2 (from pyOpenSSL>=0.14; extra ==
"secure"->urllib3[secure]->awscurl)
  Downloading
https://files.pythonhosted.org/packages/65/eb/1f97cb97bfc2390a276969c6fae16075da282f5058082d4cb10c6c5c1dba/six-1.14.0-py2.py3-none-any.whl
Collecting cffi!=1.11.3,>=1.8 (from cryptography>=1.3.4; extra ==
"secure"->urllib3[secure]->awscurl)
  Downloading
https://files.pythonhosted.org/packages/ff/1d/0b743dadcdf4980b717163fd2d24cd18c1c0c7a78a076268afd7e0e2c25e/cffi-1.14.0-cp37-cp37m-manylinux1_x86_64.whl
(400kB)
Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography>=1.3.4; extra
== "secure"->urllib3[secure]->awscurl)
  Downloading
https://files.pythonhosted.org/packages/ae/e7/d9c3a176ca4b02024debf82342dab36efadfc5776f9c8db077e8f6e71821/pycparser-2.20-py2.py3-none-any.whl
(112kB)
Building wheels for collected packages: configargparse
  Running setup.py bdist_wheel for configargparse: started
  Running setup.py bdist_wheel for configargparse: finished with status
'done'
  Stored in directory:
/root/.cache/pip/wheels/bd/d6/53/034032da9498bda2385cd50a51a289e88090b5da2d592b1fdf
Successfully built configargparse
Installing collected packages: configargparse, idna, pycparser, cffi, six,
cryptography, pyOpenSSL, certifi, urllib3, configparser, chardet, requests,
awscurl
Successfully installed awscurl-0.20 certifi-2020.4.5.1 cffi-1.14.0
chardet-3.0.4 configargparse-1.2.3 configparser-5.0.0 cryptography-2.9.2
idna-2.9 pyOpenSSL-19.1.0 pycparser-2.20 requests-2.23.0 six-1.14.0
urllib3-1.25.9
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
Package python3-3.7.6-1.amzn2.0.1.x86_64 already installed and latest
version
Nothing to do
WARNING: Running pip install with root privileges is generally not a good
idea. Try `pip3 install --user` instead.
Requirement already satisfied: cryptography in
/usr/local/lib64/python3.7/site-packages
Requirement already satisfied: cffi!=1.11.3,>=1.8 in
/usr/local/lib64/python3.7/site-packages (from cryptography)
Requirement already satisfied: six>=1.4.1 in
/usr/local/lib/python3.7/site-packages (from cryptography)
Requirement already satisfied: pycparser in
/usr/local/lib/python3.7/site-packages (from
cffi!=1.11.3,>=1.8->cryptography)
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
Resolving Dependencies
--> Running transaction check
---> Package gcc.x86_64 0:7.3.1-6.amzn2.0.4 will be installed
--> Processing Dependency: cpp = 7.3.1-6.amzn2.0.4 for package:
gcc-7.3.1-6.amzn2.0.4.x86_64
--> Processing Dependency: glibc-devel >= 2.2.90-12 for package:
gcc-7.3.1-6.amzn2.0.4.x86_64
--> Processing Dependency: libubsan.so.0()(64bit) for package:
gcc-7.3.1-6.amzn2.0.4.x86_64
--> Processing Dependency: libtsan.so.0()(64bit) for package:
gcc-7.3.1-6.amzn2.0.4.x86_64
--> Processing Dependency: libquadmath.so.0()(64bit) for package:
gcc-7.3.1-6.amzn2.0.4.x86_64
--> Processing Dependency: libmpxwrappers.so.2()(64bit) for package:
gcc-7.3.1-6.amzn2.0.4.x86_64
--> Processing Dependency: libmpx.so.2()(64bit) for package:
gcc-7.3.1-6.amzn2.0.4.x86_64
--> Processing Dependency: libmpfr.so.4()(64bit) for package:
gcc-7.3.1-6.amzn2.0.4.x86_64
--> Processing Dependency: libmpc.so.3()(64bit) for package:
gcc-7.3.1-6.amzn2.0.4.x86_64
--> Processing Dependency: liblsan.so.0()(64bit) for package:
gcc-7.3.1-6.amzn2.0.4.x86_64
--> Processing Dependency: libitm.so.1()(64bit) for package:
gcc-7.3.1-6.amzn2.0.4.x86_64
--> Processing Dependency: libcilkrts.so.5()(64bit) for package:
gcc-7.3.1-6.amzn2.0.4.x86_64
--> Processing Dependency: libatomic.so.1()(64bit) for package:
gcc-7.3.1-6.amzn2.0.4.x86_64
--> Processing Dependency: libasan.so.4()(64bit) for package:
gcc-7.3.1-6.amzn2.0.4.x86_64
---> Package libcurl-devel.x86_64 0:7.61.1-12.amzn2.0.1 will be installed
---> Package openssl-devel.x86_64 1:1.0.2k-19.amzn2.0.3 will be installed
--> Processing Dependency: zlib-devel(x86-64) for package:
1:openssl-devel-1.0.2k-19.amzn2.0.3.x86_64
--> Processing Dependency: krb5-devel(x86-64) for package:
1:openssl-devel-1.0.2k-19.amzn2.0.3.x86_64
--> Running transaction check
---> Package cpp.x86_64 0:7.3.1-6.amzn2.0.4 will be installed
---> Package glibc-devel.x86_64 0:2.26-34.amzn2 will be installed
--> Processing Dependency: glibc-headers = 2.26-34.amzn2 for package:
glibc-devel-2.26-34.amzn2.x86_64
--> Processing Dependency: glibc-headers for package:
glibc-devel-2.26-34.amzn2.x86_64
---> Package krb5-devel.x86_64 0:1.15.1-37.amzn2.2.2 will be installed
--> Processing Dependency: libkadm5(x86-64) = 1.15.1-37.amzn2.2.2 for
package: krb5-devel-1.15.1-37.amzn2.2.2.x86_64
--> Processing Dependency: libverto-devel for package:
krb5-devel-1.15.1-37.amzn2.2.2.x86_64
--> Processing Dependency: libselinux-devel for package:
krb5-devel-1.15.1-37.amzn2.2.2.x86_64
--> Processing Dependency: libcom_err-devel for package:
krb5-devel-1.15.1-37.amzn2.2.2.x86_64
--> Processing Dependency: keyutils-libs-devel for package:
krb5-devel-1.15.1-37.amzn2.2.2.x86_64
---> Package libatomic.x86_64 0:7.3.1-6.amzn2.0.4 will be installed
---> Package libcilkrts.x86_64 0:7.3.1-6.amzn2.0.4 will be installed
---> Package libitm.x86_64 0:7.3.1-6.amzn2.0.4 will be installed
---> Package libmpc.x86_64 0:1.0.1-3.amzn2.0.2 will be installed
---> Package libmpx.x86_64 0:7.3.1-6.amzn2.0.4 will be installed
---> Package libquadmath.x86_64 0:7.3.1-6.amzn2.0.4 will be installed
---> Package libsanitizer.x86_64 0:7.3.1-6.amzn2.0.4 will be installed
---> Package mpfr.x86_64 0:3.1.1-4.amzn2.0.2 will be installed
---> Package zlib-devel.x86_64 0:1.2.7-18.amzn2 will be installed
--> Running transaction check
---> Package glibc-headers.x86_64 0:2.26-34.amzn2 will be installed
--> Processing Dependency: kernel-headers >= 2.2.1 for package:
glibc-headers-2.26-34.amzn2.x86_64
--> Processing Dependency: kernel-headers for package:
glibc-headers-2.26-34.amzn2.x86_64
---> Package keyutils-libs-devel.x86_64 0:1.5.8-3.amzn2.0.2 will be
installed
---> Package libcom_err-devel.x86_64 0:1.42.9-12.amzn2.0.2 will be installed
---> Package libkadm5.x86_64 0:1.15.1-37.amzn2.2.2 will be installed
---> Package libselinux-devel.x86_64 0:2.5-12.amzn2.0.2 will be installed
--> Processing Dependency: libsepol-devel(x86-64) >= 2.5-6 for package:
libselinux-devel-2.5-12.amzn2.0.2.x86_64
--> Processing Dependency: pkgconfig(libsepol) for package:
libselinux-devel-2.5-12.amzn2.0.2.x86_64
--> Processing Dependency: pkgconfig(libpcre) for package:
libselinux-devel-2.5-12.amzn2.0.2.x86_64
---> Package libverto-devel.x86_64 0:0.2.5-4.amzn2.0.2 will be installed
--> Running transaction check
---> Package kernel-headers.x86_64 0:4.14.173-137.229.amzn2 will be
installed
---> Package libsepol-devel.x86_64 0:2.5-8.1.amzn2.0.2 will be installed
---> Package pcre-devel.x86_64 0:8.32-17.amzn2.0.2 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                Arch      Version                   Repository
Size
================================================================================
Installing:
 gcc                    x86_64    7.3.1-6.amzn2.0.4         amzn2-core
22 M
 libcurl-devel          x86_64    7.61.1-12.amzn2.0.1       amzn2-core
 827 k
 openssl-devel          x86_64    1:1.0.2k-19.amzn2.0.3     amzn2-core
 1.5 M
Installing for dependencies:
 cpp                    x86_64    7.3.1-6.amzn2.0.4         amzn2-core
 9.2 M
 glibc-devel            x86_64    2.26-34.amzn2             amzn2-core
 989 k
 glibc-headers          x86_64    2.26-34.amzn2             amzn2-core
 510 k
 kernel-headers         x86_64    4.14.173-137.229.amzn2    amzn2-core
 1.1 M
 keyutils-libs-devel    x86_64    1.5.8-3.amzn2.0.2         amzn2-core
37 k
 krb5-devel             x86_64    1.15.1-37.amzn2.2.2       amzn2-core
 272 k
 libatomic              x86_64    7.3.1-6.amzn2.0.4         amzn2-core
44 k
 libcilkrts             x86_64    7.3.1-6.amzn2.0.4         amzn2-core
84 k
 libcom_err-devel       x86_64    1.42.9-12.amzn2.0.2       amzn2-core
31 k
 libitm                 x86_64    7.3.1-6.amzn2.0.4         amzn2-core
83 k
 libkadm5               x86_64    1.15.1-37.amzn2.2.2       amzn2-core
 179 k
 libmpc                 x86_64    1.0.1-3.amzn2.0.2         amzn2-core
52 k
 libmpx                 x86_64    7.3.1-6.amzn2.0.4         amzn2-core
50 k
 libquadmath            x86_64    7.3.1-6.amzn2.0.4         amzn2-core
 188 k
 libsanitizer           x86_64    7.3.1-6.amzn2.0.4         amzn2-core
 640 k
 libselinux-devel       x86_64    2.5-12.amzn2.0.2          amzn2-core
 187 k
 libsepol-devel         x86_64    2.5-8.1.amzn2.0.2         amzn2-core
77 k
 libverto-devel         x86_64    0.2.5-4.amzn2.0.2         amzn2-core
12 k
 mpfr                   x86_64    3.1.1-4.amzn2.0.2         amzn2-core
 208 k
 pcre-devel             x86_64    8.32-17.amzn2.0.2         amzn2-core
 480 k
 zlib-devel             x86_64    1.2.7-18.amzn2            amzn2-core
50 k

Transaction Summary
================================================================================
Install  3 Packages (+21 Dependent packages)

Total download size: 39 M
Installed size: 102 M
Downloading packages:
--------------------------------------------------------------------------------
Total                                               67 MB/s |  39 MB  00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : mpfr-3.1.1-4.amzn2.0.2.x86_64
1/24
  Installing : libmpc-1.0.1-3.amzn2.0.2.x86_64
2/24
  Installing : cpp-7.3.1-6.amzn2.0.4.x86_64
 3/24
  Installing : keyutils-libs-devel-1.5.8-3.amzn2.0.2.x86_64
 4/24
  Installing : pcre-devel-8.32-17.amzn2.0.2.x86_64
5/24
  Installing : libsanitizer-7.3.1-6.amzn2.0.4.x86_64
6/24
  Installing : libatomic-7.3.1-6.amzn2.0.4.x86_64
 7/24
  Installing : zlib-devel-1.2.7-18.amzn2.x86_64
 8/24
  Installing : libcom_err-devel-1.42.9-12.amzn2.0.2.x86_64
9/24
  Installing : libverto-devel-0.2.5-4.amzn2.0.2.x86_64
 10/24
  Installing : libcilkrts-7.3.1-6.amzn2.0.4.x86_64
 11/24
  Installing : libsepol-devel-2.5-8.1.amzn2.0.2.x86_64
 12/24
  Installing : libselinux-devel-2.5-12.amzn2.0.2.x86_64
13/24
  Installing : libkadm5-1.15.1-37.amzn2.2.2.x86_64
 14/24
  Installing : krb5-devel-1.15.1-37.amzn2.2.2.x86_64
 15/24
  Installing : libitm-7.3.1-6.amzn2.0.4.x86_64
 16/24
  Installing : kernel-headers-4.14.173-137.229.amzn2.x86_64
17/24
  Installing : glibc-headers-2.26-34.amzn2.x86_64
18/24
  Installing : glibc-devel-2.26-34.amzn2.x86_64
19/24
  Installing : libmpx-7.3.1-6.amzn2.0.4.x86_64
 20/24
  Installing : libquadmath-7.3.1-6.amzn2.0.4.x86_64
21/24
  Installing : gcc-7.3.1-6.amzn2.0.4.x86_64
22/24
  Installing : 1:openssl-devel-1.0.2k-19.amzn2.0.3.x86_64
23/24
  Installing : libcurl-devel-7.61.1-12.amzn2.0.1.x86_64
24/24
  Verifying  : gcc-7.3.1-6.amzn2.0.4.x86_64
 1/24
  Verifying  : libcurl-devel-7.61.1-12.amzn2.0.1.x86_64
 2/24
  Verifying  : libquadmath-7.3.1-6.amzn2.0.4.x86_64
 3/24
  Verifying  : glibc-headers-2.26-34.amzn2.x86_64
 4/24
  Verifying  : libmpx-7.3.1-6.amzn2.0.4.x86_64
5/24
  Verifying  : kernel-headers-4.14.173-137.229.amzn2.x86_64
 6/24
  Verifying  : libmpc-1.0.1-3.amzn2.0.2.x86_64
7/24
  Verifying  : 1:openssl-devel-1.0.2k-19.amzn2.0.3.x86_64
 8/24
  Verifying  : libitm-7.3.1-6.amzn2.0.4.x86_64
9/24
  Verifying  : glibc-devel-2.26-34.amzn2.x86_64
10/24
  Verifying  : libkadm5-1.15.1-37.amzn2.2.2.x86_64
 11/24
  Verifying  : libsepol-devel-2.5-8.1.amzn2.0.2.x86_64
 12/24
  Verifying  : libcilkrts-7.3.1-6.amzn2.0.4.x86_64
 13/24
  Verifying  : libverto-devel-0.2.5-4.amzn2.0.2.x86_64
 14/24
  Verifying  : libcom_err-devel-1.42.9-12.amzn2.0.2.x86_64
 15/24
  Verifying  : zlib-devel-1.2.7-18.amzn2.x86_64
16/24
  Verifying  : mpfr-3.1.1-4.amzn2.0.2.x86_64
 17/24
  Verifying  : libatomic-7.3.1-6.amzn2.0.4.x86_64
18/24
  Verifying  : libselinux-devel-2.5-12.amzn2.0.2.x86_64
19/24
  Verifying  : krb5-devel-1.15.1-37.amzn2.2.2.x86_64
 20/24
  Verifying  : libsanitizer-7.3.1-6.amzn2.0.4.x86_64
 21/24
  Verifying  : pcre-devel-8.32-17.amzn2.0.2.x86_64
 22/24
  Verifying  : keyutils-libs-devel-1.5.8-3.amzn2.0.2.x86_64
23/24
  Verifying  : cpp-7.3.1-6.amzn2.0.4.x86_64
24/24

Installed:
  gcc.x86_64 0:7.3.1-6.amzn2.0.4
  libcurl-devel.x86_64 0:7.61.1-12.amzn2.0.1
  openssl-devel.x86_64 1:1.0.2k-19.amzn2.0.3

Dependency Installed:
  cpp.x86_64 0:7.3.1-6.amzn2.0.4
  glibc-devel.x86_64 0:2.26-34.amzn2
  glibc-headers.x86_64 0:2.26-34.amzn2
  kernel-headers.x86_64 0:4.14.173-137.229.amzn2
  keyutils-libs-devel.x86_64 0:1.5.8-3.amzn2.0.2
  krb5-devel.x86_64 0:1.15.1-37.amzn2.2.2
  libatomic.x86_64 0:7.3.1-6.amzn2.0.4
  libcilkrts.x86_64 0:7.3.1-6.amzn2.0.4
  libcom_err-devel.x86_64 0:1.42.9-12.amzn2.0.2
  libitm.x86_64 0:7.3.1-6.amzn2.0.4
  libkadm5.x86_64 0:1.15.1-37.amzn2.2.2
  libmpc.x86_64 0:1.0.1-3.amzn2.0.2
  libmpx.x86_64 0:7.3.1-6.amzn2.0.4
  libquadmath.x86_64 0:7.3.1-6.amzn2.0.4
  libsanitizer.x86_64 0:7.3.1-6.amzn2.0.4
  libselinux-devel.x86_64 0:2.5-12.amzn2.0.2
  libsepol-devel.x86_64 0:2.5-8.1.amzn2.0.2
  libverto-devel.x86_64 0:0.2.5-4.amzn2.0.2
  mpfr.x86_64 0:3.1.1-4.amzn2.0.2
  pcre-devel.x86_64 0:8.32-17.amzn2.0.2
  zlib-devel.x86_64 0:1.2.7-18.amzn2

Complete!
WARNING: Running pip install with root privileges is generally not a good
idea. Try `pip3 install --user` instead.
Collecting apache-airflow[celery,crypto,postgres,s3]==1.10.9
...
villasv commented 4 years ago

After fixing #168, #169 and #174, I've deployed a test stack on us-west-1 and it worked properly. If you could try again, maybe we can find the issue more easily. Hopefully it will work right away.

ztyree42 commented 4 years ago

I'm failing to create VPCStack resource w/ following status reason:

Embedded stack
arn:aws:cloudformation:us-west-1:698886251042:stack/turbine-test-VPCStack-IPZX79T73OPW/b3c924a0-8826-11ea-bb51-025517c8b4d7
was not successfully created: The following resource(s) failed to create:
[NAT1EIP, NAT2EIP, PrivateSubnet2ARouteTableAssociation, PublicSubnetRoute,
PrivateSubnet1ARouteTableAssociation].

This is an attempt to use the "Launch Stack" button.

On Sun, Apr 26, 2020 at 5:45 PM Victor Villas notifications@github.com wrote:

After fixing #168 https://github.com/villasv/aws-airflow-stack/issues/168, #169 https://github.com/villasv/aws-airflow-stack/issues/169 and #174 https://github.com/villasv/aws-airflow-stack/issues/174, I've deployed a test stack on us-west-1 and it worked properly. If you could try again, maybe we can find the issue more easily. Hopefully it will work right away.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/villasv/aws-airflow-stack/issues/170#issuecomment-619652551, or unsubscribe https://github.com/notifications/unsubscribe-auth/AECDZPLZTJPLHTPBDIXFW3TROTIRFANCNFSM4MQQJXBQ .

villasv commented 4 years ago

Hmm, there's probably a more detailed error description on one of those resources that should pinpoint the culprit. My guess is that you're hitting some resource limit for your account (happened to me once).

ztyree42 commented 4 years ago

It's working! Successfully deployed into my own VPC, if you like I can share the parameters file that will allow you to connect it to an arbitrary vpc in case there are others like me that want the cluster stack but already have a decent amount of infrastructure built w/ the aws UI (and thus hard to port over to CFN). Thanks so much for your help!

villasv commented 4 years ago

Congrats! I absolutely want it. If it wasn't trivial, I want to know why and the nuts and bolts you had to adjust. I guess I also need to document better the pre-conditions for the deployment in existing VPCs (e.g. subnets in at least two AZs), so this is a good opportunity to kickstart that.

ilyanoskov commented 4 years ago

@ztyree42 can you please share your config file? I am currently experiencing the same issues on eu-central-1, @villasv

ztyree42 commented 4 years ago

Sorry for late replay, what specifically would you like me to share? The parameters being set in the .template files? The .template files themselves?

On Sat, May 16, 2020 at 7:52 AM Ilya Noskov notifications@github.com wrote:

@ztyree42 https://github.com/ztyree42 can you please share your config file? I am currently experiencing the same issues on eu-central-1, @villasv https://github.com/villasv

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/villasv/aws-airflow-stack/issues/170#issuecomment-629658212, or unsubscribe https://github.com/notifications/unsubscribe-auth/AECDZPJCVOLAIXTPXWDU323RR2SELANCNFSM4MQQJXBQ .