Open mattsawyer77 opened 5 years ago
Getting this as well, I actually had an error manifest when installing the blastradius package, probably related:
$ pip3 install blastradius
Collecting blastradius
Downloading https://files.pythonhosted.org/packages/19/5a/154aa8523324dc3ffb6e2f6daeab0c34363bfe07b5405d61ffdb01e6f75d/BlastRadius-0.1.23.tar.gz (587kB)
100% |████████████████████████████████| 593kB 7.3MB/s
Collecting Flask (from blastradius)
Downloading https://files.pythonhosted.org/packages/9a/74/670ae9737d14114753b8c8fdf2e8bd212a05d3b361ab15b44937dfd40985/Flask-1.0.3-py2.py3-none-any.whl (92kB)
100% |████████████████████████████████| 92kB 621kB/s
Requirement already satisfied: jinja2 in /usr/local/lib/python3.7/site-packages (from blastradius) (2.10.1)
Collecting pyhcl>=0.3.10 (from blastradius)
Downloading https://files.pythonhosted.org/packages/8c/90/411f698550155532a1375d0367da08c2d0ecc922d4c0362bd4236893238b/pyhcl-0.3.12.tar.gz
Collecting requests (from blastradius)
Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)
100% |████████████████████████████████| 61kB 3.7MB/s
Collecting BeautifulSoup4 (from blastradius)
Downloading https://files.pythonhosted.org/packages/1d/5d/3260694a59df0ec52f8b4883f5d23b130bc237602a1411fa670eae12351e/beautifulsoup4-4.7.1-py3-none-any.whl (94kB)
100% |████████████████████████████████| 102kB 683kB/s
Collecting Werkzeug>=0.14 (from Flask->blastradius)
Downloading https://files.pythonhosted.org/packages/9f/57/92a497e38161ce40606c27a86759c6b92dd34fcdb33f64171ec559257c02/Werkzeug-0.15.4-py2.py3-none-any.whl (327kB)
100% |████████████████████████████████| 327kB 3.7MB/s
Collecting itsdangerous>=0.24 (from Flask->blastradius)
Downloading https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl
Collecting click>=5.1 (from Flask->blastradius)
Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB)
100% |████████████████████████████████| 81kB 5.7MB/s
Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/site-packages (from jinja2->blastradius) (1.1.1)
Collecting ply<4,>=3.8 (from pyhcl>=0.3.10->blastradius)
Downloading https://files.pythonhosted.org/packages/a3/58/35da89ee790598a0700ea49b2a66594140f44dec458c07e8e3d4979137fc/ply-3.11-py2.py3-none-any.whl (49kB)
100% |████████████████████████████████| 51kB 5.7MB/s
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/site-packages (from requests->blastradius) (1.25.3)
Collecting certifi>=2017.4.17 (from requests->blastradius)
Downloading https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl (158kB)
100% |████████████████████████████████| 163kB 3.5MB/s
Collecting idna<2.9,>=2.5 (from requests->blastradius)
Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)
100% |████████████████████████████████| 61kB 7.9MB/s
Collecting chardet<3.1.0,>=3.0.2 (from requests->blastradius)
Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
100% |████████████████████████████████| 143kB 2.0MB/s
Collecting soupsieve>=1.2 (from BeautifulSoup4->blastradius)
Downloading https://files.pythonhosted.org/packages/b9/a5/7ea40d0f8676bde6e464a6435a48bc5db09b1a8f4f06d41dd997b8f3c616/soupsieve-1.9.1-py2.py3-none-any.whl
Building wheels for collected packages: blastradius, pyhcl
Building wheel for blastradius (setup.py) ... done
Stored in directory: /Users/connerswann/Library/Caches/pip/wheels/80/56/24/cb2935009cd7da82c49161ef1ac239654046c3a8d22c370c53
Building wheel for pyhcl (setup.py) ... error
Complete output from command /usr/local/opt/python/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/rb/wg8c4zfs3n18cnt6v_fk9ck00000gn/T/pip-install-ges1al1l/pyhcl/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /private/var/folders/rb/wg8c4zfs3n18cnt6v_fk9ck00000gn/T/pip-wheel-9wfnetp_ --python-tag cp37:
running bdist_wheel
running build
running build_py
Generating parse table...
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/folders/rb/wg8c4zfs3n18cnt6v_fk9ck00000gn/T/pip-install-ges1al1l/pyhcl/setup.py", line 101, in <module>
"Topic :: Text Processing",
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/local/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 192, in run
self.run_command('build')
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/private/var/folders/rb/wg8c4zfs3n18cnt6v_fk9ck00000gn/T/pip-install-ges1al1l/pyhcl/setup.py", line 39, in run
self.execute(_pre_install, (), msg="Generating parse table...")
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/cmd.py", line 335, in execute
util.execute(func, args, msg, dry_run=self.dry_run)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/util.py", line 291, in execute
func(*args)
File "/private/var/folders/rb/wg8c4zfs3n18cnt6v_fk9ck00000gn/T/pip-install-ges1al1l/pyhcl/setup.py", line 31, in _pre_install
import hcl
File "/private/var/folders/rb/wg8c4zfs3n18cnt6v_fk9ck00000gn/T/pip-install-ges1al1l/pyhcl/src/hcl/__init__.py", line 1, in <module>
from .api import dumps, load, loads
File "/private/var/folders/rb/wg8c4zfs3n18cnt6v_fk9ck00000gn/T/pip-install-ges1al1l/pyhcl/src/hcl/api.py", line 2, in <module>
from .parser import HclParser
File "/private/var/folders/rb/wg8c4zfs3n18cnt6v_fk9ck00000gn/T/pip-install-ges1al1l/pyhcl/src/hcl/parser.py", line 4, in <module>
from .lexer import Lexer
File "/private/var/folders/rb/wg8c4zfs3n18cnt6v_fk9ck00000gn/T/pip-install-ges1al1l/pyhcl/src/hcl/lexer.py", line 3, in <module>
import ply.lex as lex
ModuleNotFoundError: No module named 'ply'
----------------------------------------
Failed building wheel for pyhcl
Running setup.py clean for pyhcl
Successfully built blastradius
Failed to build pyhcl
Installing collected packages: Werkzeug, itsdangerous, click, Flask, ply, pyhcl, certifi, idna, chardet, requests, soupsieve, BeautifulSoup4, blastradius
Running setup.py install for pyhcl ... done
Successfully installed BeautifulSoup4-4.7.1 Flask-1.0.3 Werkzeug-0.15.4 blastradius-0.1.23 certifi-2019.3.9 chardet-3.0.4 click-7.0 idna-2.8 itsdangerous-1.1.0 ply-3.11 pyhcl-0.3.12 requests-2.22.0 soupsieve-1.9.1
@yourbuddyconner please have a look at https://github.com/obourdon/blast-radius/commit/2ed019146a3910315d701b20850e9050af1d0c43 which should fix your issue
@mattsawyer77 I found it very difficult to run this great piece of software which is blast-radius natively on my systems due to all these messy environments where python versions and dependencies are not always what you would expect them to be (both MacOSX and Linux [several flavours]). However using the docker container to do so worked like a charm. You can also have a look in my own repository where I have added a couple of fixes and potential enhancements for this docker container.
@obourdon I think you're definitely on the right track here, I'll fiddle with it!
Please note that the only "messy part" with docker container, can be if, like me, you are using remote states (S3 bucket+DynamoDB locks) as you'll have to pay close attention to the volumes you mount to be able to access the proper infos
@obourdon Literally running into that problem right now, considering mounting the ~/.aws
directory into the container, is that what you have done to pass credentials?
EDIT: this worked exactly how I expected it would. I cloned the repo, made minor updates as reflected in Oliver's linked commit above, built the container, and ran this command:
docker run --security-opt apparmor:unconfined --cap-add=SYS_ADMIN -it --rm -p 5000:5000 -v ~/.aws:/root/.aws -v $(pwd):/workdir:ro 28mm/blast-radius --serve path/to/terraform/stack
Still running into the error @mattsawyer77 encountered though, so not 100% working.
Relevant error:
Error loading configuration: Error parsing <SNIP>: At 43:15: Unknown token: 43:15 IDENT data.aws_secretsmanager_secret.elasticsearch_additional_ips.id
Coincidentally, due to my plugins being out of date, my IDE tells me this this line is wrong when in reality it's perfectly valid in TF 0.12.x:
data "aws_secretsmanager_secret_version" "selected" {
secret_id = data.aws_secretsmanager_secret.elasticsearch_additional_ips.id
}
When changed to the following, the error in blast-radius goes away:
data "aws_secretsmanager_secret_version" "selected" {
secret_id = "${data.aws_secretsmanager_secret.elasticsearch_additional_ips.id}"
}
And I run into another one stemming from my use of the compact
function which is valid in terraform 0.12.x.
This leads me to believe it's just an out of date HCL parser that doesn't support TF 0.12.x.
@yourbuddyconner very good analysis (and usage of .aws bind mount) and I guess the error comes from this line in the pyhcl module which has already been reported as issue #55 a few days ago.
I used the container cmd to run, but it still have the same error:
$ docker run --rm -it -p 5000:5000 -v $(pwd):/data:ro --security-opt apparmor:unconfined --cap-add=SYS_ADMIN terraform-blast-radius-build-1Initializing modules...
Initializing the backend...
Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "aws" (terraform-providers/aws) 2.23.0...
- Downloading plugin for provider "google" (terraform-providers/google) 2.12.0...
The following providers do not have any version constraints in configuration,
so the latest version was installed.
To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.
* provider.aws: version = "~> 2.23"
* provider.google: version = "~> 2.12"
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
* Serving Flask app "blastradius.server.server" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
* Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
172.17.0.1 - - [13/Aug/2019 21:00:14] "GET / HTTP/1.1" 200 -
172.17.0.1 - - [13/Aug/2019 21:00:14] "GET /static/css/bootstrap.min.css HTTP/1.1" 200 -
172.17.0.1 - - [13/Aug/2019 21:00:15] "GET /favicon.ico HTTP/1.1" 404 -
172.17.0.1 - - [13/Aug/2019 21:00:19] "GET /graph.svg HTTP/1.1" 200 -
[2019-08-13 21:00:24,238] ERROR in app: Exception on /graph.json [GET]
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2311, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1834, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1737, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 36, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1832, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1818, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/src/blastradius/server/server.py", line 61, in graph_json
tf = Terraform(os.getcwd())
File "/src/blastradius/handlers/terraform.py", line 26, in __init__
self.config = hcl.load(config_io)
File "/usr/local/lib/python3.7/site-packages/hcl/api.py", line 54, in load
return loads(fp.read())
File "/usr/local/lib/python3.7/site-packages/hcl/api.py", line 66, in loads
return HclParser().parse(s)
File "/usr/local/lib/python3.7/site-packages/hcl/parser.py", line 326, in parse
return self.yacc.parse(s, lexer=Lexer())
File "/usr/local/lib/python3.7/site-packages/ply/yacc.py", line 333, in parse
return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
File "/usr/local/lib/python3.7/site-packages/ply/yacc.py", line 1201, in parseopt_notrack
tok = call_errorfunc(self.errorfunc, errtoken, self)
File "/usr/local/lib/python3.7/site-packages/ply/yacc.py", line 192, in call_errorfunc
r = errorfunc(token)
File "/usr/local/lib/python3.7/site-packages/hcl/parser.py", line 318, in p_error
raise ValueError(msg)
ValueError: Line 4, column 61: unexpected IDENTIFIER
172.17.0.1 - - [13/Aug/2019 21:00:24] "GET /graph.json HTTP/1.1" 500 -
172.17.0.1 - - [13/Aug/2019 21:04:11] "GET /static/css/bootstrap.min.css HTTP/1.1" 200 -
172.17.0.1 - - [13/Aug/2019 21:04:12] "GET /static/css/selectize.css HTTP/1.1" 200 -
172.17.0.1 - - [13/Aug/2019 21:04:12] "GET /static/css/style.css HTTP/1.1" 200 -
Did anyone who used the Docker Image get around this issue / work out what caused it? It seems ok to me when using on small TF projects, but occurs when using it on bigger projects.
@mattsawyer77 I found it very difficult to run this great piece of software which is blast-radius natively on my systems due to all these messy environments where python versions and dependencies are not always what you would expect them to be (both MacOSX and Linux [several flavours]). However using the docker container to do so worked like a charm. You can also have a look in my own repository where I have added a couple of fixes and potential enhancements for this docker container.
@obourdon I used your repo to run and got the errors,
172.17.0.1 - - [14/May/2020 08:31:00] "GET / HTTP/1.1" 200 - [2020-05-14 08:31:00,202] ERROR in app: Exception on /graph.svg [GET] Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2447, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1952, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1821, in handle_user_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 39, in reraise raise value File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1950, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1936, in dispatch_request return self.view_functionsrule.endpoint File "/src/blastradius/server/server.py", line 37, in graph_svg dot = DotGraph('', file_contents=run_tf_graph()) File "/src/blastradius/server/server.py", line 76, in run_tf_graph raise Exception('Execution error', completed.stderr) Exception: ('Execution error', b'\x1b[31m\n\x1b[1m\x1b[31mError: \x1b[0m\x1b[0m\x1b[1mError reading /workdir-rw/main.tf: open /workdir-rw/main.tf: no such file or directory\x1b[0m\n\n\x1b[0m\x1b[0m\x1b[0m\n') 172.17.0.1 - - [14/May/2020 08:31:00] "GET /graph.svg HTTP/1.1" 500 -
I'm having the same problem with this main.tf
provider "aws" {
region = var.aws_region
access_key = var.access_key
secret_key = var.secret_key
}
data "aws_availability_zones" "available" {
state = "available"
}
terraform {
required_version = "0.12.3"
}
resource "aws_vpc" "my_vpc" {
cidr_block = "10.0.0.0/16"
enable_dns_hostnames = true
enable_dns_support = true
tags = {
Name = "${var.name_prefix}-vpc"
}
}
#internet gateway
resource "aws_internet_gateway" "my_internet_gateway" {
vpc_id = aws_vpc.my_vpc.id
tags = {
Name = "${var.name_prefix}-internet_gateway"
}
}
# Route tables
resource "aws_default_route_table" "my_private_rt" {
default_route_table_id = aws_vpc.my_vpc.default_route_table_id
route {
cidr_block = "0.0.0.0/0"
nat_gateway_id = aws_nat_gateway.my-nat.id
}
tags = {
Name = "${var.name_prefix}-private-rTable"
}
}
resource "aws_subnet" "my_public_subnet" {
vpc_id = aws_vpc.my_vpc.id
cidr_block = "10.0.1.0/24"
map_public_ip_on_launch = true
availability_zone = data.aws_availability_zones.available.names[1]
tags = {
Name = "${var.name_prefix}-_public2"
}
}
resource "aws_subnet" "my_private1_subnet" {
vpc_id = aws_vpc.my_vpc.id
cidr_block = "10.0.2.0/24"
map_public_ip_on_launch = false
availability_zone = data.aws_availability_zones.available.names[0]
tags = {
Name = "${var.name_prefix}-_private1"
}
}
resource "aws_subnet" "my_private2_subnet" {
vpc_id = aws_vpc.my_vpc.id
cidr_block = "10.0.3.0/24"
map_public_ip_on_launch = false
availability_zone = data.aws_availability_zones.available.names[1]
tags = {
Name = "${var.name_prefix}-_private2"
}
}
resource "aws_subnet" "my_rds1_subnet" {
vpc_id = aws_vpc.my_vpc.id
cidr_block = "10.0.4.0/24"
map_public_ip_on_launch = false
availability_zone = data.aws_availability_zones.available.names[1]
tags = {
Name = "${var.name_prefix}-_rds1"
}
}
resource "aws_subnet" "my_rds2_subnet" {
vpc_id = aws_vpc.my_vpc.id
cidr_block = "10.0.5.0/24"
map_public_ip_on_launch = false
availability_zone = data.aws_availability_zones.available.names[2]
tags = {
Name = "${var.name_prefix}-_rds2"
}
}
resource "aws_db_subnet_group" "my_rds_subnetgroup" {
name = lower("${var.name_prefix}-subnetgroup-rds")
subnet_ids = [
aws_subnet.my_rds1_subnet.id,
aws_subnet.my_rds2_subnet.id
]
tags = {
Name = "${var.name_prefix}-rds_subnetgroup"
}
}
whether I use the docker container or the native command blast-radius
I always get:
tok = call_errorfunc(self.errorfunc, errtoken, self)
File "/usr/local/lib/python3.7/site-packages/ply/yacc.py", line 192, in call_errorfunc
r = errorfunc(token)
File "/usr/local/lib/python3.7/site-packages/hcl/parser.py", line 318, in p_error
raise ValueError(msg)
ValueError: Line 2, column 32: unexpected IDENTIFIER
172.17.0.1 - - [05/Jul/2020 18:50:08] "GET /graph.json HTTP/1.1" 500 -
I have ran into the same issue.
The issue is indeed that variables must be surrounded by "${}"
For exemple,
vpc_id = aws_vpc.my_vpc.id
needs to be changed to:
vpc_id = "${aws_vpc.my_vpc.id}"
I don't know the effort to make this work, but it would be great. Thanks
Hello, I'm trying this out with a terraform 0.12 project (which I suspect could be the root issue). The terraform is planned/applied successfully. The blast-radius graph draws on the screen but is not interactive. The blast-radius server shows this output:
Is there a way to find the unexpected identifier to debug further? I tried running
blast-radius
with--debug
but it was not a valid flag.