search

28mm / blast-radius

Interactive visualizations of Terraform dependency graphs using d3.js
https://28mm.github.io/blast-radius-docs/
MIT License
2.02k stars 256 forks source link

HCL Parser Error #28

Open ChairmanTubeAmp opened 6 years ago

ChairmanTubeAmp commented 6 years ago

I get the following error with blast-radius 0.1.12:

A debug mode would be helpful for finding the file that the parser doesn't like. Terraform plan/apply works fine so it's happy with the HCL blast-radius is parsing.

28mm commented 6 years ago

@ChairmanTubeAmp thanks for reporting. I think a debug mode will be a good idea. When I've had issues with PyHCL in the past, I've tried to isolate the file and line number, with the snippet below.

from glob import iglob
import hcl

for filename in iglob('**.tf'):
    with open(filename, 'r') as f:
        print(filename)
        hcl.loads(f.read())

Maybe the debug mode should fall back to something like this. Curious to know how the unexpected RIGHTBRACE comes up in your case.

robertpeteuil commented 6 years ago

I'm getting a similar error. It occurs whether blast-radius is locally installed, using the official docker image or using a custom docker image with a newer version of terraform (0.11.7).

terraform init, plan, graph, deploy and destroy all work fine. So I'm unsure what is causing the issues.

Here the full error log:

 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
172.17.0.1 - - [11/May/2018 23:37:26] "GET / HTTP/1.1" 200 -
172.17.0.1 - - [11/May/2018 23:37:30] "GET /graph.svg HTTP/1.1" 200 -
[2018-05-11 23:37:32,840] ERROR in app: Exception on /graph.json [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/src/blastradius/server/server.py", line 57, in graph_json
    tf = Terraform(os.getcwd())
  File "/src/blastradius/handlers/terraform.py", line 52, in __init__
    self.modules[name] = Terraform(directory=self.directory+'/'+source, settings=mod)
  File "/src/blastradius/handlers/terraform.py", line 26, in __init__
    self.config = hcl.load(config_io)
  File "/usr/local/lib/python3.6/dist-packages/hcl/api.py", line 51, in load
    return loads(fp.read())
  File "/usr/local/lib/python3.6/dist-packages/hcl/api.py", line 61, in loads
    if isHcl(s):
  File "/usr/local/lib/python3.6/dist-packages/hcl/api.py", line 39, in isHcl
    raise ValueError("No HCL object could be decoded")
ValueError: No HCL object could be decoded
172.17.0.1 - - [11/May/2018 23:37:32] "GET /graph.json HTTP/1.1" 500 -
28mm commented 6 years ago

@robertpeteuil

I regret the useless error message. This happens when (a) PyHCL fails to parse a terraform config (unlikely), (b) when a module is pulled in from a source blast-radius doesn't know how to handle, or (c) some unknown/uncharacterized condition. Hopefully (b), since it offers the easiest path to resolution :)

Does this configuration pull modules from anywhere interesting (ie not the local filesystem?)

Thanks, Patrick.

robertpeteuil commented 6 years ago

Ahh that makes sense.

Yes, I tried a few different configurations and they both use modules from the "terraform module registry". Their source is something like source = "terraform-aws-modules/security-group/aws"

So if I make local copies of those on the file system, it may work?

Thanks for the quick reply! Robert

28mm commented 6 years ago

@robertpeteuil

Yes--or add another condition here. I'm reluctant to try a patch from mobile, but otherwise happy to commit tomorrow. That whole block needs to be reconciled with the various module sources Terraform supports.

Regret the hassle :)

robertpeteuil commented 6 years ago

@28mm

That great news, thanks! No hurries on the commit, as I don't need to create a chart until later next week. I might also replace the module with direct resources as it may simplify the dependency chart.

While I have you here - I do have a feature request (I can move this to a separate issue if you want)

Thanks again for the quick response & triage.

28mm commented 6 years ago

@robertpeteuil that makes a lot of sense, thanks for supplying a use case :)

I'm thinking, additionally, of using a split-dropdown component, so that you can disable/re-enable tooltips in a single click.

robertpeteuil commented 6 years ago

Thanks for the commit!

I'll test this out in the next couple of days with some of my more complex configurations.

The split-dropdown component sounds awesome.

i5okie commented 6 years ago

I get this error too:

[2018-07-04 16:27:04,222] ERROR in app: Exception on /graph.json [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/src/blastradius/server/server.py", line 57, in graph_json
    tf = Terraform(os.getcwd())
  File "/src/blastradius/handlers/terraform.py", line 57, in __init__
    self.modules[name] = Terraform(directory=self.directory+'/'+source, settings=mod)
  File "/src/blastradius/handlers/terraform.py", line 26, in __init__
    self.config = hcl.load(config_io)
  File "/usr/local/lib/python3.6/dist-packages/hcl/api.py", line 51, in load
    return loads(fp.read())
  File "/usr/local/lib/python3.6/dist-packages/hcl/api.py", line 61, in loads
    if isHcl(s):
  File "/usr/local/lib/python3.6/dist-packages/hcl/api.py", line 39, in isHcl
    raise ValueError("No HCL object could be decoded")
ValueError: No HCL object could be decoded
172.17.0.1 - - [04/Jul/2018 16:27:04] "GET /graph.json HTTP/1.1" 500 -
gugalnikov commented 5 years ago

Hi, the docker image used to be working fine for me a few weeks ago, but now the visualization fails with the following error when I run it against the same codebase in same PC:

28mm commented 5 years ago

@gugalnikov

Huh. My first thought was something different about PyHCL, or PLY, but it doesn't look like PyHCL has updated since February 2018 https://pypi.org/project/pyhcl/#history (and it locks the PLY version).

It would be helpful to narrow down the file and line # where the error is encountered. The following should do the job...

Presumes pyhcl is installed:

pip3 install pyhcl

Then from a script or python shell:

from glob import iglob
import hcl

for filename in iglob('**.tf'):
    with open(filename, 'r') as f:
        print(filename)
        hcl.loads(f.read())
cappetta commented 5 years ago

@28mm seeing this issue arise on docker as a first time user. Tried to install the packages via pip and that was hitting a separate error. Looks like a great little package. Here's the output that I have from the cmd above...

output.tf {'output': {'web-alb-dns-name': {'value': '${module.instances.web-alb-dns-name}'}, 'web-instance-ips': {'value': '${module.instances.web-instance-ips}'}}} main.tf {'module': {'network': {'source': 'modules/network', 'environment': '${var.environment}'}, 'secdevops': {'vpc-cidr-block': '${module.network.vpc-cidr-block}', 'private-a-subnet-id': '${module.network.private-a-subnet-id}', 'source': 'modules/secdevops', 'private-b-subnet-id': '${module.network.private-b-subnet-id}', 'environment': '${var.environment}', 'public-a-subnet-id': '${module.network.public-a-subnet-id}', 'vpc-id': '${module.network.vpc-id}', 'public-b-subnet-id': '${module.network.public-b-subnet-id}'}, 'instances': {'vpc-cidr-block': '${module.network.vpc-cidr-block}', 'db_endpoint': '${module.databases.db_endpoint}', 'source': 'modules/instances', 'environment': '${var.environment}', 'public-a-subnet-id': '${module.network.public-a-subnet-id}', 'vpc-id': '${module.network.vpc-id}', 'public-b-subnet-id': '${module.network.public-b-subnet-id}'}, 'databases': {'vpc-cidr-block': '${module.network.vpc-cidr-block}', 'source': 'modules/databases', 'private-b-subnet-id': '${module.network.private-b-subnet-id}', 'environment': '${var.environment}', 'vpc-id': '${module.network.vpc-id}', 'private-a-subnet-id': '${module.network.private-a-subnet-id}'}}} variables.tf {'variable': {'environment': {}}}

I'm using a customized derivative of https://github.com/fedekau/terraform-with-circleci-example

abitrolly commented 5 years ago

Running with --json flag strangely gives the same error.

✗ terraform graph -draw-cycles | podman run -i --entrypoint blast-radius 28mm/blast-radius --json
Traceback (most recent call last):
  File "/usr/local/bin/blast-radius", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/src/bin/blast-radius", line 96, in <module>
    main()
  File "/src/bin/blast-radius", line 81, in main
    tf = Terraform(args.directory)
  File "/src/blastradius/handlers/terraform.py", line 26, in __init__
    self.config = hcl.load(config_io)
  File "/usr/local/lib/python3.6/dist-packages/hcl/api.py", line 54, in load
    return loads(fp.read())
  File "/usr/local/lib/python3.6/dist-packages/hcl/api.py", line 65, in loads
    if isHcl(s):
  File "/usr/local/lib/python3.6/dist-packages/hcl/api.py", line 42, in isHcl
    raise ValueError("No HCL object could be decoded")
ValueError: No HCL object could be decoded