Error during prepare due to missing ? in shared/node.py

[samyongsj]

Getting error during prepare Traceback (most recent call last): File "cloudmapper.py", line 72, in <module> main() File "cloudmapper.py", line 66, in main commands[command].run(arguments) File "/home/ubuntu/Documents/cloudmapper/commands/prepare.py", line 819, in run prepare(account, config, outputfilter) File "/home/ubuntu/Documents/cloudmapper/commands/prepare.py", line 660, in prepare cytoscape_json = build_data_structure(account, config, outputfilter) File "/home/ubuntu/Documents/cloudmapper/commands/prepare.py", line 477, in build_data_structure add_node_to_subnets(region, node, nodes) File "/home/ubuntu/Documents/cloudmapper/commands/prepare.py", line 329, in add_node_to_subnets if len(node.subnets) == 0 and vpc.local_id == node._parent.local_id: File "/home/ubuntu/Documents/cloudmapper/shared/nodes.py", line 499, in subnets ".DBSubnetGroup.Subnets[].SubnetIdentifier", self._json_blob File "/home/ubuntu/.local/share/virtualenvs/cloudmapper-BV2qN1UK/lib/python3.7/site-packages/pyjq.py", line 49, in all return compile(script, vars, library_paths).all(_get_value(value, url, opener)) File "_pyjq.pyx", line 209, in _pyjq.Script.all (_pyjq.c:2561) _pyjq.ScriptRuntimeError: Cannot iterate over null (null) Solved by adding ? to line 499 in shared/nodes.py

".DBSubnetGroup.Subnets[]?.SubnetIdentifier", self._json_blob

[nimrodkor]

@samyongsj For which region is this failure? Might be related to something I've stumbled into

[samyongsj]

eu-west-1

[sogaoh]

I've encountered a similar error.

/opt/cloudmapper # pipenv run python cloudmapper.py prepare --account XXXXX
Loading .env environment variables…
Building data for account XXXXX (XXXXX)
- 0 nodes built in region eu-north-1
- 0 nodes built in region ap-south-1
- 0 nodes built in region eu-west-3
- 0 nodes built in region eu-west-2
- 0 nodes built in region eu-west-1
- 0 nodes built in region ap-northeast-2
Traceback (most recent call last):
  File "cloudmapper.py", line 72, in <module>
    main()
  File "cloudmapper.py", line 66, in main
    commands[command].run(arguments)
  File "/opt/cloudmapper/commands/prepare.py", line 819, in run
    prepare(account, config, outputfilter)
  File "/opt/cloudmapper/commands/prepare.py", line 660, in prepare
    cytoscape_json = build_data_structure(account, config, outputfilter)
  File "/opt/cloudmapper/commands/prepare.py", line 477, in build_data_structure
    add_node_to_subnets(region, node, nodes)
  File "/opt/cloudmapper/commands/prepare.py", line 329, in add_node_to_subnets
    if len(node.subnets) == 0 and vpc.local_id == node._parent.local_id:
  File "/opt/cloudmapper/shared/nodes.py", line 498, in subnets
    ".DBSubnetGroup.Subnets[].SubnetIdentifier", self._json_blob
  File "/root/.local/share/virtualenvs/cloudmapper-3ZMnTcmW/lib/python3.7/site-packages/pyjq.py", line 49, in all
    return compile(script, vars, library_paths).all(_get_value(value, url, opener))
  File "_pyjq.pyx", line 209, in _pyjq.Script.all (_pyjq.c:2561)
_pyjq.ScriptRuntimeError: Cannot iterate over null (null)
/opt/cloudmapper #

Environment:

• region : ap-northeast-1
• uses docker-cloudmapper

[0xdabbad00]

Generically, this issue has been a problem in the past, for example #340. The issue is basically that I expect .DBSubnetGroup.Subnets[] to exist in a data structure, but in some cases it doesn't. I think generally the solution is that every time I reference an array with [], that I need to put []? . I don't know if there is a way to make jq always act like that, or if I have to do a search and replace for this.

[kbroughton]

You will also trigger this error if you run "prepare" before "collect"

[nmarchini]

Any update on a fix for this?

[0xdabbad00]

@nmarchini I'm unlikely to fix it unless I see it in an environment so I can better understand why it is happening. Otherwise, someone will need to send a PR, ideally with info on why it happened.