search

nchammas / flintrock

A command-line tool for launching Apache Spark clusters.
Apache License 2.0
638 stars 116 forks source link

Documentation bug: KeyError 'providers' #116

Closed hahnicity closed 8 years ago

hahnicity commented 8 years ago

If you are reporting a potential bug, include as many of the following as are relevant (and delete this sentence, of course):

I followed your README doc's advice for initializing a cluster and my config file looks like this

provider: ec2

services:
  spark:
    version: 1.6.1

launch:
  num-slaves: 5

ec2:
  key-name: gr-pair-gondolin
  identity-file: /Users/greg/.ssh/gr-pair-gondolin.pem
  instance-type: t2.micro
  region: us-west1b
  ami: ami-d1315fb1
  user: ec2-user

However when I run this file with flintrock using the command flintrock --config flintrock.config.yml launch foo-cluster I get this as my error. 

Traceback (most recent call last):
  File "/Users/greg/workspace/python/ecs251-final-project/venv/bin/flintrock", line 11, in <module>
    sys.exit(main())
  File "/Users/greg/workspace/python/ecs251-final-project/venv/lib/python3.5/site-packages/flintrock/flintrock.py", line 871, in main
    cli(obj={})
  File "/Users/greg/workspace/python/ecs251-final-project/venv/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/Users/greg/workspace/python/ecs251-final-project/venv/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/Users/greg/workspace/python/ecs251-final-project/venv/lib/python3.5/site-packages/click/core.py", line 1057, in invoke
    Command.invoke(self, ctx)
  File "/Users/greg/workspace/python/ecs251-final-project/venv/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/greg/workspace/python/ecs251-final-project/venv/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/Users/greg/workspace/python/ecs251-final-project/venv/lib/python3.5/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/greg/workspace/python/ecs251-final-project/venv/lib/python3.5/site-packages/flintrock/flintrock.py", line 169, in cli
    config_map = config_to_click(normalize_keys(config_raw))
  File "/Users/greg/workspace/python/ecs251-final-project/venv/lib/python3.5/site-packages/flintrock/flintrock.py", line 770, in config_to_click
    'ec2_' + k: v for (k, v) in config['providers']['ec2'].items()}
KeyError: 'providers'

As a result I think the documentation seems off because it doesn't mention using 'providers' as a key it just mentions using 'provider'

nchammas commented 8 years ago

Good catch!

Just curious, are you aware that you can call flintrock configure and Flintrock will setup a valid config file for you? I've always wondered if that particular feature wasn't presented clearly enough to users.