Fix configuration to not require section in airflow.cfg

[MiriamSexton]

Closes #5

This allows a clean build and test from just setup.py. Previously, when creating a clean install, airflow freaked out that a [fileflow] section of the config file didn't exist. This interfered with testing and building docs, although on a real install it wasn't a problem.

This also fixes a typo in the code that sets the aws_secret_access_key.

To test:

Checkout master in a clean virtualenv (and don't have AIRFLOW_HOME set to a custom config file). Run python setup.py build and python setup.py test. Observe the errors.

Checkout this branch and do the same. Nicer, right?

[lauralorenz]

I do python setup.py build and python setup.py test from the most recent master and I keep getting this weird traceback. Haven't been able to figure it out yet so putting the traceback down for now:

Traceback (most recent call last):
  File "setup.py", line 25, in <module>
    'pep8-naming==0.3.3']
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/setuptools/command/test.py", line 132, in run
    self.distribution.install_requires)
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/setuptools/dist.py", line 311, in fetch_build_eggs
    replace_conflicting=True,
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/pkg_resources/__init__.py", line 797, in resolve
    dist = best[req.key] = env.best_match(req, ws, installer)
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1047, in best_match
    return self.obtain(req, installer)
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1059, in obtain
    return installer(requirement)
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/setuptools/dist.py", line 378, in fetch_build_egg
    return cmd.easy_install(req)
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 623, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 653, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 838, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1066, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1052, in run_setup
    run_setup(setup_script, args)
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/setuptools/sandbox.py", line 238, in run_setup
    raise
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/setuptools/sandbox.py", line 191, in setup_context
    yield
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/setuptools/sandbox.py", line 162, in save_modules
    saved_exc.resume()
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/setuptools/sandbox.py", line 137, in resume
    compat.reraise(type, exc, self._tb)
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/setuptools/sandbox.py", line 150, in save_modules
    yield saved
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/setuptools/sandbox.py", line 191, in setup_context
    yield
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/setuptools/sandbox.py", line 235, in run_setup
    DirectorySandbox(setup_dir).run(runner)
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/setuptools/sandbox.py", line 265, in run
    return func()
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/setuptools/sandbox.py", line 234, in runner
    _execfile(setup_script, ns)
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/setuptools/sandbox.py", line 46, in _execfile
    exec(code, globals, locals)
  File "/var/folders/f6/b427f1ns42n4sj9szglhcpsc0000gn/T/easy_install-eCxO2z/python-daemon-2.1.2/setup.py", line 86, in <module>

  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 111, in setup
    _setup_distribution = dist = klass(attrs)
  File "/var/folders/f6/b427f1ns42n4sj9szglhcpsc0000gn/T/easy_install-eCxO2z/python-daemon-2.1.2/version.py", line 640, in __init__
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/setuptools/dist.py", line 265, in __init__
    self.fetch_build_eggs(attrs['setup_requires'])
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/setuptools/dist.py", line 311, in fetch_build_eggs
    replace_conflicting=True,
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/pkg_resources/__init__.py", line 797, in resolve
    dist = best[req.key] = env.best_match(req, ws, installer)
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1047, in best_match
    return self.obtain(req, installer)
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1059, in obtain
    return installer(requirement)
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/setuptools/dist.py", line 376, in fetch_build_egg
    cmd.ensure_finalized()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 226, in finalize_options
    'dist_version': self.distribution.get_version(),
  File "/var/folders/f6/b427f1ns42n4sj9szglhcpsc0000gn/T/easy_install-eCxO2z/python-daemon-2.1.2/version.py", line 656, in get_version
  File "/var/folders/f6/b427f1ns42n4sj9szglhcpsc0000gn/T/easy_install-eCxO2z/python-daemon-2.1.2/version.py", line 651, in get_version_info
  File "/var/folders/f6/b427f1ns42n4sj9szglhcpsc0000gn/T/easy_install-eCxO2z/python-daemon-2.1.2/version.py", line 552, in get_changelog_path
  File "/Users/llorenz/Envs/fileflow_bad/bin/../lib/python2.7/posixpath.py", line 129, in dirname
    i = p.rfind('/') + 1
AttributeError: 'NoneType' object has no attribute 'rfind'

[lauralorenz]

The Internet fixed itself/whatever was wrong with python-daemon is gone now because I got it to install master fine today. Patience is a virtue eh

# no problems finding versions today
Collecting python-daemon<2.2,>=2.1.1 (from airflow~=1.7.0->fileflow)
  Using cached python_daemon-2.1.2-py2.py3-none-any.whl

[lauralorenz]

Ok finished the tests. Did observe the config errors on master:

======================================================================
ERROR: Failure: NoSectionError (No section: 'fileflow')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/llorenz/Development/Repos/fileflow/.eggs/nose-1.3.7-py2.7.egg/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/Users/llorenz/Development/Repos/fileflow/.eggs/nose-1.3.7-py2.7.egg/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/Users/llorenz/Development/Repos/fileflow/.eggs/nose-1.3.7-py2.7.egg/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/Users/llorenz/Development/Repos/fileflow/tests/task_runners/test_taskRunner.py", line 8, in <module>
    from fileflow.task_runners import TaskRunner
  File "/Users/llorenz/Development/Repos/fileflow/fileflow/task_runners/__init__.py", line 1, in <module>
    from .task_runner import TaskRunner
  File "/Users/llorenz/Development/Repos/fileflow/fileflow/task_runners/task_runner.py", line 13, in <module>
    from fileflow.storage_drivers import get_storage_driver
  File "/Users/llorenz/Development/Repos/fileflow/fileflow/storage_drivers/__init__.py", line 4, in <module>
    from .. import configuration
  File "/Users/llorenz/Development/Repos/fileflow/fileflow/configuration.py", line 11, in <module>
    airflow_configuration.set('fileflow', 'environment', 'production')
  File "/Users/llorenz/Envs/fileflow_bad/lib/python2.7/site-packages/airflow/configuration.py", line 691, in set
    return conf.set(section, option, value)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ConfigParser.py", line 396, in set
    raise NoSectionError(section)
NoSectionError: No section: 'fileflow'

----------------------------------------------------------------------
Ran 7 tests in 1.593s

FAILED (errors=7)

and not on this branch:

----------------------------------------------------------------------
Ran 37 tests in 2.463s

OK

Meanwhile, I don't see anything open on Airflow's JIRA about making the Airflow configuration system more amenable to plugins like this one, so I will write something up there to put it on the radar and get some feedback on.