search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.16k stars 14.04k forks source link

Setup travis-cli for automated testing / continuous integration #251

Closed JackDanger closed 8 years ago

JackDanger commented 9 years ago

I cloned the repo, followed the contribution guidelines, and the tests failed. There appears to be an issue with some modules not importable and the connection sqlite table isn't created as part of the test setup.

I'm reporting this as an issue rather than just asking for help because I don't see Airflow configured in Travis.ci and I'm wondering if the unit tests have been broken for a while now.

I'm not very experienced at Python but if somebody could point me in the right direction I could try to fix this all up.

$ pip install -r requirements.txt && ./run_unit_tests.sh
Requirement already satisfied (use --upgrade to upgrade): boto in /usr/local/lib/python2.7/site-packages (from -r requirements.txt (line 1))
[... pip continues to find everything successfully ...]

Doctest: airflow.hooks.hive_hooks.HiveCliHook.run_cli ... FAIL
Doctest: airflow.hooks.hive_hooks.HiveMetastoreHook.check_for_partition ... FAIL
Doctest: airflow.hooks.hive_hooks.HiveMetastoreHook.get_partitions ... FAIL
Doctest: airflow.hooks.hive_hooks.HiveMetastoreHook.get_table ... FAIL
Doctest: airflow.hooks.hive_hooks.HiveMetastoreHook.max_partition ... FAIL
Doctest: airflow.hooks.hive_hooks.HiveServer2Hook.get_pandas_df ... FAIL
Doctest: airflow.hooks.hive_hooks.HiveServer2Hook.get_records ... FAIL
Failure: ImportError (No module named pymssql) ... ERROR
Failure: ImportError (No module named psycopg2) ... ERROR
Doctest: airflow.hooks.presto_hook.PrestoHook ... FAIL
Doctest: airflow.macros.hive.closest_ds_partition ... FAIL
Doctest: airflow.macros.hive.max_partition ... FAIL
Doctest: airflow.macros.ds_add ... ok
Doctest: airflow.macros.ds_format ... ok
Failure: ImportError (cannot import name MsSqlHook) ... ERROR
Failure: ImportError (No module named pymssql) ... ERROR
Failure: ImportError (cannot import name PostgresHook) ... ERROR
Failure: OperationalError ((sqlite3.OperationalError) no such table: connection [SQL: u'SELECT connection.conn_id AS connection_conn_id \nFROM connection GROUP BY connection.conn_id']) ... ERROR
Failure: OperationalError ((sqlite3.OperationalError) no such table: connection [SQL: u'SELECT connection.conn_id AS connection_conn_id \nFROM connection GROUP BY connection.conn_id']) ... ERROR

======================================================================
ERROR: Failure: ImportError (No module named pymssql)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/local/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/local/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/www/vendor/airflow/airflow/hooks/mssql_hook.py", line 1, in <module>
    import pymssql
ImportError: No module named pymssql

======================================================================
ERROR: Failure: ImportError (No module named psycopg2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/local/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/local/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/www/vendor/airflow/airflow/hooks/postgres_hook.py", line 1, in <module>
    import psycopg2
ImportError: No module named psycopg2

======================================================================
ERROR: Failure: ImportError (cannot import name MsSqlHook)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/local/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/local/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/www/vendor/airflow/airflow/operators/mssql_operator.py", line 3, in <module>
    from airflow.hooks import MsSqlHook
ImportError: cannot import name MsSqlHook

======================================================================
ERROR: Failure: ImportError (No module named pymssql)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/local/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/local/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/www/vendor/airflow/airflow/operators/mssql_to_hive.py", line 6, in <module>
    import pymssql
ImportError: No module named pymssql

======================================================================
ERROR: Failure: ImportError (cannot import name PostgresHook)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/local/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/local/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/www/vendor/airflow/airflow/operators/postgres_operator.py", line 3, in <module>
    from airflow.hooks import PostgresHook
ImportError: cannot import name PostgresHook

======================================================================
ERROR: Failure: OperationalError ((sqlite3.OperationalError) no such table: connection [SQL: u'SELECT connection.conn_id AS connection_conn_id \nFROM connection GROUP BY connection.conn_id'])
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/local/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/local/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/www/vendor/airflow/airflow/www/app.py", line 1712, in <module>
    class ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
  File "/www/vendor/airflow/airflow/www/app.py", line 1796, in ChartModelView
    .group_by(models.Connection.conn_id)
  File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2516, in __iter__
    return self._execute_and_instances(context)
  File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2531, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) no such table: connection [SQL: u'SELECT connection.conn_id AS connection_conn_id \nFROM connection GROUP BY connection.conn_id']

======================================================================
ERROR: Failure: OperationalError ((sqlite3.OperationalError) no such table: connection [SQL: u'SELECT connection.conn_id AS connection_conn_id \nFROM connection GROUP BY connection.conn_id'])
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/local/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/local/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/www/vendor/airflow/tests/__init__.py", line 2, in <module>
    from .core import *
  File "/www/vendor/airflow/tests/core.py", line 7, in <module>
    from airflow.www.app import app
  File "/www/vendor/airflow/airflow/www/app.py", line 1712, in <module>
    class ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
  File "/www/vendor/airflow/airflow/www/app.py", line 1796, in ChartModelView
    .group_by(models.Connection.conn_id)
  File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2516, in __iter__
    return self._execute_and_instances(context)
  File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2531, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) no such table: connection [SQL: u'SELECT connection.conn_id AS connection_conn_id \nFROM connection GROUP BY connection.conn_id']
-------------------- >> begin captured logging << --------------------
root: INFO: Filling up the DagBag from /Users/jackdanger/airflow/dags
root: INFO: Importing /www/vendor/airflow/airflow/example_dags/example_bash_operator.py
root: INFO: Loaded DAG <DAG: example_bash_operator>
root: INFO: Importing /www/vendor/airflow/airflow/example_dags/example_branch_operator.py
root: INFO: Loaded DAG <DAG: example_branch_operator>
root: INFO: Importing /www/vendor/airflow/airflow/example_dags/example_http_operator.py
root: INFO: Loaded DAG <DAG: example_http_operator>
root: INFO: Importing /www/vendor/airflow/airflow/example_dags/example_python_operator.py
root: INFO: Loaded DAG <DAG: example_python_operator>
root: INFO: Importing /www/vendor/airflow/airflow/example_dags/tutorial.py
root: INFO: Loaded DAG <DAG: tutorial>
root: INFO: Importing /www/vendor/airflow/airflow/example_dags/tutorial.py

======================================================================
FAIL: Doctest: airflow.hooks.hive_hooks.HiveCliHook.run_cli
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 2226, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for airflow.hooks.hive_hooks.HiveCliHook.run_cli
  File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 43, in run_cli

----------------------------------------------------------------------
File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 47, in airflow.hooks.hive_hooks.HiveCliHook.run_cli
Failed example:
    hh = HiveCliHook()
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest airflow.hooks.hive_hooks.HiveCliHook.run_cli[0]>", line 1, in <module>
        hh = HiveCliHook()
      File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 38, in __init__
        conn = self.get_connection(hive_cli_conn_id)
      File "/www/vendor/airflow/airflow/hooks/base_hook.py", line 36, in get_connection
        conn = random.choice(self.get_connections(conn_id))
      File "/www/vendor/airflow/airflow/hooks/base_hook.py", line 25, in get_connections
        .filter(Connection.conn_id == conn_id)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2399, in all
        return list(self)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2516, in __iter__
        return self._execute_and_instances(context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2531, in _execute_and_instances
        result = conn.execute(querycontext.statement, self._params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 914, in execute
        return meth(self, multiparams, params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
        return connection._execute_clauseelement(self, multiparams, params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
        compiled_sql, distilled_params
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1146, in _execute_context
        context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
        exc_info
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/util/compat.py", line 199, in raise_from_cause
        reraise(type(exception), exception, tb=exc_tb)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1139, in _execute_context
        context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/default.py", line 450, in do_execute
        cursor.execute(statement, parameters)
    OperationalError: (sqlite3.OperationalError) no such table: connection [SQL: u'SELECT connection.id AS connection_id, connection.conn_id AS connection_conn_id, connection.conn_type AS connection_conn_type, connection.host AS connection_host, connection.schema AS connection_schema, connection.login AS connection_login, connection.password AS connection_password, connection.port AS connection_port, connection.extra AS connection_extra \nFROM connection \nWHERE connection.conn_id = ?'] [parameters: ('hive_cli_default',)]
----------------------------------------------------------------------
File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 48, in airflow.hooks.hive_hooks.HiveCliHook.run_cli
Failed example:
    result = hh.run_cli("USE airflow;")
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest airflow.hooks.hive_hooks.HiveCliHook.run_cli[1]>", line 1, in <module>
        result = hh.run_cli("USE airflow;")
    NameError: name 'hh' is not defined
----------------------------------------------------------------------
File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 49, in airflow.hooks.hive_hooks.HiveCliHook.run_cli
Failed example:
    ("OK" in result)
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest airflow.hooks.hive_hooks.HiveCliHook.run_cli[2]>", line 1, in <module>
        ("OK" in result)
    NameError: name 'result' is not defined

-------------------- >> begin captured logging << --------------------
root: INFO: Reading the config from /Users/jackdanger/airflow/unittests.cfg
root: INFO: Using executor SequentialExecutor
root: DEBUG: Couldn't import module postgres_hook
root: DEBUG: Couldn't import module mssql_hook
root: DEBUG: Couldn't import module postgres_operator
root: DEBUG: Couldn't import module mssql_to_hive
root: DEBUG: Couldn't import module mssql_operator
root: INFO: Filling up the DagBag from /Users/jackdanger/airflow/dags
root: INFO: Importing /www/vendor/airflow/airflow/example_dags/example_bash_operator.py
root: INFO: Loaded DAG <DAG: example_bash_operator>
root: INFO: Importing /www/vendor/airflow/airflow/example_dags/example_branch_operator.py
root: INFO: Loaded DAG <DAG: example_branch_operator>
root: INFO: Importing /www/vendor/airflow/airflow/example_dags/example_http_operator.py
root: INFO: Loaded DAG <DAG: example_http_operator>
root: INFO: Importing /www/vendor/airflow/airflow/example_dags/example_python_operator.py
root: INFO: Loaded DAG <DAG: example_python_operator>
root: INFO: Importing /www/vendor/airflow/airflow/example_dags/tutorial.py
root: INFO: Loaded DAG <DAG: tutorial>
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: Doctest: airflow.hooks.hive_hooks.HiveMetastoreHook.check_for_partition
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 2226, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for airflow.hooks.hive_hooks.HiveMetastoreHook.check_for_partition
  File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 202, in check_for_partition

----------------------------------------------------------------------
File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 206, in airflow.hooks.hive_hooks.HiveMetastoreHook.check_for_partition
Failed example:
    hh = HiveMetastoreHook()
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest airflow.hooks.hive_hooks.HiveMetastoreHook.check_for_partition[0]>", line 1, in <module>
        compileflags, 1) in test.globs
      File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 175, in __init__
        self.metastore_conn = self.get_connection(metastore_conn_id)
      File "/www/vendor/airflow/airflow/hooks/base_hook.py", line 36, in get_connection
        conn = random.choice(self.get_connections(conn_id))
      File "/www/vendor/airflow/airflow/hooks/base_hook.py", line 25, in get_connections
        .filter(Connection.conn_id == conn_id)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2399, in all
        return list(self)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2516, in __iter__
        return self._execute_and_instances(context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2531, in _execute_and_instances
        result = conn.execute(querycontext.statement, self._params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 914, in execute
        return meth(self, multiparams, params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
        return connection._execute_clauseelement(self, multiparams, params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
        compiled_sql, distilled_params
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1146, in _execute_context
        context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
        exc_info
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/util/compat.py", line 199, in raise_from_cause
        reraise(type(exception), exception, tb=exc_tb)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1139, in _execute_context
        context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/default.py", line 450, in do_execute
        cursor.execute(statement, parameters)
    OperationalError: (sqlite3.OperationalError) no such table: connection [SQL: u'SELECT connection.id AS connection_id, connection.conn_id AS connection_conn_id, connection.conn_type AS connection_conn_type, connection.host AS connection_host, connection.schema AS connection_schema, connection.login AS connection_login, connection.password AS connection_password, connection.port AS connection_port, connection.extra AS connection_extra \nFROM connection \nWHERE connection.conn_id = ?'] [parameters: ('metastore_default',)]
----------------------------------------------------------------------
File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 208, in airflow.hooks.hive_hooks.HiveMetastoreHook.check_for_partition
Failed example:
    hh.check_for_partition('airflow', t, "ds='2015-01-01'")
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest airflow.hooks.hive_hooks.HiveMetastoreHook.check_for_partition[2]>", line 1, in <module>
        hh.check_for_partition('airflow', t, "ds='2015-01-01'")
    NameError: name 'hh' is not defined

======================================================================
FAIL: Doctest: airflow.hooks.hive_hooks.HiveMetastoreHook.get_partitions
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 2226, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for airflow.hooks.hive_hooks.HiveMetastoreHook.get_partitions
  File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 258, in get_partitions

----------------------------------------------------------------------
File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 264, in airflow.hooks.hive_hooks.HiveMetastoreHook.get_partitions
Failed example:
    hh = HiveMetastoreHook()
Failed example:d:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest airflow.hooks.hive_hooks.HiveMetastoreHook.get_partitions[0]>", line 1, in <module>
        hh = HiveMetastoreHook()
      File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 175, in __init__
        self.metastore_conn = self.get_connection(metastore_conn_id)
      File "/www/vendor/airflow/airflow/hooks/base_hook.py", line 36, in get_connection
        conn = random.choice(self.get_connections(conn_id))
      File "/www/vendor/airflow/airflow/hooks/base_hook.py", line 25, in get_connections
        .filter(Connection.conn_id == conn_id)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2399, in all
        return list(self)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2516, in __iter__
        return self._execute_and_instances(context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2531, in _execute_and_instances
        result = conn.execute(querycontext.statement, self._params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 914, in execute
        return meth(self, multiparams, params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
        return connection._execute_clauseelement(self, multiparams, params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
        compiled_sql, distilled_params
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1146, in _execute_context
        context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
        exc_info
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/util/compat.py", line 199, in raise_from_cause
        reraise(type(exception), exception, tb=exc_tb)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1139, in _execute_context
        context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/default.py", line 450, in do_execute
        cursor.execute(statement, parameters)
    OperationalError: (sqlite3.OperationalError) no such table: connection [SQL: u'SELECT connection.id AS connection_id, connection.conn_id AS connection_conn_id, connection.conn_type AS connection_conn_type, connection.host AS connection_host, connection.schema AS connection_schema, connection.login AS connection_login, connection.password AS connection_password, connection.port AS connection_port, connection.extra AS connection_extra \nFROM connection \nWHERE connection.conn_id = ?'] [parameters: ('metastore_default',)]
----------------------------------------------------------------------
File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 266, in airflow.hooks.hive_hooks.HiveMetastoreHook.get_partitions
Failed example:
    parts = hh.get_partitions(schema='airflow', table_name=t)
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest airflow.hooks.hive_hooks.HiveMetastoreHook.get_partitions[2]>", line 1, in <module>
        parts = hh.get_partitions(schema='airflow', table_name=t)
    NameError: name 'hh' is not defined
----------------------------------------------------------------------
File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 267, in airflow.hooks.hive_hooks.HiveMetastoreHook.get_partitions
Failed example:
    len(parts)
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest airflow.hooks.hive_hooks.HiveMetastoreHook.get_partitions[3]>", line 1, in <module>
        len(parts)
    NameError: name 'parts' is not defined
----------------------------------------------------------------------
File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 269, in airflow.hooks.hive_hooks.HiveMetastoreHook.get_partitions
Failed example:
    parts
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest airflow.hooks.hive_hooks.HiveMetastoreHook.get_partitions[4]>", line 1, in <module>
        parts
    NameError: name 'parts' is not defined

======================================================================
FAIL: Doctest: airflow.hooks.hive_hooks.HiveMetastoreHook.get_table
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 2226, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for airflow.hooks.hive_hooks.HiveMetastoreHook.get_table
  File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 220, in get_table

----------------------------------------------------------------------
File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 224, in airflow.hooks.hive_hooks.HiveMetastoreHook.get_table
Failed example:
    hh = HiveMetastoreHook()
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest airflow.hooks.hive_hooks.HiveMetastoreHook.get_table[0]>", line 1, in <module>
        hh = HiveMetastoreHook()
      File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 175, in __init__
        self.metastore_conn = self.get_connection(metastore_conn_id)
      File "/www/vendor/airflow/airflow/hooks/base_hook.py", line 36, in get_connection
        conn = random.choice(self.get_connections(conn_id))
      File "/www/vendor/airflow/airflow/hooks/base_hook.py", line 25, in get_connections
        .filter(Connection.conn_id == conn_id)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2399, in all
        return list(self)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2516, in __iter__
        return self._execute_and_instances(context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2531, in _execute_and_instances
        result = conn.execute(querycontext.statement, self._params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 914, in execute
        return meth(self, multiparams, params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
        return connection._execute_clauseelement(self, multiparams, params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
        compiled_sql, distilled_params
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1146, in _execute_context
        context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
        exc_info
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/util/compat.py", line 199, in raise_from_cause
        exc_infotype(exception), exception, tb=exc_tb)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1139, in _execute_context
        context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/default.py", line 450, in do_execute
        cursor.execute(statement, parameters)
    OperationalError: (sqlite3.OperationalError) no such table: connection [SQL: u'SELECT connection.id AS connection_id, connection.conn_id AS connection_conn_id, connection.conn_type AS connection_conn_type, connection.host AS connection_host, connection.schema AS connection_schema, connection.login AS connection_login, connection.password AS connection_password, connection.port AS connection_port, connection.extra AS connection_extra \nFROM connection \nWHERE connection.conn_id = ?'] [parameters: ('metastore_default',)]
----------------------------------------------------------------------
File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 225, in airflow.hooks.hive_hooks.HiveMetastoreHook.get_table
Failed example:
    t = hh.get_table(db='airflow', table_name='static_babynames')
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest airflow.hooks.hive_hooks.HiveMetastoreHook.get_table[1]>", line 1, in <module>
        t = hh.get_table(db='airflow', table_name='static_babynames')
    NameError: name 'hh' is not defined
----------------------------------------------------------------------
File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 226, in airflow.hooks.hive_hooks.HiveMetastoreHook.get_table
Failed example:
    t.tableName
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest airflow.hooks.hive_hooks.HiveMetastoreHook.get_table[2]>", line 1, in <module>
        t.tableName
    NameError: name 't' is not defined
----------------------------------------------------------------------
File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 228, in airflow.hooks.hive_hooks.HiveMetastoreHook.get_table
Failed example:
    [col.name for col in t.sd.cols]
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest airflow.hooks.hive_hooks.HiveMetastoreHook.get_table[3]>", line 1, in <module>
        [col.name for col in t.sd.cols]
    NameError: name 't' is not defined

======================================================================
FAIL: Doctest: airflow.hooks.hive_hooks.HiveMetastoreHook.max_partition
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 2226, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for airflow.hooks.hive_hooks.HiveMetastoreHook.max_partition
  File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 289, in max_partition

----------------------------------------------------------------------
File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 295, in airflow.hooks.hive_hooks.HiveMetastoreHook.max_partition
Failed example:
    hh = HiveMetastoreHook()
Exception raised:
    Traceback (most recent call last):
Exception raised:local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest airflow.hooks.hive_hooks.HiveMetastoreHook.max_partition[0]>", line 1, in <module>
        hh = HiveMetastoreHook()
      File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 175, in __init__
        self.metastore_conn = self.get_connection(metastore_conn_id)
      File "/www/vendor/airflow/airflow/hooks/base_hook.py", line 36, in get_connection
        conn = random.choice(self.get_connections(conn_id))
      File "/www/vendor/airflow/airflow/hooks/base_hook.py", line 25, in get_connections
        .filter(Connection.conn_id == conn_id)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2399, in all
        return list(self)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2516, in __iter__
        return self._execute_and_instances(context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2531, in _execute_and_instances
        result = conn.execute(querycontext.statement, self._params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 914, in execute
        return meth(self, multiparams, params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
        return connection._execute_clauseelement(self, multiparams, params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
        compiled_sql, distilled_params
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1146, in _execute_context
        context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
        exc_info
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/util/compat.py", line 199, in raise_from_cause
        reraise(type(exception), exception, tb=exc_tb)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1139, in _execute_context
        context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/default.py", line 450, in do_execute
        cursor.execute(statement, parameters)
    OperationalError: (sqlite3.OperationalError) no such table: connection [SQL: u'SELECT connection.id AS connection_id, connection.conn_id AS connection_conn_id, connection.conn_type AS connection_conn_type, connection.host AS connection_host, connection.schema AS connection_schema, connection.login AS connection_login, connection.password AS connection_password, connection.port AS connection_port, connection.extra AS connection_extra \nFROM connection \nWHERE connection.conn_id = ?'] [parameters: ('metastore_default',)]
----------------------------------------------------------------------
File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 297, in airflow.hooks.hive_hooks.HiveMetastoreHook.max_partition
Failed example:
    hh.max_partition(schema='airflow', table_name=t)
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest airflow.hooks.hive_hooks.HiveMetastoreHook.max_partition[2]>", line 1, in <module>
        hh.max_partition(schema='airflow', table_name=t)
    NameError: name 'hh' is not defined

======================================================================
FAIL: Doctest: airflow.hooks.hive_hooks.HiveServer2Hook.get_pandas_df
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 2226, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for airflow.hooks.hive_hooks.HiveServer2Hook.get_pandas_df
  File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 381, in get_pandas_df

----------------------------------------------------------------------
File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 387, in airflow.hooks.hive_hooks.HiveServer2Hook.get_pandas_df
----------------------------------------------------------------------
    df = hh.get_pandas_df(sql)
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest airflow.hooks.hive_hooks.HiveServer2Hook.get_pandas_df[2]>", line 1, in <module>
        df = hh.get_pandas_df(sql)
      File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 392, in get_pandas_df
        res = self.get_results(hql, schema=schema)
      File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 334, in get_results
        with self.get_conn() as conn:
      File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 325, in get_conn
        db = self.get_connection(self.hiveserver2_conn_id)
      File "/www/vendor/airflow/airflow/hooks/base_hook.py", line 36, in get_connection
        conn = random.choice(self.get_connections(conn_id))
      File "/www/vendor/airflow/airflow/hooks/base_hook.py", line 25, in get_connections
        .filter(Connection.conn_id == conn_id)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2399, in all
        return list(self)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2516, in __iter__
        return self._execute_and_instances(context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2531, in _execute_and_instances
        result = conn.execute(querycontext.statement, self._params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 914, in execute
        return meth(self, multiparams, params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
        return connection._execute_clauseelement(self, multiparams, params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
        compiled_sql, distilled_params
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1146, in _execute_context
        context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
        exc_info
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/util/compat.py", line 199, in raise_from_cause
        reraise(type(exception), exception, tb=exc_tb)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1139, in _execute_context
        context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/default.py", line 450, in do_execute
        cursor.execute(statement, parameters)
    OperationalError: (sqlite3.OperationalError) no such table: connection [SQL: u'SELECT connection.id AS connection_id, connection.conn_id AS connection_conn_id, connection.conn_type AS connection_conn_type, connection.host AS connection_host, connection.schema AS connection_schema, connection.login AS connection_login, connection.password AS connection_password, connection.port AS connection_port, connection.extra AS connection_extra \nFROM connection \nWHERE connection.conn_id = ?'] [parameters: ('hiveserver2_default',)]
----------------------------------------------------------------------
File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 388, in airflow.hooks.hive_hooks.HiveServer2Hook.get_pandas_df
Failed example:
    len(df.index)
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest airflow.hooks.hive_hooks.HiveServer2Hook.get_pandas_df[3]>", line 1, in <module>
        len(df.index)
    NameError: name 'df' is not defined

======================================================================
FAIL: Doctest: airflow.hooks.hive_hooks.HiveServer2Hook.get_records
======================================================================
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 2226, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for airflow.hooks.hive_hooks.HiveServer2Hook.get_records
  File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 370, in get_records

----------------------------------------------------------------------
File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 376, in airflow.hooks.hive_hooks.HiveServer2Hook.get_records
Failed example:
    len(hh.get_records(sql))
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest airflow.hooks.hive_hooks.HiveServer2Hook.get_records[2]>", line 1, in <module>
        len(hh.get_records(sql))
      File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 379, in get_records
        return self.get_results(hql, schema=schema)['data']
      File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 334, in get_results
        with self.get_conn() as conn:
      File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 325, in get_conn
        db = self.get_connection(self.hiveserver2_conn_id)
      File "/www/vendor/airflow/airflow/hooks/base_hook.py", line 36, in get_connection
        conn = random.choice(self.get_connections(conn_id))
      File "/www/vendor/airflow/airflow/hooks/base_hook.py", line 25, in get_connections
        .filter(Connection.conn_id == conn_id)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2399, in all
        return list(self)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2516, in __iter__
        return self._execute_and_instances(context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2531, in _execute_and_instances
        result = conn.execute(querycontext.statement, self._params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 914, in execute
        return meth(self, multiparams, params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
        return connection._execute_clauseelement(self, multiparams, params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
        compiled_sql, distilled_params
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1146, in _execute_context
        context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
        exc_info
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/util/compat.py", line 199, in raise_from_cause
        reraise(type(exception), exception, tb=exc_tb)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1139, in _execute_context
        context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/default.py", line 450, in do_execute
        cursor.execute(statement, parameters)
    OperationalError: (sqlite3.OperationalError) no such table: connection [SQL: u'SELECT connection.id AS connection_id, connection.conn_id AS connection_conn_id, connection.conn_type AS connection_conn_type, connection.host AS connection_host, connection.schema AS connection_schema, connection.login AS connection_login, connection.password AS connection_password, connection.port AS connection_port, connection.extra AS connection_extra \nFROM connection \nWHERE connection.conn_id = ?'] [parameters: ('hiveserver2_default',)]

======================================================================
FAIL: Doctest: airflow.hooks.presto_hook.PrestoHook
----------------------------------------------------------------------
FAIL: Doctest: airflow.hooks.presto_hook.PrestoHook
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 2226, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for airflow.hooks.presto_hook.PrestoHook
  File "/www/vendor/airflow/airflow/hooks/presto_hook.py", line 15, in PrestoHook

----------------------------------------------------------------------
File "/www/vendor/airflow/airflow/hooks/presto_hook.py", line 21, in airflow.hooks.presto_hook.PrestoHook
Failed example:
    ph.get_records(sql)
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest airflow.hooks.presto_hook.PrestoHook[2]>", line 1, in <module>
        ph.get_records(sql)
      File "/www/vendor/airflow/airflow/hooks/presto_hook.py", line 48, in get_records
        self._strip_sql(hql), parameters)
      File "/www/vendor/airflow/airflow/hooks/dbapi_hook.py", line 60, in get_records
        conn = self.get_conn()
      File "/www/vendor/airflow/airflow/hooks/presto_hook.py", line 30, in get_conn
        db = self.get_connection(self.presto_conn_id)
      File "/www/vendor/airflow/airflow/hooks/base_hook.py", line 36, in get_connection
        conn = random.choice(self.get_connections(conn_id))
      File "/www/vendor/airflow/airflow/hooks/base_hook.py", line 25, in get_connections
        .filter(Connection.conn_id == conn_id)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2399, in all
        return list(self)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2516, in __iter__
        return self._execute_and_instances(context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2531, in _execute_and_instances
        result = conn.execute(querycontext.statement, self._params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 914, in execute
        return meth(self, multiparams, params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
        return connection._execute_clauseelement(self, multiparams, params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
        compiled_sql, distilled_params
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1146, in _execute_context
        context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
        exc_info
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/util/compat.py", line 199, in raise_from_cause
        reraise(type(exception), exception, tb=exc_tb)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1139, in _execute_context
        context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/default.py", line 450, in do_execute
        cursor.execute(statement, parameters)
    OperationalError: (sqlite3.OperationalError) no such table: connection [SQL: u'SELECT connection.id AS connection_id, connection.conn_id AS connection_conn_id, connection.conn_type AS connection_conn_type, connection.host AS connection_host, connection.schema AS connection_schema, connection.login AS connection_login, connection.password AS connection_password, connection.port AS connection_port, connection.extra AS connection_extra \nFROM connection \nWHERE connection.conn_id = ?'] [parameters: ('presto_default',)]

======================================================================
FAIL: Doctest: airflow.macros.hive.closest_ds_partition
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 2226, in runTest
Traceback (most recent call last):lf.format_failure(new.getvalue()))
AssertionError: Failed doctest test for airflow.macros.hive.closest_ds_partition
  File "/www/vendor/airflow/airflow/macros/hive.py", line 63, in closest_ds_partition

----------------------------------------------------------------------
File "/www/vendor/airflow/airflow/macros/hive.py", line 78, in airflow.macros.hive.closest_ds_partition
Failed example:
    closest_ds_partition(tbl, '2015-01-02')
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest airflow.macros.hive.closest_ds_partition[1]>", line 1, in <module>
        closest_ds_partition(tbl, '2015-01-02')
      File "/www/vendor/airflow/airflow/macros/hive.py", line 84, in closest_ds_partition
        hh = HiveMetastoreHook(metastore_conn_id=metastore_conn_id)
      File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 175, in __init__
        self.metastore_conn = self.get_connection(metastore_conn_id)
      File "/www/vendor/airflow/airflow/hooks/base_hook.py", line 36, in get_connection
        conn = random.choice(self.get_connections(conn_id))
      File "/www/vendor/airflow/airflow/hooks/base_hook.py", line 25, in get_connections
        .filter(Connection.conn_id == conn_id)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2399, in all
        return list(self)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2516, in __iter__
        return self._execute_and_instances(context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2531, in _execute_and_instances
        result = conn.execute(querycontext.statement, self._params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 914, in execute
        return meth(self, multiparams, params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
        return connection._execute_clauseelement(self, multiparams, params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
        compiled_sql, distilled_params
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1146, in _execute_context
        context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
        exc_info
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/util/compat.py", line 199, in raise_from_cause
        reraise(type(exception), exception, tb=exc_tb)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1139, in _execute_context
        context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/default.py", line 450, in do_execute
        cursor.execute(statement, parameters)
    OperationalError: (sqlite3.OperationalError) no such table: connection [SQL: u'SELECT connection.id AS connection_id, connection.conn_id AS connection_conn_id, connection.conn_type AS connection_conn_type, connection.host AS connection_host, connection.schema AS connection_schema, connection.login AS connection_login, connection.password AS connection_password, connection.port AS connection_port, connection.extra AS connection_extra \nFROM connection \nWHERE connection.conn_id = ?'] [parameters: ('metastore_default',)]

======================================================================
FAIL: Doctest: airflow.macros.hive.max_partition
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 2226, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for airflow.macros.hive.max_partition
  File "/www/vendor/airflow/airflow/macros/hive.py", line 6, in max_partition

  File "/www/vendor/airflow/airflow/macros/hive.py", line 6, in max_partition
File "/www/vendor/airflow/airflow/macros/hive.py", line 25, in airflow.macros.hive.max_partition
Failed example:
    max_partition('airflow.static_babynames_partitioned')
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest airflow.macros.hive.max_partition[0]>", line 1, in <module>
        max_partition('airflow.static_babynames_partitioned')
      File "/www/vendor/airflow/airflow/macros/hive.py", line 31, in max_partition
        hh = HiveMetastoreHook(metastore_conn_id=metastore_conn_id)
      File "/www/vendor/airflow/airflow/hooks/hive_hooks.py", line 175, in __init__
        self.metastore_conn = self.get_connection(metastore_conn_id)
      File "/www/vendor/airflow/airflow/hooks/base_hook.py", line 36, in get_connection
        conn = random.choice(self.get_connections(conn_id))
      File "/www/vendor/airflow/airflow/hooks/base_hook.py", line 25, in get_connections
        .filter(Connection.conn_id == conn_id)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2399, in all
        return list(self)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2516, in __iter__
        return self._execute_and_instances(context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/orm/query.py", line 2531, in _execute_and_instances
        result = conn.execute(querycontext.statement, self._params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 914, in execute
        return meth(self, multiparams, params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
        return connection._execute_clauseelement(self, multiparams, params)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
        compiled_sql, distilled_params
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1146, in _execute_context
        context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
        exc_info
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/util/compat.py", line 199, in raise_from_cause
        reraise(type(exception), exception, tb=exc_tb)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/base.py", line 1139, in _execute_context
        context)
      File "build/bdist.macosx-10.9-x86_64/egg/sqlalchemy/engine/default.py", line 450, in do_execute
        cursor.execute(statement, parameters)
    OperationalError: (sqlite3.OperationalError) no such table: connection [SQL: u'SELECT connection.id AS connection_id, connection.conn_id AS connection_conn_id, connection.conn_type AS connection_conn_type, connection.host AS connection_host, connection.schema AS connection_schema, connection.login AS connection_login, connection.password AS connection_password, connection.port AS connection_port, connection.extra AS connection_extra \nFROM connection \nWHERE connection.conn_id = ?'] [parameters: ('metastore_default',)]

[... coverage information ...]

FAILED (errors=7, failures=10)
mistercrunch commented 9 years ago

The unit tests require a bit more setup than documented. It's also tricky because most people won't be able to test many operators because you may not have some of the systems in your infrastructure. For instance I can't test Microsoft SQL Server of Postgres related hooks and operators because I don't have any of those laying around.

I'll try to generalize the tests and document the setup process as much as possible. In the meantime, you can run the first few lines of rununittests.sh to point to a unit tests environment, run airflow initdb and airflow webserver to get in the UI and provide that connection information for the systems you support (mysql_default, presto_default, hdfs_default, ).

Note that some tests may (at the moment) require data structures that you don't have, notably a "babynames" dataset. One of the tests should probably one that populates said dataset...

Maybe we can work together at making running tests outside of Airbnb as easy as possible.

I also run the tests before I merge any significant PRs to master, and thoroughly before pushing a new release to Pypi. So it's ok to submit a PR without having passed all tests, at least for now.

JackDanger commented 9 years ago

I'll try to generalize the tests and document the setup process as much as possible.

@mistercrunch I'd be happy to do this work. Luigi runs tests on Travis CI using MySQL, PostgreSQL and Hadoop as dependencies so I'm sure we can get those working here as well.

run airflow initdb and airflow webserver

Ah, I see. I'll make a PR that generates the airflow executable and runs these commands as a preliminary step to running the tests so developers don't need to worry about them.

Maybe we can work together at making running tests outside of Airbnb as easy as possible.

That's my goal! I won't be able to run Airflow at Square (even just to play around) if it doesn't have automated tests with ok coverage that can run and pass.

mistercrunch commented 9 years ago

Sounds awesome. Really glad to have you on board.

You can totally squeeze an airflow initdb line in here: https://github.com/airbnb/airflow/blob/master/run_unit_tests.sh

I'm planning on doing a coverage blitz, and @artwr is hooking up travis-ci. I've been looking at the travis widget in other Github project thinking I should do whatever is needed to enable this!

mistercrunch commented 9 years ago

I added a section in the CONTRIBUTING.md with loose guideline on how to setup the unit tests https://github.com/airbnb/airflow/blob/master/CONTRIBUTING.md#running-unit-tests

btw @artwr is working on setting up travis!