search

puckel / docker-airflow

Docker Apache Airflow
Apache License 2.0
3.78k stars 544 forks source link

Update to Airflow 2.0.0 #618

Open neylsoncrepalde opened 3 years ago

neylsoncrepalde commented 3 years ago

This is a pull request to update your repo to work with Airflow 2.0.0 and Python 3.8. Everything was tested locally and is apparently working fine.

The main changes were:

Many thanks for your hard work putting this together and I hope this pull request can be handy.

rikr2sanchez commented 3 years ago

After login the app crash:

Something bad has happened.
Please consider letting us know by creating a bug report using GitHub.

Python version: 3.8.7
Airflow version: 2.0.0
Node: 7de345de891a
-------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.8/site-packages/airflow/www/auth.py", line 34, in decorated
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/www/views.py", line 491, in index
    filter_dag_ids = current_app.appbuilder.sm.get_accessible_dag_ids(g.user)
  File "/usr/local/lib/python3.8/site-packages/airflow/www/security.py", line 250, in get_accessible_dag_ids
    accessible_dags = self.get_accessible_dags(
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/session.py", line 65, in wrapper
    return func(*args, session=session, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/www/security.py", line 264, in get_accessible_dags
    joinedload(User.roles)
AttributeError: '_UnboundLoad' object has no attribute 'options'
shihabcsedu09 commented 3 years ago

Hi @neylsoncrepalde

I am getting the following error while running the container after building it.

Unable to load the config, contains a configuration error.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/logging/config.py", line 563, in configure
    handler = self.configure_handler(handlers[name])
  File "/usr/local/lib/python3.8/logging/config.py", line 744, in configure_handler
    result = factory(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 43, in __init__
    self.filename_template, self.filename_jinja_template = parse_template_string(filename_template)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/helpers.py", line 134, in parse_template_string
    return None, Template(template_string)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 1031, in __new__
    return env.from_string(source, template_class=cls)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 941, in from_string
    return cls.from_code(self, self.compile(source), globals, None)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 638, in compile
    self.handle_exception(source=source_hint)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/usr/local/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "<unknown>", line 1, in template
jinja2.exceptions.TemplateSyntaxError: expected token ':', got '}'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 5, in <module>
    from airflow.__main__ import main
  File "/usr/local/lib/python3.8/site-packages/airflow/__init__.py", line 46, in <module>
    settings.initialize()
  File "/usr/local/lib/python3.8/site-packages/airflow/settings.py", line 432, in initialize
    LOGGING_CLASS_PATH = configure_logging()
  File "/usr/local/lib/python3.8/site-packages/airflow/logging_config.py", line 62, in configure_logging
    raise e
  File "/usr/local/lib/python3.8/site-packages/airflow/logging_config.py", line 57, in configure_logging
    dictConfig(logging_config)
  File "/usr/local/lib/python3.8/logging/config.py", line 808, in dictConfig
    dictConfigClass(config).configure()
  File "/usr/local/lib/python3.8/logging/config.py", line 570, in configure
    raise ValueError('Unable to configure handler '
ValueError: Unable to configure handler 'processor'
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/logging/config.py", line 563, in configure
    handler = self.configure_handler(handlers[name])
  File "/usr/local/lib/python3.8/logging/config.py", line 744, in configure_handler
    result = factory(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 43, in __init__
    self.filename_template, self.filename_jinja_template = parse_template_string(filename_template)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/helpers.py", line 134, in parse_template_string
    return None, Template(template_string)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 1031, in __new__
    return env.from_string(source, template_class=cls)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 941, in from_string
    Unable to load the config, contains a configuration error.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/logging/config.py", line 563, in configure
return cls.from_code(self, self.compile(source), globals, None)    handler = self.configure_handler(handlers[name])
  File "/usr/local/lib/python3.8/logging/config.py", line 744, in configure_handler

  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 638, in compile
    result = factory(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 43, in __init__
    self.filename_template, self.filename_jinja_template = parse_template_string(filename_template)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/helpers.py", line 134, in parse_template_string
    return None, Template(template_string)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 1031, in __new__
    return env.from_string(source, template_class=cls)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 941, in from_string
        return cls.from_code(self, self.compile(source), globals, None)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 638, in compile
self.handle_exception(source=source_hint)    self.handle_exception(source=source_hint)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception

  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/usr/local/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "<unknown>", line 1, in template
jinja2.exceptions.TemplateSyntaxError: expected token ':', got '}'
neylsoncrepalde commented 3 years ago

Hello @shihabcsedu09 and @rikr2sanchez.

Could you please provide information on the commands you ran to get the error you described?

I tried locally and everything is working fine. Maybe, the problem is that up to this point, there is no public docker image puckel/docker-airflow:2.0.0 so I had to build it. In order to do that, I ran

git clone https://github.com/neylsoncrepalde/docker-airflow.git
cd docker-airflow
docker build -t puckel/docker-airflow:2.0.0 .
docker-compose -f docker-compose-CeleryExecutor.yml up

Many thanks! Best!

shihabcsedu09 commented 3 years ago

@neylsoncrepalde Ignore it. I can now run this. I didn't use the updated cfg file. Thanks for this pull request

rikr2sanchez commented 3 years ago

Hello @shihabcsedu09 and @rikr2sanchez.

Could you please provide information on the commands you ran to get the error you described?

I tried locally and everything is working fine. Maybe, the problem is that up to this point, there is no public docker image puckel/docker-airflow:2.0.0 so I had to build it. In order to do that, I ran

git clone https://github.com/neylsoncrepalde/docker-airflow.git
cd docker-airflow
docker build -t puckel/docker-airflow:2.0.0 .
docker-compose -f docker-compose-CeleryExecutor.yml up

Many thanks! Best!

Up and running ! Thanks !

brow86 commented 3 years ago

Hi @neylsoncrepalde

I am getting the following error while running the container after building it.


Unable to load the config, contains a configuration error.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/logging/config.py", line 563, in configure
    handler = self.configure_handler(handlers[name])
  File "/usr/local/lib/python3.8/logging/config.py", line 744, in configure_handler
    result = factory(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 43, in __init__
    self.filename_template, self.filename_jinja_template = parse_template_string(filename_template)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/helpers.py", line 134, in parse_template_string
    return None, Template(template_string)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 1031, in __new__
    return env.from_string(source, template_class=cls)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 941, in from_string
    return cls.from_code(self, self.compile(source), globals, None)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 638, in compile
    self.handle_exception(source=source_hint)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/usr/local/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "<unknown>", line 1, in template
jinja2.exceptions.TemplateSyntaxError: expected token ':', got '}'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 5, in <module>
    from airflow.__main__ import main
  File "/usr/local/lib/python3.8/site-packages/airflow/__init__.py", line 46, in <module>
    settings.initialize()
  File "/usr/local/lib/python3.8/site-packages/airflow/settings.py", line 432, in initialize
    LOGGING_CLASS_PATH = configure_logging()
  File "/usr/local/lib/python3.8/site-packages/airflow/logging_config.py", line 62, in configure_logging
    raise e
  File "/usr/local/lib/python3.8/site-packages/airflow/logging_config.py", line 57, in configure_logging
    dictConfig(logging_config)
  File "/usr/local/lib/python3.8/logging/config.py", line 808, in dictConfig
    dictConfigClass(config).configure()
  File "/usr/local/lib/python3.8/logging/config.py", line 570, in configure
    raise ValueError('Unable to configure handler '
ValueError: Unable to configure handler 'processor'
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/logging/config.py", line 563, in configure
    handler = self.configure_handler(handlers[name])
  File "/usr/local/lib/python3.8/logging/config.py", line 744, in configure_handler
    result = factory(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 43, in __init__
    self.filename_template, self.filename_jinja_template = parse_template_string(filename_template)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/helpers.py", line 134, in parse_template_string
    return None, Template(template_string)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 1031, in __new__
    return env.from_string(source, template_class=cls)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 941, in from_string
    Unable to load the config, contains a configuration error.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/logging/config.py", line 563, in configure
return cls.from_code(self, self.compile(source), globals, None)    handler = self.configure_handler(handlers[name])
  File "/usr/local/lib/python3.8/logging/config.py", line 744, in configure_handler

  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 638, in compile
    result = factory(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 43, in __init__
    self.filename_template, self.filename_jinja_template = parse_template_string(filename_template)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/helpers.py", line 134, in parse_template_string
    return None, Template(template_string)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 1031, in __new__
    return env.from_string(source, template_class=cls)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 941, in from_string
        return cls.from_code(self, self.compile(source), globals, None)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 638, in compile
self.handle_exception(source=source_hint)    self.handle_exception(source=source_hint)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception

  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/usr/local/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "<unknown>", line 1, in template
jinja2.exceptions.TemplateSyntaxError: expected token ':', got '}'

I hit this today too, got it to work by using:

log_filename_template = {{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log
log_processor_filename_template = {{ filename }}.log

with the default 1.10.14 from github, in case anyone else runs into this.

levyitay commented 3 years ago

@neylsoncrepalde Using docker-compose-LocalExecuter.yml does not work when building locally, but does work with your image: neylsoncrepalde/airflow-docker This is the Error I'm getting: 

webserver_1  | INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
webserver_1  | INFO  [alembic.runtime.migration] Will assume transactional DDL.
webserver_1  | Traceback (most recent call last):
webserver_1  |   File "/usr/local/bin/airflow", line 8, in <module>
webserver_1  |     sys.exit(main())
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
webserver_1  |     args.func(args)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
webserver_1  |     return func(*args, **kwargs)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/cli/commands/db_command.py", line 31, in initdb
webserver_1  |     db.initdb()
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/utils/db.py", line 549, in initdb
webserver_1  |     upgradedb()
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/utils/db.py", line 688, in upgradedb
webserver_1  |     command.upgrade(config, 'heads')
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/command.py", line 294, in upgrade
webserver_1  |     script.run_env()
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/script/base.py", line 490, in run_env
webserver_1  |     util.load_python_file(self.dir, "env.py")
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 97, in load_python_file
webserver_1  |     module = load_module_py(module_id, path)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/util/compat.py", line 182, in load_module_py
webserver_1  |     spec.loader.exec_module(module)
webserver_1  |   File "<frozen importlib._bootstrap_external>", line 783, in exec_module
webserver_1  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/migrations/env.py", line 108, in <module>
webserver_1  |     run_migrations_online()
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/migrations/env.py", line 102, in run_migrations_online
webserver_1  |     context.run_migrations()
webserver_1  |   File "<string>", line 8, in run_migrations
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/runtime/environment.py", line 813, in run_migrations
webserver_1  |     self.get_context().run_migrations(**kw)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/runtime/migration.py", line 548, in run_migrations
webserver_1  |     for step in self._migrations_fn(heads, self):
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/command.py", line 283, in upgrade
webserver_1  |     return script._upgrade_revs(revision, rev)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/script/base.py", line 365, in _upgrade_revs
webserver_1  |     revs = list(revs)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/script/revision.py", line 916, in _iterate_revisions
webserver_1  |     uppers = util.dedupe_tuple(self.get_revisions(upper))
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/script/revision.py", line 457, in get_revisions
webserver_1  |     resolved_id, branch_label = self._resolve_revision_number(id_)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/script/revision.py", line 640, in _resolve_revision_number
webserver_1  |     self._revision_map
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/util/langhelpers.py", line 234, in __get__
webserver_1  |     obj.__dict__[self.__name__] = result = self.fget(obj)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/script/revision.py", line 156, in _revision_map
webserver_1  |     for revision in self._generator():
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/script/base.py", line 115, in _load_revisions
webserver_1  |     script = Script._from_filename(self, vers, file_)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/script/base.py", line 904, in _from_filename
webserver_1  |     module = util.load_python_file(dir_, filename)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 97, in load_python_file
webserver_1  |     module = load_module_py(module_id, path)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/util/compat.py", line 182, in load_module_py
webserver_1  |     spec.loader.exec_module(module)
webserver_1  |   File "<frozen importlib._bootstrap_external>", line 783, in exec_module
webserver_1  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/migrations/versions/2c6edca13270_resource_based_permissions.py", line 27, in <module>
webserver_1  |     from airflow.www.app import create_app
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/www/app.py", line 38, in <module>
webserver_1  |     from airflow.www.extensions.init_views import (
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/www/extensions/init_views.py", line 29, in <module>
webserver_1  |     from airflow.www.views import lazy_add_provider_discovered_options_to_connection_form
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/www/views.py", line 95, in <module>
webserver_1  |     from airflow.www import auth, utils as wwwutils
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/www/utils.py", line 27, in <module>
webserver_1  |     from flask_appbuilder.models.sqla.interface import SQLAInterface
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/models/sqla/interface.py", line 16, in <module>
webserver_1  |     from sqlalchemy_utils.types.uuid import UUIDType
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy_utils/__init__.py", line 1, in <module>
webserver_1  |     from .aggregates import aggregated  # noqa
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy_utils/aggregates.py", line 372, in <module>
webserver_1  |     from .functions.orm import get_column_key
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy_utils/functions/__init__.py", line 1, in <module>
webserver_1  |     from .database import (  # noqa
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy_utils/functions/database.py", line 11, in <module>
webserver_1  |     from .orm import quote
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy_utils/functions/orm.py", line 14, in <module>
webserver_1  |     from sqlalchemy.orm.query import _ColumnEntity
webserver_1  | ImportError: cannot import name '_ColumnEntity' from 'sqlalchemy.orm.query' (/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py)
webserver_1  | Traceback (most recent call last):
webserver_1  |   File "/usr/local/bin/airflow", line 8, in <module>
webserver_1  |     sys.exit(main())
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
webserver_1  |     args.func(args)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 47, in command
webserver_1  |     func = import_string(import_path)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/utils/module_loading.py", line 32, in import_string
webserver_1  |     module = import_module(module_path)
webserver_1  |   File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
webserver_1  |     return _bootstrap._gcd_import(name[level:], package, level)
webserver_1  |   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
webserver_1  |   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
webserver_1  |   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
webserver_1  |   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
webserver_1  |   File "<frozen importlib._bootstrap_external>", line 783, in exec_module
webserver_1  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/cli/commands/user_command.py", line 29, in <module>
webserver_1  |     from airflow.www.app import cached_app
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/www/app.py", line 38, in <module>
webserver_1  |     from airflow.www.extensions.init_views import (
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/www/extensions/init_views.py", line 29, in <module>
webserver_1  |     from airflow.www.views import lazy_add_provider_discovered_options_to_connection_form
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/www/views.py", line 95, in <module>
webserver_1  |     from airflow.www import auth, utils as wwwutils
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/www/utils.py", line 27, in <module>
webserver_1  |     from flask_appbuilder.models.sqla.interface import SQLAInterface
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/models/sqla/interface.py", line 16, in <module>
webserver_1  |     from sqlalchemy_utils.types.uuid import UUIDType
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy_utils/__init__.py", line 1, in <module>
webserver_1  |     from .aggregates import aggregated  # noqa
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy_utils/aggregates.py", line 372, in <module>
webserver_1  |     from .functions.orm import get_column_key
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy_utils/functions/__init__.py", line 1, in <module>
webserver_1  |     from .database import (  # noqa
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy_utils/functions/database.py", line 11, in <module>
webserver_1  |     from .orm import quote
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy_utils/functions/orm.py", line 14, in <module>
webserver_1  |     from sqlalchemy.orm.query import _ColumnEntity
webserver_1  | ImportError: cannot import name '_ColumnEntity' from 'sqlalchemy.orm.query' (/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py)
levyitay commented 3 years ago

@neylsoncrepalde The only thing that fixed that for me is adding pip install SQLAlchemy==1.3.23 \ in the Dockerfile

K-7 commented 3 years ago

I dont run celery , I use executor = LocalExecutor. After doing the migration to 2.0 when I run my DAG i get the following error

[INFO] Handling signal: ttin [INFO] Booting worker with pid: 32308 INFO - DAG test already has 18 active runs, not queuing any tasks for run 2021-04-15 [INFO] Worker exiting (pid: 458) [INFO] Handling signal: ttou

Not sure what exactly is wrong

cjphaha commented 3 years ago

This is a pull request to update your repo to work with Airflow 2.0.0 and Python 3.8. Everything was tested locally and is apparently working fine.

The main changes were:

  • update commands worker and flower to celery in scripts/entrypoint.sh
  • update image name in docker-compose files
  • update Dockerfile to use python:3.8-slim-buster
  • changed airflow.cfg for the 2.0.0 version default one
  • set up a default username and password to log in airflow (needed for 2.0.0 version).

Many thanks for your hard work putting this together and I hope this pull request can be handy.

thank u bro, u have solved my problem!👍

elliot-jsq commented 3 years ago

Are there any remaining issues for this PR or is this ready to go?

agung2001 commented 3 years ago

Hello @shihabcsedu09 and @rikr2sanchez.

Could you please provide information on the commands you ran to get the error you described?

I tried locally and everything is working fine. Maybe, the problem is that up to this point, there is no public docker image puckel/docker-airflow:2.0.0 so I had to build it. In order to do that, I ran

git clone https://github.com/neylsoncrepalde/docker-airflow.git
cd docker-airflow
docker build -t puckel/docker-airflow:2.0.0 .
docker-compose -f docker-compose-CeleryExecutor.yml up

Many thanks! Best!

thx mate, you save my day, I can run smoothly upgrading from version 1.10.9 to 2.0.0