Airflow install fails on fresh Meltano 3.0 install

konung commented 1 year ago

Hi. Trying out Airflow utility, ran into problems on mac.

This a new Meltano 3.0 install on osx 13.6

Fresh new Python 3.11.5 using asdf.

Able to install a few other utilities, extractors and loaders, but not airflow.

Here is the error message:


> meltano --log-level=debug add utility airflow                                             1561ms  Mon Sep 25 23:22:05 2023
2023-09-26T04:22:08.904344Z [debug    ] /etc/localtime found
2023-09-26T04:22:08.906191Z [debug    ] 1 found:
 {'/etc/localtime is a symlink to': 'America/Chicago'}
2023-09-26T04:22:08.912804Z [debug    ] Creating DB engine for project at '/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt' with DB URI 'sqlite://Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/meltano.db'
2023-09-26T04:22:09.760607Z [debug    ] Found plugin parent            parent=airflow plugin=airflow source=<DefinitionSource.HUB: 1>
2023-09-26T04:22:09.815159Z [debug    ] Locked plugin definition       path=PosixPath('/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/plugins/utilities/airflow--apache.lock')
Added utility 'airflow' to your Meltano project
Variant:    apache (default)
Repository: https://github.com/apache/airflow
Documentation:  https://hub.meltano.com/utilities/airflow--apache

Installing utility 'airflow'...
2023-09-26T04:22:09.915768Z [debug    ] Packages for 'utilities/airflow' have changed so performing a clean install.
2023-09-26T04:22:09.916138Z [debug    ] Removed old virtual environment for 'utilities/airflow'
2023-09-26T04:22:09.916194Z [debug    ] Creating virtual environment for 'utilities/airflow'
2023-09-26T04:22:10.554591Z [debug    ] Upgrading pip for 'utilities/airflow'
2023-09-26T04:22:10.554722Z [debug    ] Upgrading with args '--upgrade pip' in existing virtual environment for 'utilities/airflow'
2023-09-26T04:22:12.852463Z [debug    ] Installing with args 'git+https://github.com/meltano/airflow-ext.git@main apache-airflow==2.3.3 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.3.3/constraints-no-providers-3.11.txt' into virtual environment for 'utilities/airflow'
Utility 'airflow' could not be installed: failed to install plugin 'airflow'.

2023-09-26T04:22:14.022794Z [debug    ] Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

Failed to install plugin(s)
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/__init__.py:102 │
│ in _run_cli                                                                                      │
│                                                                                                  │
│    99 │   """                                                                                    │
│   100 │   try:                                                                                   │
│   101 │   │   try:  # noqa: WPS225, WPS505                                                       │
│ ❱ 102 │   │   │   cli(obj={"project": None})                                                     │
│   103 │   │   except ProjectReadonly as err:                                                     │
│   104 │   │   │   raise CliError(                                                                │
│   105 │   │   │   │   f"The requested action could not be completed: {err}",                     │
│                                                                                                  │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/click/core.py:1157 in       │
│ __call__                                                                                         │
│                                                                                                  │
│   1154 │                                                                                         │
│   1155 │   def __call__(self, *args: t.Any, **kwargs: t.Any) -> t.Any:                           │
│   1156 │   │   """Alias for :meth:`main`."""                                                     │
│ ❱ 1157 │   │   return self.main(*args, **kwargs)                                                 │
│   1158                                                                                           │
│   1159                                                                                           │
│   1160 class Command(BaseCommand):                                                               │
│                                                                                                  │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/cli.py:42 in    │
│ main                                                                                             │
│                                                                                                  │
│    39 │   │   │   args: Positional arguments for the Click group.                                │
│    40 │   │   │   kwargs: Keyword arguments for the Click group.                                 │
│    41 │   │   """                                                                                │
│ ❱  42 │   │   return super().main(*args, windows_expand_args=False, **kwargs)                    │
│    43                                                                                            │
│    44                                                                                            │
│    45 @click.group(                                                                              │
│                                                                                                  │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/click/core.py:1078 in main  │
│                                                                                                  │
│   1075 │   │   try:                                                                              │
│   1076 │   │   │   try:                                                                          │
│   1077 │   │   │   │   with self.make_context(prog_name, args, **extra) as ctx:                  │
│ ❱ 1078 │   │   │   │   │   rv = self.invoke(ctx)                                                 │
│   1079 │   │   │   │   │   if not standalone_mode:                                               │
│   1080 │   │   │   │   │   │   return rv                                                         │
│   1081 │   │   │   │   │   # it's not safe to `ctx.exit(rv)` here!                               │
│                                                                                                  │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/utils.py:612 in │
│ invoke                                                                                           │
│                                                                                                  │
│   609 │   │   enact_environment_behavior(self.environment_behavior, ctx)                         │
│   610 │   │   if ctx.obj.get("tracker"):                                                         │
│   611 │   │   │   ctx.obj["tracker"].add_contexts(CliContext.from_click_context(ctx))            │
│ ❱ 612 │   │   super().invoke(ctx)                                                                │
│   613                                                                                            │
│   614                                                                                            │
│   615 class InstrumentedDefaultGroup(InstrumentedGroupMixin, DefaultGroup, DYMGroup):            │
│                                                                                                  │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/click/core.py:1688 in       │
│ invoke                                                                                           │
│                                                                                                  │
│   1685 │   │   │   │   super().invoke(ctx)                                                       │
│   1686 │   │   │   │   sub_ctx = cmd.make_context(cmd_name, args, parent=ctx)                    │
│   1687 │   │   │   │   with sub_ctx:                                                             │
│ ❱ 1688 │   │   │   │   │   return _process_result(sub_ctx.command.invoke(sub_ctx))               │
│   1689 │   │                                                                                     │
│   1690 │   │   # In chain mode we create the contexts step by step, but after the                │
│   1691 │   │   # base command has been invoked.  Because at that point we do not                 │
│                                                                                                  │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/utils.py:661 in │
│ invoke                                                                                           │
│                                                                                                  │
│   658 │   │   if ctx.obj.get("tracker"):                                                         │
│   659 │   │   │   ctx.obj["tracker"].add_contexts(CliContext.from_click_context(ctx))            │
│   660 │   │   │   ctx.obj["tracker"].track_command_event(CliEvent.started)                       │
│ ❱ 661 │   │   super().invoke(ctx)                                                                │
│   662                                                                                            │
│                                                                                                  │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/click/core.py:1434 in       │
│ invoke                                                                                           │
│                                                                                                  │
│   1431 │   │   │   echo(style(message, fg="red"), err=True)                                      │
│   1432 │   │                                                                                     │
│   1433 │   │   if self.callback is not None:                                                     │
│ ❱ 1434 │   │   │   return ctx.invoke(self.callback, **ctx.params)                                │
│   1435 │                                                                                         │
│   1436 │   def shell_complete(self, ctx: Context, incomplete: str) -> t.List["CompletionItem"]:  │
│   1437 │   │   """Return a list of completions for the incomplete value. Looks                   │
│                                                                                                  │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/click/core.py:783 in invoke │
│                                                                                                  │
│    780 │   │                                                                                     │
│    781 │   │   with augment_usage_errors(__self):                                                │
│    782 │   │   │   with ctx:                                                                     │
│ ❱  783 │   │   │   │   return __callback(*args, **kwargs)                                        │
│    784 │                                                                                         │
│    785 │   def forward(                                                                          │
│    786 │   │   __self, __cmd: "Command", *args: t.Any, **kwargs: t.Any  # noqa: B902             │
│                                                                                                  │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/params.py:27 in │
│ decorate                                                                                         │
│                                                                                                  │
│   24 │   │   if database_uri:                                                                    │
│   25 │   │   │   ProjectSettingsService.config_override["database_uri"] = database_uri           │
│   26 │   │                                                                                       │
│ ❱ 27 │   │   return func(*args, **kwargs)                                                        │
│   28 │                                                                                           │
│   29 │   return functools.update_wrapper(decorate, func)                                         │
│   30                                                                                             │
│                                                                                                  │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/params.py:75 in │
│ decorate                                                                                         │
│                                                                                                  │
│   72 │   │   │   │   except MigrationError as err:                                               │
│   73 │   │   │   │   │   raise CliError(str(err)) from err                                       │
│   74 │   │   │                                                                                   │
│ ❱ 75 │   │   │   func(project, *args, **kwargs)                                                  │
│   76 │   │                                                                                       │
│   77 │   │   return functools.update_wrapper(decorate, func)                                     │
│   78                                                                                             │
│                                                                                                  │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/click/decorators.py:33 in   │
│ new_func                                                                                         │
│                                                                                                  │
│    30 │   """                                                                                    │
│    31 │                                                                                          │
│    32 │   def new_func(*args: "P.args", **kwargs: "P.kwargs") -> "R":                            │
│ ❱  33 │   │   return f(get_current_context(), *args, **kwargs)                                   │
│    34 │                                                                                          │
│    35 │   return update_wrapper(new_func, f)                                                     │
│    36                                                                                            │
│                                                                                                  │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/add.py:163 in   │
│ add                                                                                              │
│                                                                                                  │
│   160 │   │                                                                                      │
│   161 │   │   if not success:                                                                    │
│   162 │   │   │   tracker.track_command_event(CliEvent.failed)                                   │
│ ❱ 163 │   │   │   raise CliError("Failed to install plugin(s)")                                  │
│   164 │                                                                                          │
│   165 │   _print_plugins(plugins)                                                                │
│   166 │   tracker.track_command_event(CliEvent.completed)                                        │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
CliError: Failed to install plugin(s)

The above exception was the direct cause of the following exception:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/__init__.py:112 │
│ in _run_cli                                                                                      │
│                                                                                                  │
│   109 │   │   except MeltanoError as err:                                                        │
│   110 │   │   │   handle_meltano_error(err)                                                      │
│   111 │   │   except Exception as err:                                                           │
│ ❱ 112 │   │   │   raise CliError(f"{troubleshooting_message}\n{err}") from err                   │
│   113 │   except CliError as cli_error:                                                          │
│   114 │   │   cli_error.print()                                                                  │
│   115 │   │   sys.exit(1)                                                                        │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
CliError: Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

Failed to install plugin(s)

Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

Failed to install plugin(s)

edgarrmondragon commented 1 year ago

Can probably be fixed by updating the pip_url in meltano.yml:

- pip_url: git+https://github.com/meltano/airflow-ext.git@main apache-airflow==2.3.3
-   --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.3.3/constraints-no-providers-${MELTANO__PYTHON_VERSION}.txt
+ pip_url: git+https://github.com/meltano/airflow-ext.git@main apache-airflow==2.7.1
+   --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.7.1/constraints-no-providers-${MELTANO__PYTHON_VERSION}.txt

If that fixes the problem, then the plugin should be updated: https://github.com/meltano/hub/blob/0fde97d2d899a7e1127259304d890097715a4b5a/_data/meltano/utilities/airflow/apache.yml#L54-L55

konung commented 1 year ago

Thank you for the suggestion. No luck


meltano --log-level=debug install                                                                                                                                                                                          57.9s  Tue Sep 26 04:30:25 2023
2023-09-26T09:31:30.217656Z [debug    ] /etc/localtime found
2023-09-26T09:31:30.219564Z [debug    ] 1 found:
 {'/etc/localtime is a symlink to': 'America/Chicago'}
2023-09-26T09:31:30.228482Z [debug    ] Creating DB engine for project at '/Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt' with DB URI 'sqlite://Users/konung/Projects/konungshop/Code/konungshop-elt-meltano/konungshop-elt/.meltano/meltano.db'
2023-09-26T09:31:30.309150Z [debug    ] Found plugin parent            parent=tap-spreadsheets-anywhere plugin=tap-spreadsheets-anywhere source=<DefinitionSource.LOCKFILE: 4>
2023-09-26T09:31:30.310576Z [debug    ] Found plugin parent            parent=tap-shopify plugin=tap-shopify source=<DefinitionSource.LOCKFILE: 4>
2023-09-26T09:31:30.312584Z [debug    ] Found plugin parent            parent=target-postgres plugin=target-postgres source=<DefinitionSource.LOCKFILE: 4>
2023-09-26T09:31:30.313481Z [debug    ] Found plugin parent            parent=target-jsonl plugin=target-jsonl source=<DefinitionSource.LOCKFILE: 4>
2023-09-26T09:31:30.314407Z [debug    ] Found plugin parent            parent=target-csv plugin=target-csv source=<DefinitionSource.LOCKFILE: 4>
2023-09-26T09:31:30.316423Z [debug    ] Found plugin parent            parent=airflow plugin=airflow source=<DefinitionSource.LOCKFILE: 4>
Installing 6 plugins...
Installing extractor 'tap-spreadsheets-anywhere'...
2023-09-26T09:31:30.401483Z [debug    ] Installing with args 'git+https://github.com/ets/tap-spreadsheets-anywhere.git' into virtual environment for 'extractors/tap-spreadsheets-anywhere'
Installing extractor 'tap-shopify'...
2023-09-26T09:31:30.677982Z [debug    ] Installing with args 'git+https://github.com/sehnem/tap-shopify.git' into virtual environment for 'extractors/tap-shopify'
Installing loader 'target-postgres'...
2023-09-26T09:31:30.889848Z [debug    ] Installing with args 'meltanolabs-target-postgres~=0.0.7' into virtual environment for 'loaders/target-postgres'
Installing loader 'target-jsonl'...
2023-09-26T09:31:30.974898Z [debug    ] Installing with args 'target-jsonl' into virtual environment for 'loaders/target-jsonl'
Installing loader 'target-csv'...
2023-09-26T09:31:31.061081Z [debug    ] Installing with args 'git+https://github.com/hotgluexyz/target-csv.git' into virtual environment for 'loaders/target-csv'
Installing utility 'airflow'...
2023-09-26T09:31:31.182350Z [debug    ] Packages for 'utilities/airflow' have changed so performing a clean install.
2023-09-26T09:31:31.349110Z [debug    ] Removed old virtual environment for 'utilities/airflow'
2023-09-26T09:31:31.349200Z [debug    ] Creating virtual environment for 'utilities/airflow'
2023-09-26T09:31:32.068699Z [debug    ] Upgrading pip for 'utilities/airflow'
2023-09-26T09:31:32.068854Z [debug    ] Upgrading with args '--upgrade pip' in existing virtual environment for 'utilities/airflow'
Installed loader 'target-jsonl'
Installed loader 'target-postgres'
2023-09-26T09:31:34.076076Z [debug    ] Installing with args 'git+https://github.com/meltano/airflow-ext.git@main apache-airflow==2.7.1 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.7.1/constraints-no-providers-3.11.txt' into virtual environment for 'utilities/airflow'
Installed loader 'target-csv'
Installed extractor 'tap-spreadsheets-anywhere'
Installed extractor 'tap-shopify'
Utility 'airflow' could not be installed: failed to install plugin 'airflow'.

Installed 5/6 plugins
2023-09-26T09:32:10.328108Z [debug    ] Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

Failed to install plugin(s)
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/__init__.py:102 │
│ in _run_cli                                                                                      │
│                                                                                                  │
│    99 │   """                                                                                    │
│   100 │   try:                                                                                   │
│   101 │   │   try:  # noqa: WPS225, WPS505                                                       │
│ ❱ 102 │   │   │   cli(obj={"project": None})                                                     │
│   103 │   │   except ProjectReadonly as err:                                                     │
│   104 │   │   │   raise CliError(                                                                │
│   105 │   │   │   │   f"The requested action could not be completed: {err}",                     │
│                                                                                                  │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/click/core.py:1157 in       │
│ __call__                                                                                         │
│                                                                                                  │
│   1154 │                                                                                         │
│   1155 │   def __call__(self, *args: t.Any, **kwargs: t.Any) -> t.Any:                           │
│   1156 │   │   """Alias for :meth:`main`."""                                                     │
│ ❱ 1157 │   │   return self.main(*args, **kwargs)                                                 │
│   1158                                                                                           │
│   1159                                                                                           │
│   1160 class Command(BaseCommand):                                                               │
│                                                                                                  │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/cli.py:42 in    │
│ main                                                                                             │
│                                                                                                  │
│    39 │   │   │   args: Positional arguments for the Click group.                                │
│    40 │   │   │   kwargs: Keyword arguments for the Click group.                                 │
│    41 │   │   """                                                                                │
│ ❱  42 │   │   return super().main(*args, windows_expand_args=False, **kwargs)                    │
│    43                                                                                            │
│    44                                                                                            │
│    45 @click.group(                                                                              │
│                                                                                                  │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/click/core.py:1078 in main  │
│                                                                                                  │
│   1075 │   │   try:                                                                              │
│   1076 │   │   │   try:                                                                          │
│   1077 │   │   │   │   with self.make_context(prog_name, args, **extra) as ctx:                  │
│ ❱ 1078 │   │   │   │   │   rv = self.invoke(ctx)                                                 │
│   1079 │   │   │   │   │   if not standalone_mode:                                               │
│   1080 │   │   │   │   │   │   return rv                                                         │
│   1081 │   │   │   │   │   # it's not safe to `ctx.exit(rv)` here!                               │
│                                                                                                  │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/utils.py:612 in │
│ invoke                                                                                           │
│                                                                                                  │
│   609 │   │   enact_environment_behavior(self.environment_behavior, ctx)                         │
│   610 │   │   if ctx.obj.get("tracker"):                                                         │
│   611 │   │   │   ctx.obj["tracker"].add_contexts(CliContext.from_click_context(ctx))            │
│ ❱ 612 │   │   super().invoke(ctx)                                                                │
│   613                                                                                            │
│   614                                                                                            │
│   615 class InstrumentedDefaultGroup(InstrumentedGroupMixin, DefaultGroup, DYMGroup):            │
│                                                                                                  │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/click/core.py:1688 in       │
│ invoke                                                                                           │
│                                                                                                  │
│   1685 │   │   │   │   super().invoke(ctx)                                                       │
│   1686 │   │   │   │   sub_ctx = cmd.make_context(cmd_name, args, parent=ctx)                    │
│   1687 │   │   │   │   with sub_ctx:                                                             │
│ ❱ 1688 │   │   │   │   │   return _process_result(sub_ctx.command.invoke(sub_ctx))               │
│   1689 │   │                                                                                     │
│   1690 │   │   # In chain mode we create the contexts step by step, but after the                │
│   1691 │   │   # base command has been invoked.  Because at that point we do not                 │
│                                                                                                  │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/utils.py:661 in │
│ invoke                                                                                           │
│                                                                                                  │
│   658 │   │   if ctx.obj.get("tracker"):                                                         │
│   659 │   │   │   ctx.obj["tracker"].add_contexts(CliContext.from_click_context(ctx))            │
│   660 │   │   │   ctx.obj["tracker"].track_command_event(CliEvent.started)                       │
│ ❱ 661 │   │   super().invoke(ctx)                                                                │
│   662                                                                                            │
│                                                                                                  │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/click/core.py:1434 in       │
│ invoke                                                                                           │
│                                                                                                  │
│   1431 │   │   │   echo(style(message, fg="red"), err=True)                                      │
│   1432 │   │                                                                                     │
│   1433 │   │   if self.callback is not None:                                                     │
│ ❱ 1434 │   │   │   return ctx.invoke(self.callback, **ctx.params)                                │
│   1435 │                                                                                         │
│   1436 │   def shell_complete(self, ctx: Context, incomplete: str) -> t.List["CompletionItem"]:  │
│   1437 │   │   """Return a list of completions for the incomplete value. Looks                   │
│                                                                                                  │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/click/core.py:783 in invoke │
│                                                                                                  │
│    780 │   │                                                                                     │
│    781 │   │   with augment_usage_errors(__self):                                                │
│    782 │   │   │   with ctx:                                                                     │
│ ❱  783 │   │   │   │   return __callback(*args, **kwargs)                                        │
│    784 │                                                                                         │
│    785 │   def forward(                                                                          │
│    786 │   │   __self, __cmd: "Command", *args: t.Any, **kwargs: t.Any  # noqa: B902             │
│                                                                                                  │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/click/decorators.py:33 in   │
│ new_func                                                                                         │
│                                                                                                  │
│    30 │   """                                                                                    │
│    31 │                                                                                          │
│    32 │   def new_func(*args: "P.args", **kwargs: "P.kwargs") -> "R":                            │
│ ❱  33 │   │   return f(get_current_context(), *args, **kwargs)                                   │
│    34 │                                                                                          │
│    35 │   return update_wrapper(new_func, f)                                                     │
│    36                                                                                            │
│                                                                                                  │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/params.py:27 in │
│ decorate                                                                                         │
│                                                                                                  │
│   24 │   │   if database_uri:                                                                    │
│   25 │   │   │   ProjectSettingsService.config_override["database_uri"] = database_uri           │
│   26 │   │                                                                                       │
│ ❱ 27 │   │   return func(*args, **kwargs)                                                        │
│   28 │                                                                                           │
│   29 │   return functools.update_wrapper(decorate, func)                                         │
│   30                                                                                             │
│                                                                                                  │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/params.py:75 in │
│ decorate                                                                                         │
│                                                                                                  │
│   72 │   │   │   │   except MigrationError as err:                                               │
│   73 │   │   │   │   │   raise CliError(str(err)) from err                                       │
│   74 │   │   │                                                                                   │
│ ❱ 75 │   │   │   func(project, *args, **kwargs)                                                  │
│   76 │   │                                                                                       │
│   77 │   │   return functools.update_wrapper(decorate, func)                                     │
│   78                                                                                             │
│                                                                                                  │
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/install.py:110  │
│ in install                                                                                       │
│                                                                                                  │
│   107 │   )                                                                                      │
│   108 │   if not success:                                                                        │
│   109 │   │   tracker.track_command_event(CliEvent.failed)                                       │
│ ❱ 110 │   │   raise CliError("Failed to install plugin(s)")                                      │
│   111 │   tracker.track_command_event(CliEvent.completed)                                        │
│   112                                                                                            │
│   113                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
CliError: Failed to install plugin(s)

The above exception was the direct cause of the following exception:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/konung/.local/pipx/venvs/meltano/lib/python3.11/site-packages/meltano/cli/__init__.py:112 │
│ in _run_cli                                                                                      │
│                                                                                                  │
│   109 │   │   except MeltanoError as err:                                                        │
│   110 │   │   │   handle_meltano_error(err)                                                      │
│   111 │   │   except Exception as err:                                                           │
│ ❱ 112 │   │   │   raise CliError(f"{troubleshooting_message}\n{err}") from err                   │
│   113 │   except CliError as cli_error:                                                          │
│   114 │   │   cli_error.print()                                                                  │
│   115 │   │   sys.exit(1)                                                                        │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
CliError: Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

Failed to install plugin(s)

Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

Failed to install plugin(s)

konung commented 1 year ago

This might be a Mac only issue. I was able to install airflow utility plugin -v 2.3.3 inside Meltano docker project.

lclalalalala commented 9 months ago

Can probably be fixed by updating the pip_url in meltano.yml:
- pip_url: git+https://github.com/meltano/airflow-ext.git@main apache-airflow==2.3.3
-   --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.3.3/constraints-no-providers-${MELTANO__PYTHON_VERSION}.txt
+ pip_url: git+https://github.com/meltano/airflow-ext.git@main apache-airflow==2.7.1
+   --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.7.1/constraints-no-providers-${MELTANO__PYTHON_VERSION}.txt
If that fixes the problem, then the plugin should be updated: https://github.com/meltano/hub/blob/0fde97d2d899a7e1127259304d890097715a4b5a/_data/meltano/utilities/airflow/apache.yml#L54-L55

Thanks. This works for me.

lclalalalala commented 9 months ago

This might be a Mac only issue. I was able to install airflow utility plugin -v 2.3.3 inside Meltano docker project.

Yes. I change to apache-airflow==2.8.0. works fine in docker container. Failed on MacOS. on Mac I used anaconda virual env. Python=3.11.

edgarrmondragon commented 9 months ago

Yes. I change to apache-airflow==2.8.0. works fine in docker container. Failed on MacOS. on Mac I used anaconda virual env. Python=3.11.

I guess it's a problem for newish Apple Silicon macs, probably because there aren't any compatibles wheels for some package, and the build step is failing.

edgarrmondragon commented 9 months ago

For ref: https://github.com/meltano/hub/pull/1647

meltano / airflow-ext

Airflow install fails on fresh Meltano 3.0 install #48