search

quantile-development / dagster-meltano

A Dagster plugin that allows you to run Meltano in Dagster
MIT License
41 stars 17 forks source link

FileNotFoundError: [Errno 2] No such file or directory: 'meltano' #53

Open ilkkapeltola opened 4 months ago

ilkkapeltola commented 4 months ago

When I run this line:

meltano_jobs = load_jobs_from_meltano_project(meltano_project_dir)

I get an error 

2024-02-14 22:33:00 +0200 - dagster.daemon.SchedulerDaemon - WARNING - Could not load location tutorial to check for schedules due to the following error: dagster._core.errors.DagsterUserCodeLoadError: Error occurred during the loading of Dagster definitions in
executable_path=/home/ilkka/.venv/dagster/bin/python, module_name=tutorial, working_directory=/home/ilkka/ageras-repos/tutorial

Stack Trace:
  File "/home/ilkka/.venv/dagster/lib/python3.10/site-packages/dagster/_grpc/server.py", line 408, in __init__
    self._loaded_repositories: Optional[LoadedRepositories] = LoadedRepositories(
  File "/home/ilkka/.venv/dagster/lib/python3.10/site-packages/dagster/_grpc/server.py", line 235, in __init__
    with user_code_error_boundary(
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/ilkka/.venv/dagster/lib/python3.10/site-packages/dagster/_core/errors.py", line 297, in user_code_error_boundary
    raise new_error from e

The above exception was caused by the following exception:
FileNotFoundError: [Errno 2] No such file or directory: 'meltano'

Stack Trace:
  File "/home/ilkka/.venv/dagster/lib/python3.10/site-packages/dagster/_core/errors.py", line 287, in user_code_error_boundary
    yield
  File "/home/ilkka/.venv/dagster/lib/python3.10/site-packages/dagster/_grpc/server.py", line 242, in __init__
    loadable_targets = get_loadable_targets(
  File "/home/ilkka/.venv/dagster/lib/python3.10/site-packages/dagster/_grpc/utils.py", line 50, in get_loadable_targets
    else loadable_targets_from_python_module(module_name, working_directory)
  File "/home/ilkka/.venv/dagster/lib/python3.10/site-packages/dagster/_core/workspace/autodiscovery.py", line 35, in loadable_targets_from_python_module
    module = load_python_module(
  File "/home/ilkka/.venv/dagster/lib/python3.10/site-packages/dagster/_core/code_pointer.py", line 135, in load_python_module
    return importlib.import_module(module_name)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/ilkka/ageras-repos/tutorial/tutorial/__init__.py", line 4, in <module>
    from .assets import ageras_group_dbt_asset
  File "/home/ilkka/ageras-repos/tutorial/tutorial/assets.py", line 9, in <module>
    meltano_jobs = load_jobs_from_meltano_project(meltano_project_dir)
  File "/home/ilkka/.venv/dagster/lib/python3.10/site-packages/dagster_meltano/generation.py", line 34, in load_jobs_from_meltano_project
    return list(meltano_jobs)
  File "/home/ilkka/.venv/dagster/lib/python3.10/site-packages/dagster_meltano/meltano_resource.py", line 148, in jobs
    for meltano_job in self.meltano_jobs:
  File "/usr/lib/python3.10/functools.py", line 981, in __get__
    val = self.func(instance)
  File "/home/ilkka/.venv/dagster/lib/python3.10/site-packages/dagster_meltano/meltano_resource.py", line 125, in meltano_jobs
    meltano_job_list = self.meltano_yaml["jobs"]
  File "/usr/lib/python3.10/functools.py", line 981, in __get__
    val = self.func(instance)
  File "/home/ilkka/.venv/dagster/lib/python3.10/site-packages/dagster_meltano/meltano_resource.py", line 120, in meltano_yaml
    jobs, schedules = asyncio.run(self.gather_meltano_yaml_information())
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/ilkka/.venv/dagster/lib/python3.10/site-packages/dagster_meltano/meltano_resource.py", line 106, in gather_meltano_yaml_information
    jobs, schedules = await asyncio.gather(
  File "/home/ilkka/.venv/dagster/lib/python3.10/site-packages/dagster_meltano/meltano_resource.py", line 88, in load_json_from_cli
    proc = await asyncio.create_subprocess_exec(
  File "/usr/lib/python3.10/asyncio/subprocess.py", line 218, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1681, in subprocess_exec
    transport = await self._make_subprocess_transport(
  File "/usr/lib/python3.10/asyncio/unix_events.py", line 207, in _make_subprocess_transport
    transp = _UnixSubprocessTransport(self, protocol, args, shell,
  File "/usr/lib/python3.10/asyncio/base_subprocess.py", line 36, in __init__
    self._start(args=args, shell=shell, stdin=stdin, stdout=stdout,
  File "/usr/lib/python3.10/asyncio/unix_events.py", line 799, in _start
    self._proc = subprocess.Popen(
  File "/usr/lib/python3.10/subprocess.py", line 971, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.10/subprocess.py", line 1863, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
though the folder path is correct.
using wsl
SebRollen commented 1 month ago

@ilkkapeltola I know this was from a few months ago, but I just ran into the same issue and resolved it; here's what worked for me in case anyone runs into this issue in the future:

The error message was a bit of a red-herring since I went looking for files/directories with the name meltano, but in my case, I had installed dagster-meltano into my virtual environment, but not meltano itself. Adding meltano to my virtual env resolved the issue