lantrns-analytics / damn

The DAMN (Data Assets Metric Navigation) tool extracts and reports metrics about your data assets
https://pypi.org/project/damn-tool/
11 stars 0 forks source link

Support defining only an orchestrator connector #30

Closed olivierdupuis closed 1 year ago

olivierdupuis commented 1 year ago

At the moment, commands require specific types of connectors to get data. But we really only need an orchetrator connector type to get the basic functionalities working. When currently running a command with only an orchestrator connector, I get the following error...

olivierdupuis@MacBook-Pro damn % sudo poetry run damn metrics gdelt/gdelt_gkg_articles
Password:
Traceback (most recent call last):
  File "/Users/olivierdupuis/git/discursus/damn/damn_tool/utils/helpers.py", line 64, in load_config
    first_key = list(config[connector].keys())[0]
KeyError: 'io-manager'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/olivierdupuis/git/discursus/damn/.venv/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/olivierdupuis/git/discursus/damn/.venv/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/olivierdupuis/git/discursus/damn/.venv/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/olivierdupuis/git/discursus/damn/.venv/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/olivierdupuis/git/discursus/damn/.venv/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/olivierdupuis/git/discursus/damn/damn_tool/metrics.py", line 164, in metrics
    orchestrator_connector, io_manager_connector, data_warehouse_connector = init_connectors(orchestrator, io_manager, data_warehouse)
  File "/Users/olivierdupuis/git/discursus/damn/damn_tool/utils/helpers.py", line 32, in init_connectors
    io_manager_connector_type, io_manager_config = load_config('io-manager', io_manager)
  File "/Users/olivierdupuis/git/discursus/damn/damn_tool/utils/helpers.py", line 69, in load_config
    raise ValueError(f"No configuration found for connector '{connector}' with profile '{profile}'")
ValueError: No configuration found for connector 'io-manager' with profile 'None'
olivierdupuis commented 1 year ago

30 - Gracefully fail when passing an non-existant asset name

olivierdupuis commented 1 year ago

30 - Support defining only an orchestrator connector