Closed netphantom closed 7 months ago
Hey @netphantom π
Looks like you have not added the config patterns needed to load the Neptune config to your project's settings.py
-> CONFIG_LOADER_ARGS
.
You will need to update CONFIG_LOADER_ARGS
as below:
CONFIG_LOADER_ARGS = {
...,
"config_patterns": {
...,
"credentials_neptune" : ["credentials_neptune*"],
"neptune": ["neptune*"],
}
}
Here is the step-by-step installation and setup guide for kedro-neptune
: https://docs.neptune.ai/integrations/kedro/
Please let me know if this helps, or if there is anything else I can help you with.
Hi @SiddhantSadangi , thank you for your welcome :).
I have updated the CONFIG_LOADER_ARGS
, however as the init command fails, it does not create the yaml files in the config folders with the required credentials. As a matter of fact, if I add the config_patterns
and I run the kedro neptune init, I have the following error:
Traceback (most recent call last):
File "
File "C:\Users\kedro-project\venv\Lib\site-packages\kedro\config\omegaconf_config.py", line 244, in __getitem__
raise MissingConfigException(
kedro.config.abstract_config.MissingConfigException: No files of YAML or JSON format found in conf\base or conf\local matching the glob pattern(s): ['neptune*']
If I manually create the files, it keeps going with other errors/configurations missing
Could you share the config_patterns
you are using?
Looks like you might have missed adding the *
at the end of "neptune*"
:)
Sure, here it is:
CONFIG_LOADER_ARGS = {
"custom_resolvers": {
"merge": merge_dicts,
},
"config_patterns": {
"credentials_neptune": ["credentials_neptune*"],
"neptune": ["neptune*"],
}
}
Thats'weird.
I am not able to reproduce this issue. Could you take me through the steps you followed, in order, while creating your kedro project, and share the structure of your project?
The settings.py
file is used only for running the kedro project, and does not need to be updated before you run kedro neptune init
.
Also, from your previous traceback, I would assume that the pattern in your config_patterns
was ["neptune"]
, and not ["neptune*"]
, but that doesn't seem to be the case. Can you reconfirm that you are not able to run your project using the config_patterns
you shared?
Hi @SiddhantSadangi, the steps I did were simple:
pip install -U kedro-neptune
(I also tried pip install -U "neptune[kedro]"
but it didn't change the procedure outcome)kedro neptune init
, receiving the error I mentioned.I cloned the same repository I am using on a Linux (Manjaro) machine, run kedro neptune init
and everything worked perfectly.
It seems to be some weirdo Windows problem, I don't know where to look to solve this. By the way this is the project structure:
.
βββ conf
βΒ Β βββ base
βΒ Β βΒ Β βββ catalog.yml
βΒ Β βΒ Β βββ logging.yml
βΒ Β βΒ Β βββ parameters_pipeline1.yml
βΒ Β βΒ Β βββ parameters_pipeline2.yml
βΒ Β βΒ Β βββ parameters_pipeline3.yml
βΒ Β βΒ Β βββ parameters_pipeline4.yml
βΒ Β βΒ Β βββ parameters.yml
βΒ Β βββ local
βΒ Β βΒ Β βββ credentials.yml
βΒ Β βββ README.md
βββ docs
βΒ Β βββ source
βΒ Β βββ conf.py
βΒ Β βββ index.rst
βββ info.log
βββ notebooks
βββ pyproject.toml
βββ README.md
βββ src
βΒ Β βββ lstm
βΒ Β βΒ Β βββ extras
βΒ Β βΒ Β βΒ Β βββ dataset
βΒ Β βΒ Β βΒ Β βΒ Β βββ __init__.py
βΒ Β βΒ Β βΒ Β βΒ Β βββ torch_set.py
βΒ Β βΒ Β βΒ Β βββ __init__.py
βΒ Β βΒ Β βββ __init__.py
βΒ Β βΒ Β βββ __main__.py
βΒ Β βΒ Β βββ pipeline_registry.py
βΒ Β βΒ Β βββ pipelines
βΒ Β βΒ Β βΒ Β βββ pipeline1
βΒ Β βΒ Β βΒ Β βΒ Β βββ accessories.py
βΒ Β βΒ Β βΒ Β βΒ Β βββ __init__.py
βΒ Β βΒ Β βΒ Β βΒ Β βββ nodes.py
βΒ Β βΒ Β βΒ Β βΒ Β βββ pipeline.py
βΒ Β βΒ Β βΒ Β βββ __init__.py
βΒ Β βΒ Β βΒ Β βββ pipeline2
βΒ Β βΒ Β βΒ Β βΒ Β βββ __init__.py
βΒ Β βΒ Β βΒ Β βΒ Β βββ nodes.py
βΒ Β βΒ Β βΒ Β βΒ Β βββ pipeline.py
βΒ Β βΒ Β βΒ Β βββ pipeline3
βΒ Β βΒ Β βΒ Β βΒ Β βββ __init__.py
βΒ Β βΒ Β βΒ Β βΒ Β βββ nodes.py
βΒ Β βΒ Β βΒ Β βΒ Β βββ pipeline.py
βΒ Β βΒ Β βΒ Β βββ pipeline4
βΒ Β βΒ Β βΒ Β βββ __init__.py
βΒ Β βΒ Β βΒ Β βββ nodes.py
βΒ Β βΒ Β βΒ Β βββ pipeline.py
βΒ Β βΒ Β βββ settings.py
βΒ Β βββ pyproject.toml
βΒ Β βββ requirements.txt
βΒ Β βββ tests
βΒ Β βΒ Β βββ __init__.py
βΒ Β βΒ Β βββ pipelines
βΒ Β βΒ Β βΒ Β βββ pipeline1
βΒ Β βΒ Β βΒ Β βΒ Β βββ __init__.py
βΒ Β βΒ Β βΒ Β βΒ Β βββ test_pipeline.py
βΒ Β βΒ Β βΒ Β βββ __init__.py
βΒ Β βΒ Β βΒ Β βββ pipeline2
βΒ Β βΒ Β βΒ Β βΒ Β βββ __init__.py
βΒ Β βΒ Β βΒ Β βΒ Β βββ test_pipeline.py
βΒ Β βΒ Β βΒ Β βββ pipeline3
βΒ Β βΒ Β βΒ Β βΒ Β βββ __init__.py
βΒ Β βΒ Β βΒ Β βΒ Β βββ test_pipeline.py
βΒ Β βΒ Β βΒ Β βββ pipeline4
βΒ Β βΒ Β βΒ Β βββ __init__.py
βΒ Β βΒ Β βΒ Β βββ test_pipeline.py
βΒ Β βΒ Β βββ test_run.py
βΒ Β βββ utilities
βΒ Β βββ data.py
βΒ Β βββ __init__.py
βΒ Β βββ model.py
βΒ Β βββ plotting.py
βΒ Β βββ testing.py
βΒ Β βββ training.py
βββ stats.json
I am not able to reproduce this error on a Windows machine too π
Would it be possible for you to share a minimal project (as a ZIP file, or a GitHub repo) that I can use to reproduce this?
Can you also try upgrading kedro to the latest version - 0.19.2?
Unfortunately I cannot share the code and the project I am using as there are sensible information.
However, I just tried the simpliest thing: I removed and reinstalled kedro and kedro-telemetry (always the 0.18.14, due to catalog compatibility issues), run kedro neptune init... and it worked. So, I could imagine that the problem was some dirty in the python venv.
Glad that you were able to get this resolved! π
Since it seems to be an issue with kedro
/kedro-telemetry
, can you try raising an issue in kedro?
Meanwhile, I am closing this. But please feel free to reopen or comment if need be π€
Describe the bug
I am following the guide to integrate Neptune into my Kedro project but I am unable to run the command kedro neptune init. When I run it, I have the error: KeyError: "No config patterns were found for 'credentials_neptune' in your config loader"
Reproduction
I have an existing Kedro project, I run into the project folder and run the init command, provide the API and the project information
Expected behavior
The initialized neptune project folder
Traceback
Traceback (most recent call last): File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "C:\Users\kedro-project\venv\Scripts\kedro.exe__main.py", line 7, in
File "C:\Users\kedro-project\venv\Lib\site-packages\kedro\framework\cli\cli.py", line 211, in main
cli_collection()
File "C:\Users\kedro-project\venv\Lib\site-packages\click\core.py", line 1157, in call
return self.main(args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\kedro\framework\cli\cli.py", line 139, in main
super().main(
File "C:\Users\kedro-project\venv\Lib\site-packages\click\core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\click\core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\click\core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\click\core.py", line 1434, in invoke
return ctx.invoke(self.callback, ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\click\core.py", line 783, in invoke
return __callback(args, *kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\click\decorators.py", line 45, in new_func
return f(get_current_context().obj, args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\kedro_neptune__init__.py", line 177, in init
context = session.load_context()
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\kedro\framework\session\session.py", line 284, in load_context
self._hook_manager.hook.after_context_created(context=context)
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_hooks.py", line 433, in call
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_manager.py", line 112, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_manager.py", line 418, in traced_hookexec
return outcome.get_result()
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_result.py", line 108, in get_result
raise exc.with_traceback(exc.traceback)
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_result.py", line 70, in from_call
result = func()
^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_manager.py", line 415, in
lambda: oldcall(hook_name, hook_impls, caller_kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_callers.py", line 116, in _multicall
raise exception.with_traceback(exception. traceback)
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_callers.py", line 80, in _multicall
res = hook_impl.function(*args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\kedro_telemetry\plugin.py", line 128, in after_context_created
catalog = context.catalog
^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\kedro\framework\context\context.py", line 226, in catalog
return self._get_catalog()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\kedro\framework\context\context.py", line 287, in _get_catalog
self._hook_manager.hook.after_catalog_created(
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_hooks.py", line 433, in call
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_manager.py", line 112, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_manager.py", line 418, in traced_hookexec
return outcome.get_result()
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_result.py", line 108, in get_result
raise exc.with_traceback(exc.traceback)
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_result.py", line 70, in from_call
result = func()
^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_manager.py", line 415, in
lambda: oldcall(hook_name, hook_impls, caller_kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_callers.py", line 116, in _multicall
raise exception.with_traceback(exception. traceback)
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_callers.py", line 80, in _multicall
res = hook_impl.function(*args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\kedro_neptune\init__.py", line 425, in after_catalog_created
config = get_neptune_config(settings)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\kedro_neptune\config.py", line 36, in get_neptune_config
credentials = config_loader["credentials_neptune"]