Open ankatiyar opened 7 months ago
Should this get moved to discussion or is it ready to be discussed? @ankatiyar
This hasn't been discussed yet, it's in the Inbox
https://github.com/kedro-org/kedro/commit/84b3e5c960ca4308cf95980c809b66db140045f7
This was the original PR that creates this regression. The PR was about removing custom syntax, there is a minor refactoring that use OmegaConf to remove the custom nest dict update function. This is the root cause of the bug.
Things to do IMO:
Another way to confirm this is to reproduce the error and run this in older version (maybe 0.18.14)
I am coming back to this as I currently cannot bump our code from 0.18.14 to 0.19.8. A full minimal example is at https://github.com/PetitLepton/kedro-runtime-params.
@PetitLepton thanks for bumping this, I think we have a clear solution here already, which is do not use Omegaconf in KedroContext
. I have commented that we should not do this , the simple solution here is to revert to a recursively dict update.
Description
This case came up in a user question (@PetitLepton) - https://linen-slack.kedro.org/t/16417334/hi-folks-i-encountered-an-unexpected-behavior-regarding-the-#8d5f69d7-25ce-409f-b009-0a6a50cd064c
Context
The user has a custom resolver in their config which converts a string to a
pandas.Timestamp
type -settings.py
:Third-party
import pandas
def parse_timezone_aware_datetime_string(datetime_string: str) -> pandas.Timestamp: timestamp = pandas.Timestamp(datetime_string).astimezone(tz=timezone.utc) return timestamp
Keyword arguments to pass to the
CONFIG_LOADER_CLASS
constructor.CONFIG_LOADER_ARGS = { "custom_resolvers": { "parse_timezone_aware_datetime_string": parse_timezone_aware_datetime_string, }, "base_env": "base", "default_run_env": "local", } }
This works fine when you do a
kedro run
without any run time parameters but fails with the following error when you do akedro run --params="the_test=1"
:This is because when there's run time params, there is an additional merge step - https://github.com/kedro-org/kedro/blob/80ad182a220d691c77efc05cc4e2b36dcecb6ac7/kedro/framework/context/context.py#L203-L205 This happens after the parameter with the custom resolver has already been resolved into the
pandas.Timestamp
type andomegaconf
apparently does not support non primitive types in the config.Possible Implementation
Context
Possible Alternatives
omegaconf
'sDictConfig
type instead of converting todict
as we do right now, where the resolution happens at access time. Related issue: https://github.com/kedro-org/kedro/issues/2973