riga
commented
2 years ago Hey @pfackeldey ,
turns out that --some-parameter.foo bar --some-parameter.baz test → self.some_parameter = {"foo": "bar", "baz": "test"} is not that simple to implement in law with an intuitive interface. Three things come to mind:
-
For this to be really beneficial, one would need proper type casting so that e.g.
--some-parameter.a-flagwould directly translate to a bool,--some-parameter.some-int 1to an integer, etc. This requires to define the expected sub fields before hand, and for that one could simply introduce a set of namespace-like parameters on the user side, e.g.--nn-layers INT,--nn-activation STR, and then gather them manually in a dictionary in the task init. I think that without this type conversion, this feature would feel quite incomplete. -
Without this a-priori definition of subfields and type conversion, one would rather implement this in luigi itself, in particular at the interface with argparse. In there, after argparse handled all known parameters, one could manually treat all unknown parameters, in that one should check if some of them have a common prefix,
some-parameterabove, and then convert them into a dict (without type conversion). I'm not sure whether this could be done in a clean way. -
Internally in luigi, the cli parameter format
--A.B-C Dis reserved for setting parameterCof task familyA.BtoD. In the example above, I suppose--some-parameter.some-intwould be interpreted as task namespacesome_parameter, task classsomeand parameterint. This is another thing to check, but I suspect this is nothing one wants to change (and also the auto-completion in law would not be happy about that ;) ).
Did you see https://luigi.readthedocs.io/en/stable/api/luigi.parameter.html#luigi.parameter.DictParameter?
Dear @riga,
as discussed offline a
ConfigParameter, similar to this one would be a great addition tolaw, especially for e.g. Deep Learning applications.Best, Peter