neuro-inc / neuro-flow

Execution engine for scripts and pipelines
https://neu-ro.gitbook.io/neuro-flow/
Other
16 stars 2 forks source link

Action inputs in short form does not work #667

Open YevheniiSemendiak opened 2 years ago

YevheniiSemendiak commented 2 years ago

https://neu-ro.gitbook.io/neuro-flow/reference/actions-syntax#inputs

Example: I have a batch (left-hand file), which uses an action, which has several inputs defined in a short form (right-hand side). image

traceback ``` neuro-flow --show-traceback bake stress ╭─────────────╮ │ │ │ Bake stress │ │ │ ╰─────────────╯ [12:21:45] neuro_sdk==21.11.0 neuro_cli==21.10.0 neuro-extras==21.9.3 neuro-flow==21.10.0 Use config file /Users/ysem/work/projects/neuro/mlops/mlops-stress-tests/.neuro/stress.yml ERROR: expected a mapping node, but found 'scalar' in "/Users/ysem/work/projects/neuro/mlops/mlops-stress-tests/third-action/neuro-action.yml", line 5, column 13 Traceback (most recent call last): File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_flow/cli/main.py", line 193, in main cli.main(args=args, standalone_mode=False) File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/click/decorators.py", line 38, in new_func return f(get_current_context().obj, *args, **kwargs) File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_flow/cli/utils.py", line 40, in wrapper return runner.run(callback(*args, **kwargs)) File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_cli/asyncio_utils.py", line 55, in run return self._loop.run_until_complete(main_task) File "/Users/ysem/miniconda3/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_flow/cli/batch.py", line 82, in bake await runner.bake( File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_flow/batch_runner.py", line 448, in bake bake, flow = await self._setup_bake(batch_name, params, name, tags) File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_flow/batch_runner.py", line 378, in _setup_bake flow = await RunningBatchFlow.create( File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_flow/context.py", line 2161, in create tasks = await TaskGraphBuilder( File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_flow/context.py", line 2805, in build tasks = await super().build() File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_flow/context.py", line 2666, in build action = await self._cl.fetch_action(action_name) File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_flow/config_loader.py", line 134, in fetch_action self.__action_cache[action_name] = await self._fetch_action(action_name) File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_flow/config_loader.py", line 130, in _fetch_action return parse_action_stream(stream) File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_flow/parser.py", line 1458, in parse_action_stream ret = loader.get_single_data() File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/yaml/constructor.py", line 51, in get_single_data return self.construct_document(node) File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/yaml/constructor.py", line 55, in construct_document data = self.construct_object(node) File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/yaml/constructor.py", line 100, in construct_object data = constructor(self, node) File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_flow/parser.py", line 1406, in parse_action_main ret = parse_dict( File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_flow/parser.py", line 316, in parse_dict value = ctor.construct_object(v) # type: ignore[no-untyped-call] File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/yaml/constructor.py", line 100, in construct_object data = constructor(self, node) File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_flow/parser.py", line 1143, in parse_action_inputs value = parse_action_input(ctor, v) File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_flow/parser.py", line 1118, in parse_action_input ret = parse_dict( File "/Users/ysem/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_flow/parser.py", line 292, in parse_dict raise ConstructorError( yaml.constructor.ConstructorError: expected a mapping node, but found 'scalar' in "/Users/ysem/work/projects/neuro/mlops/mlops-stress-tests/third-action/neuro-action.yml", line 5, column 13 ```

in 21.6.23 and 21.1.4 NF versions it also didn't work

asvetlov commented 2 years ago

Is short-form correct? Looks ugly to me

YevheniiSemendiak commented 2 years ago

Why not, you could just write the default value (or leave it empty) if this action is local.

YevheniiSemendiak commented 2 years ago

However, if it is complex to support such a thing, we could just get rid of it, IMO.

asvetlov commented 2 years ago

After offline discussion with the team we decided to support at least

inputs:
  inp_a: ~
  inp_b: ~

form with explicit None values for inputs. If we cannot distinguish explicit ~ from implicit None for absent value -- we'll support it too.