new CLI structure

[deep-diver]

@sayakpaul

Currently, as we add more steps, the number of argument of CLI grows so big. I think this is not a good thing in the DX/UX perspective. Hence, I want to discuss about making changes in the following manner:

provide separate CLI script for each step.

• for instance, run python batch_inference.py --options for batch inference, run python eval.py --options for evaluation, and etc.,

then provide a entry_point.py that automatically selects which script to run based on --step option. For instance, when --step is setp to eval, all other options will be passed down to the eval.py script. But this is just a little enhancement for the users, so this is optional. we can choose.

[sayakpaul]

We could configure YAML config parsing and provide separate configurations for each step. I think that might be easier.

[sayakpaul]

And yes, providing separate CLI script for each step is definitely better and more robust.

[deep-diver]

We could configure YAML config parsing and provide separate configurations for each step. I think that might be easier.

This was something that I had in mind too. Like, we have separate config files for each step, then just feed them in for each step, right?

[sayakpaul]

Yup.