teemtee / tmt

Test Management Tool
MIT License
85 stars 129 forks source link

All `tmt try` to accept generic steps options from `tmt run` #2951

Open zdohnal opened 6 months ago

zdohnal commented 6 months ago

This new method (f.e. libvirt/system) would work around passing -c system to tmt try, which is not currently possible, so tmt try cannot create image via local libvirt.

The -c system option gives user a machine with accessible ports with your local virtual machine manager, which is useful for reproducing issues with network services.

lukaszachy commented 6 months ago

Isn't it better to teach tmt try to accept provision options?

zdohnal commented 6 months ago

I would say it would complicated things unnecessarily, which (simplicity) was (IMHO) the motivation behind tmt try - the new method could be used on other places than in tmt try.

But if there is a way how to make try accept provision options and keep simplicity, it would be great :) .

Or, since I might be an odd user, make try accept provision options in case someone needs it, but let the default behavior to be simple - I can accept that I'm an odd user and I will add additional options to the cmdline, if the implementation does not provide an easy way how to get a new provision method... :D

psss commented 6 months ago

I think, it would be ideal to find a command line option syntax which would allow adjusting individual steps in general. There are several use cases which would/could benefit from this:

Brainstorming how it could look like:

tmt try fedora@virtual provision --how virtual --connection system
tmt try rhel-9@minute prepare --insert --how install --package local.rpm

It would be nice to support the full feature set of updating, inserting or dropping step phases. From time to time a specific use case might appear when this would be useful and I'd say users would welcome this even if the command line would be a bit longer.

On the other hand, for the frequently used options I think it would be better to provide some short variant to make the experience more comfortable. Perhaps something like this?

tmt try --list-images
tmt try @minute --list-images
tmt try rhel*@minute --list-images

Or, for installing a local rpm package, something like this?

tmt try --package local.rpm

@falconizmi, these use cases were shared on yesterday's workshop. Might be good to include them in the list of prioritized features for tmt try as well.

zdohnal commented 6 months ago

Brainstorming how it could look like:

tmt try fedora@virtual provision --how virtual --connection system
tmt try rhel-9@minute prepare --insert --how install --package local.rpm

I would recommend to omit provision --how <method> - it is defined by @<method>, so user can be freed of writing the whole command - like tmt can insert those command+options internally, but no need for user to specify them (even when they want --connection system/session).

On the other hand, for the frequently used options I think it would be better to provide some short variant to make the experience more comfortable. Perhaps something like this?

tmt try --list-images
tmt try @minute --list-images
tmt try rhel*@minute --list-images

Or, for installing a local rpm package, something like this?

tmt try --package local.rpm

LGTM!

psss commented 6 months ago

I would recommend to omit provision --how <method> - it is defined by @<method>, so user can be freed of writing the whole command - like tmt can insert those command+options internally, but no need for user to specify them (even when they want --connection system/session).

Yeah, makes sense to avoid any unnecessary duplication wherever possible. One approach would be to completely transfer additional step options as if they were given to tmt run. Parsing could work like this:

The last part would mean some duplication would probably be still present (as with the provision --how virtual example above), but perhaps, we could get rid of that duplication as well. @happz, what do you think from the click perspective, I guess this should be doable, right?

happz commented 5 months ago

The last part would mean some duplication would probably be still present (as with the provision --how virtual example above), but perhaps, we could get rid of that duplication as well. @happz, what do you think from the click perspective, I guess this should be doable, right?

Maybe we could hook these step subcommand to try in the way it's done for run? With that, Click should be able to process try arguments, but also any followup step-specific commands as well.

I'm not sure how exactly you plan to use the collected options after that: with normal command lines, Click & tmt collect the "CLI invocations" in lists, attached to step classes, and then there's _apply_cli_invocations() which applies them and modifies the list of raw phases.

I guess it all depends on how tmt try should behave, but I think the majority of this code should be re-usable, e.g. _apply_cli_invocations() takes a list of raw phase data, and returns another list - feed it with what tmt try wishes to run, let it apply CLI invocations, and let try follow what you get back instead of what's been read from fmf? The question is, this being the normal flow of options, what of this behavior will you not want for tmt try? :) I suppose we will learn that later after we try it.

All those custom options, making common use cases simpler, can be transformed into raw step data, or applied to raw step data tmt try gets from fmf, then modified by whatever user decides to express with more complex options, i.e. the actual steps, plugins, and options we discuss. I think this could work - simplified to make things nice and easy, accept but not force the use of everything tmt run * allows?