Translating hardware requirements into provisioner-specific terms

[happz]

Since #330 si now merged - thanks all involved! - I guess it's time to begin working on a way how to translate this generic description into terms understandable by involved provisioning code, i.e. provisioning modules like vagrant or podman, and even for those not supported by tmt yet, like those dealing with AWS or Beaker.

I'd like to kick off some discussion on this topic, with a few starting points. It seems to me the idea is quite trivial, but it will get worse, I'm sure of that :) Please, share your comments and ideas.

Disclaimer: I'm involved with https://gitlab.com/testing-farm/artemis, which focuses on provisioning, and we'd like to use this specification there as well. I find it much easier to have just one language, shared by these tools, and it apparently supports provisioning environments tmt does not (correct me if I'm wrong). Hence my mentions of OpenStack and Beaker - tmt maybe doesn't have a dedicated module for those, yet, but Artemis does.

• on the input side, we have a generic description, then there's a battery of "translators", or "solvers", one for each supported provisioning method, and on the output, we have something understandable to provisioning modules and their respective infrastructure. In a crude way, tmt's vagrant would call hw-specs-to-vagrant translator, and get a vagrant file snippet; Artemis' OpenStack provisioning code would call hw-specs-to-openstack translator, and get an OpenStack flavor.
• some "translators" will need more inputs than this. For example, to map memory: '> 8 GB', OpenStack "translator" would need a list of available flavors and their parameters, so it could pick the most appropriate one.
• some "translators" probably won't need more information. For example, Beaker can transform the input into an XML snippet understandable by Beaker without more information.

And that's it O:) I don't know how exactly tmt uses provisioning modules, but this piece puzzle fits somewhere to the place where provisioning knows the test's requirements, and it's time to find what parameters to write into a vagrant file or what flavor to request from OpenStack. It doesn't change anything else, and it doesn't prevent the addition of options that would override the result.

Some questions:

standalone package, or part of tmt?

Shall this battery of "translators" be available as a standalone Python package, or would being a part of tmt suit us better?

In this case, I don't have strong feelings toward any of the choices as long as it's possible to use it as a standalone library. With Artemis, installing the whole tmt with all its requirements is probably not a very temtping option. For example, we have no use for anything Vagrant-related or anything that deals with plans and metadata. All we need is a code that knows how to pick the best AWS flavor for a given HW specification, nothing more. If it would be possible to install just this part (or a bit more), it would be really nice.

shall "translators" just yield their output, without checking the actual feasibility of such a result?

E.g. should Beaker "translator" not check the output yields a non-empty set of machines?

I'd say yes here, "translators" should focus on a single task, and leave the execution to their partners. If it's not possible to find a machine matching the hardware description, it's not a problem of "translator", but something between the user and tmt to resolve, e.g. by a nice error report with advice on how to proceed, or dropping HW requirements, emitting a big warning sign and proceed with testing, using what's available.

shall "translators" talk to their respective clouds?

For example, shall OpenStack "translator" acquire the list of flavors on its own?

I'd say no here: declare these as inputs, with proper types and documentation, and let users (tmt provisioner module, Artemis, etc.) gather these inputs for the "translator." It's highly likely they already do know how to communicate with the provisioning infrastructure anyway.

@dataclass
class Flavor:
  name: str
  cores: int
  ram: int
  ...

def to_openstack(specs: Any, flavors: List[Flavor]) -> Flavor:
   ...

Well-defined API, types, no magic happening behind the curtain, no network that can fail, no need to pass tokens and usernames around, possibly even pure functions, and last, but not least: it's sooo much easier to mock a list of flavors than mocking an OpenStack API server to simulate right answers to "translator"'s queries.

who's going to write the code?

I'm the one pushing this agenda, I would really like to see it happen as I strongly believe it's needed. I don't mind writing a bunch of patches if there's a chance to get them accepted (if this would be part of tmt package family), or starting a separate project (although it would be better if it'd b still somehow connected to tmt). Any help would be appreciated, though.

[pvalena]

From Vagrants point of view- the clouds, VMs, containers (etc.) are all called providers. Vagrant can as well talk to many providers (incl. openstack, aws). Vagrant is kind of a "translation" + "relay" layer on its own. There's no server or runtime that actually monitors/handles the VM status, or interferes in any way, apart from user-driven actions which translate to specific provider commands (libvirt/virtualbox/docker/ssh/...).

WRT options- some of the options are common, but generally the options are up to the providers plugins themselves. F.e. vagrant can run docker/podman containers, which AFAIR ignore most of the parameters. They can be also specified in various ways- like for libvirt vs virtualbox. I expect to deal with those differences in tmt's Vagrant plugin, and unless otherwise required(to keep some uniformity on TMT's side) I'll simply relay/map everything from tmt's .fmf config to Vagrantfile. I do even intend to support giving custom Vagrantfile to tmt (which can have any plugin/scripting/config).

---

Please also note that all tmt plugins are packaged in subpackges, so you're installing only what's necessary for your use (or none, f.e.).

[thrix]

@happz thanks for kicking this off

I have nothing to add to the intro, translators are something I always imagined this.

standalone package, or part of tmt?

I would opt for having it as a separate library, which does one thing and does it well. No need to spoil tmt with it. Seems there are already multiple clients who want to use it.

shall "translators" just yield their output, without checking the actual feasibility of such a result?

I agree with yes here, I would delegate the feasibility decision to the client.

shall "translators" talk to their respective clouds?

Thinking about it for a bit of time, no sounds reasonable here, I could not first image how it will work with Beaker, but I realized the library response in that case can just be an host xml specification for Beaker. An additional input could be specification of pools the user has access to for example, what is easy. User would then talk to Beaker himself.

who's going to write the code?

I am happy to contribute also with my knowledge of some of the clouds. I have another question, where to host such libraries? We have several groups already around on Github/Gitlab, maybe:

https://github.com/RedHatQE

Would be the best ...

[thrix]

@psss @lukaszachy @wizardofoz123 could you review this proposal, we are planning to move forward with the plan laid out by @happz

[wizardofoz123]

Disclaimer:

• I work on a project called UPT https://gitlab.com/cki-project/upt , which is trying to unify provisioning (Beaker, AWS, ...). You feed the project either a yaml file with a specified format OR s Beaker XML and UPT will do its best to monitor/provision machine until the provisioning is successfully satisfied. Afterwards, you can run tests on the machine(s) using restraint wrapper called "restraint runner". There are a couple of advanced features to rerun tests, provide partial test results, dump data in specific formats, etc. This is all meant for CKI kernel testing.

The piece that you're discussing here could possibly be integrated into UPT, to allow specifying hardware requirements in a uniform way. In some ways, we are interested in abstracting away the provisioners and only decide what type of hardware we need.

In my opinion, the value of what you do here isn't in 100% perfect specification declaration, but in having other teams using it, so it gets enough traction and attention. If we miss out window to cooperate and integrate that hardware specification, then the value of any specification is drastically diminished. We can see that with Beaker; there are tons of workflows and plugins to use it, but no "one fits it all" way.

standalone package, or part of tmt?

100% agree on "as long as it's possible to use it as a standalone library". I wouldn't be very thrilled about pulling-in tmt as a dependency, given the fact tmt might as well pull-in UPT as a dependency to run tests using restraint.

shall "translators" just yield their output, without checking the actual feasibility of such a result?

In our use-case, we have some basic requirements that we should be able to ALWAYS satisfy in Beaker. If the translation fails, it means that something has gone horribly wrong and it must be investigated. I'd expect the translator to fail (e.g. throw an exception) that translation isn't possible at all. Bugs in translation must not be silently ignored.

shall "translators" talk to their respective clouds?

Agreed, the translators shouldn't talk to their respective clouds. However, they have to be up-to-date.

[psss]

Thanks for outlining the translation concept, @happz. The overall design you propose looks fine to me.

standalone package, or part of tmt?

Agree with all others that "possible to use it as a standalone library" is the crucial part here. Both options are fine but if we expect the hardware specification to be part of L2 metadata I'd say it would be natural to have the translation implementation directly in tmt.

The code for individual provision methods could be included in respective packages (for example translator for the virtual provision method would go with tmt-provision-virtual package) so you would only install those tmt subpackages you need. Or, the translation logic could be included in the base tmt package which has minimal dependencies.

shall "translators" just yield their output, without checking the actual feasibility of such a result?

Yes, I think translators should be simple, so no checking for feasibility. But they should catch and report any hardware specification errors.

shall "translators" talk to their respective clouds?

I like the approach proposed by @happz that such information should provided as the input to the translation function.

who's going to write the code?

I'm willing to help with this as well.

[psss]

I work on a project called UPT https://gitlab.com/cki-project/upt , which is trying to unify provisioning (Beaker, AWS, ...). You feed the project either a yaml file with a specified format OR s Beaker XML and UPT will do its best to monitor/provision machine until the provisioning is successfully satisfied.

Do you have examples of such yaml files? I briefly look at the README docs and saw only beaker-like recipe templates.

In my opinion, the value of what you do here isn't in 100% perfect specification declaration, but in having other teams using it, so it gets enough traction and attention.

This is good point. It would be definitely good to get enough feedback on this.

[thrix]

Where to look for more feedback, would it make sense to send this somewhere on some internal qe list or tech list?

[happz]

shall "translators" just yield their output, without checking the actual feasibility of such a result?

In our use-case, we have some basic requirements that we should be able to ALWAYS satisfy in Beaker. If the translation fails, it means that something has gone horribly wrong and it must be investigated. I'd expect the translator to fail (e.g. throw an exception) that translation isn't possible at all. Bugs in translation must not be silently ignored.

shall "translators" just yield their output, without checking the actual feasibility of such a result?

Yes, I think translators should be simple, so no checking for feasibility. But they should catch and report any hardware specification errors.

To make it clear, by "checking the actual feasibility" I meant this: for example, in the case of Beaker, it might be possible to transform HW specification into Beaker host-filter XML - we took some inspiration in Beaker's XML - without checking whether such an XML filter matches any machine at all.

I believe checking whether the requirement can be satisfied is out of the scope of a translator, its job is simply to transcribe generic specification to provisioner-specific "language" (flavor, host-filter XML, etc.).

Raising an error when the translation is not possible, e.g. when no flavor satisfies given HW specification or HW specification contains contradicting statements or is simply broken - absolutely, this should happen.

In other words: for a given HW specification (plus other input, like a list of OpenStack flavors), the translator would yield the corresponding provisioner-specific representation of the HW specification. It would NOT check whether it's possible to execute it. The output should be syntactically valid, but not necessarily match an actual machine provisioner has to its disposal. For example, a test may require "at least 64 GB of RAM", the translator would yield corresponding host-filter XML for Beaker, but it would NOT test whether this XML snippet matches any machine of the Beaker pool. That would be the translator's caller responsibility since it has access to more context.

@wizardofoz123 does this sound aligned with your view, are we on the same page?

[happz]

The code for individual provision methods could be included in respective packages (for example translator for the virtual provision method would go with tmt-provision-virtual package) so you would only install those tmt subpackages you need. Or, the translation logic could be included in the base tmt package which has minimal dependencies.

@psss wouldn't e.g. tmt-provision-vagrant try to pull in a pile of stuff? Or how are the packages organized? For example, as an author of 3rd party provisioning service, I want just the translator for Beaker, I'm not interested in anything else tmt-provisioner-beaker contains, not even in its possible requirements, as I have my own implementation of "provision Beaker machine" workflow. But I like the idea of matching provisioner and translator in TMT, maybe tmt-translator-beaker, with Requires: tmt-translator-beaker in tmt-provisioner-beaker?

[wizardofoz123]

@psss The README file https://gitlab.com/cki-project/upt/-/blob/master/README.md explains that the Beaker-like format that describes what tasks to run is necessary for restraint, we can't get around that without changing restraint itself, which I don't think would fly with the maintainer either. There's an AWS example in the README as well.

Mind you, as I've suggested, the part of translating to generic hw specs to provisioners is exactly what I/we need to add to either UPT or our testing pipeline itself.

@happz We're on the same page on most.

What I'm a bit concerned about is the dependencies. As I've mentioned, ideally it should be a standalone library with as little dependencies as possible. I understand you're working on something for tmt mainly, but is that prefix / extra dependencies really necessary for something that should be as lightweight as this?

[happz]

@happz We're on the same page on most.

What I'm a bit concerned about is the dependencies. As I've mentioned, ideally it should be a standalone library with as little dependencies as possible. I understand you're working on something for tmt mainly, but is that prefix / extra dependencies really necessary for something that should be as lightweight as this?

No, the prefix nor extra dependencies are not needed and are unwanted in general. The connection to tmt here is the fact we need tmt to support HW requirements for tests, and it somehow makes sense to stay in touch when implementing this feature, maybe even including it tmt packages. It doesn't mean it should pull in a pile of the required packages, quite contrary: the lower the number of dependencies, the better. I am interested in the HW spec => Beaker XML translator, that means maybe Beautifulsoup to help with XML, but it definitely shouldn't force me to pull in beaker-client or requests.

I don't have a personal preference when it comes to actual packaging. It should be available as a standalone Python package, it could be part of tmt repository, it could be available (also) as tmt-foo-bar subpackages/rpms/debs/whatever, everything would be probably fine. But I believe we are on the same page here, the number of packages any tmt-hw-to-OpenStack translator package would pull in into my project should be as low as possible. I think it doesn't have to require anything even remotely related to OpenStack at all: it's my project's job to talk to my OpenStack cloud, gather necessary data, then use this translator to convert the HW specification into an OpenStack flavor. This separation of concerns I'd like to preserve: makes things easier to implement, test, and should keep the number of requirements low.

[wizardofoz123]

@happz Ok, thanks for clarification! I don't see any issues with other points.

[thrix]

I would like to state additional requirements I would like to opt for:

• Python-3.7+
• start the development without a large initial import
• typing support

In case we want this to be a library and not part of tmt.

[happz]

Agreed, it should definitely support recent Python versions, and typing annotations just make things harder to break. Even if it'd be part of tmt, I'd rather not avoid using annotations.

[thrix]

One more thing we discussed with @psss, the code should ideally also host also the specification, it makes little sense to split them.

Do we want to vote on the location or how will we proceed?

[psss]

Agreed that the package containing the translators should have minimum dependencies. In order to keep the spec close to the implementation and save unnecessary packaging work (fedora package review...) I'd vote for having it in tmt. It could be a separate subpackage, for example tmt-hardware, tmt-translate, tmt-hardware-translate or the code could be in the core tmt package which should have few dependencies anyway. Here's what we currently have:

> rpm -q --requires tmt
/usr/bin/python3
git-core
python3-tmt = 1.1-1.20201105145924304790.master.9.g4b8bae8.fc33
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(FileDigests) <= 4.6.0-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(PayloadIsZstd) <= 5.4.18-1
sshpass

So there would be just two extra deps here: git-core and sshpass. But separate subpackage for translators is fine with me as well. What do you think would be better?

[thrix]

If possible, let's not even depend on git-core and sshpass the libraries do not need them

[psss]

Ok, I see, let's go with a separate subpackage then.

[thrix]

@happz is planning to kick this off, spoke about it today on our sync

[thrix]

Update: the work on implementation did not start yet. Recently we had interesting discussions with @abraverm to discuss the possibility to use [[cuelang.org]] (used by his OpenEngine project) to define the data transformation instead of the Python code. This approach could give us more flexibility later on and easier way how to define the requirements transformation to different clouds. Alexander is working on few POCs to show the feasibility of this solution. We should report back in 2 weeks +-.