Open gtkramer opened 5 years ago
gtkramer
commented
5 years ago Similarly, I think it would negate the need for a separate tool for upstream vs downstream mixing since the differences are in how sources are defined. There is a good deal of functionality built into mixer for a use case that lets users start editing bundles with mixer, but I think this particular use case may be good for a separate tool to align better to the unix philosophy and simplify mixer's main code base a bit.
gtkramer
commented
5 years ago While the use cases where we'd actually need more than two sources for bundles and repos are probably limited, I think what we gain here is a more flexible interface for defining sources that is more straight-forward because it doesn't care about local vs remote resources, online vs offline resources, or upstream vs downstream users. There would just be logic to fetch the resources from wherever they're at and homogenize them into something that mixer can consume.
These distinctions add additional complexity to the mixer CLI and the underlying code. Given there is still some complexity in homogenizing sources, this may be something we want to require from the user's end so this complexity could be removed entirely. clr-distro-factory already does this with content.sh, which provides an interface to supplying input to mixer.
I guess the open question is, "Who is responsible for homogenizing inputs?" Mixer or DevOps? Mixer will know the format required for homogenization, and ideally this would be something that DevOps does not need to track as the format changes.
gtkramer
commented
5 years ago There is quite a bit of behavior built into mixer that checks whether it's being used correctly and sensibly. There's different checks for each of its commands. This is useful for users who are less knowledgeable about how mixer needs to work. However, I would expect this need to decrease with a CLI such as the proposed one above since users would go through less steps with fewer commands to set up mixer to build the next iteration of their mix.
For what makes sense to keep around, it may be useful to consolidate this behavior into a command that is optional for the user to run, something like mixer validate. This would help to narrow the focus other commands and keep their functions consistent with what the user expects without any additional behavior.
gtkramer
commented
5 years ago A mixer validate command may be confusing with mixer build validate. It might make sense to have mixer validate workspace and mixer validate update commands so validation operations are tied under a common subcommand of validate. This would also make sense for MCA because mixer build validate does not actually build content for consumption by the client.
gtkramer
commented
4 years ago Ideally, the mixer validate commands are optional since they are not a required part of the process. They would only give assurance to users that things are looking sensible for starting to mix and then after the mix finishes. If a user knows what they are doing, they should be able to confidently bypass these to get to producing and consuming updates faster. However, these are commands that we would want to run in any production environment.
gtkramer
commented
4 years ago The mixer validate commands would be run either just before or after the mixer build commands. Then, mixer does not attempt to perform validation as users are setting up their workspace.
jwakre
commented
4 years ago mixer repo sources 31290 file:///home/releases/www/31290/clear/x86_64/os mixer repo sources 31290,https://clearnas.ostc.intel.com file:///home/releases/www/31290/clear/x86_64/os mixer repo sources file:///home/releases/www/31290/clear/x86_64/os mixer repo sources /home/workspaces/clear/packages/koji/results mixer repo sources /home/workspaces/clear/repo mixer repo sources <location>... <location> = build number,mirror host <location> = curlable path <location> = local file path to rpms outside mixer workspace (with or without repodata)Creates any number of repo sources for mixer. Sources are layered from left to right, with sources having highest priority on the left and the lowest priority on the right (priority in the dnf sense). If a path is given that is not a valid repo, mixer copies the rpms into it's workspace and creates a repo automatically. Here, users would not know about local-rpm or local-yum folder. Using this command again wipes out any existing repo configurations in mixer's workspace. When it is run, mixer recreates the .yum-mix.conf again and its normalized repo workspace again. Required to be run every time prior to starting a mix. All sources must always be defined.
I agree that the CLI should be improved to make it easier to manage an upstream mix and that all repositories should be treated equally (no special treatment for upstream/local). Although, I disagree with repeatedly requiring all repositories to be set on the command line because it adds complexity that most downstream users would find challenging. Alternatively, I think it would be best to initialize the mix with separate default configurations for upstream and downstream mixes. For instance, when initializing an upstream mix, upstream related settings would not be configured and when initializing a downstream mix, it would be configured to work with an upstream. Mixer should allow users that desire more advanced configurations to override these settings, but basic use cases should be as simple as possible.
I agree that it is a good idea to use a more generic approach (no special treatment for upstream/local) for bundle definition files.
Since most downstream users will want to use the upstream version to synchronize the repo/bundle, I think there is value to support variables. This will prevent users from re-configuring their sources.
Additionally, with these types of changes, we need to be careful that upstream format bumps will be crossed correctly by downstream mixes.
Using the mixer CLI as an upstream is somewhat difficult due to the assumption built into mixer that the user is a downstream. This difficulty is mostly encountered when trying to define the sources (bundles and repos) for mixer. The commands that are used as an upstream work around the downstream assumptions and make the code that DevOps writes more difficult to understand because it works around unexpected behavior. What's outlined below is just a brainstorm of what might help to solve this problem.
Sets up a workspace by creating a directory skeleton, creating mixer.state with a default format, version, and last version, and creating a default builder.conf. Only does the work that is missing, and the values of existing files are not modified. If all of this already exists, then this command is a noop. This command may, however, perform a conversion of mixer.state and builder.conf to a different format if necessary. If other changes in the workspace structure are required to be converted, this would be when those changes are made. This command would always be run and would not take any arguments. No other work is done with this command.
Next logical step in the process. Same as what we have today.
Uses any of the values supplied from the command line to modify mixer.state. These commands would be run as required by automation. No other work is done with this command. Mimics what is done with the
mixer config setsubcommand. Format would be automatically modified during automatic format bumps. Last version would be automatically modified on successful creation of an update. Version is not automatically modified except by the user at the mixer CLI with this command. Any time mixer.state receives an additional key-value pair, it needs to be exposed for modification through the CLI to provide an interface for automation.Creates any number of repo sources for mixer. Sources are layered from left to right, with sources having highest priority on the left and the lowest priority on the right (priority in the dnf sense). If a path is given that is not a valid repo, mixer copies the rpms into it's workspace and creates a repo automatically. Here, users would not know about local-rpm or local-yum folder. Using this command again wipes out any existing repo configurations in mixer's workspace. When it is run, mixer recreates the .yum-mix.conf again and its normalized repo workspace again. Required to be run every time prior to starting a mix. All sources must always be defined.
Creates any number of bundle sources for mixer. Sources are layered from left to right, where subsequent sources are overlayed on top of the previous ones. A downstream would do
mixer bundle sources 31290 /home/clrbuilder/projects/clr-bundles/bundleswhereas an upstream would domixer bundle sources /home/clrbuilder/projects/clr-bundles/bundles. Here, users would not know about local-bundles folder or the mixbundles file. Using this command wipes out mixbundles file and any internal tracking of the existing sources. When it is run, mixer builds up the bundle sources again. Required to be run every time prior to starting a mix. All sources must always be defined. When running, if mixer sees that existing sources have already been cloned, it has the option of keeping them in tact. If users want to modify bundles, it needs done outside the mixer workspace and then they are brought in with this command.Pretty much the same as what we have today, except adding bundles no longer has the downstream vs upstream orientation and just asks the user, "What bundles do you want for this update now that you have defined your current sources?"
Something like this above would be all an OSV would need to do in order to get going with mixer, a possible BKM of sorts. There are a few goals that would be nice to achieve with a perhaps similar mixer CLI:
I'm sure there are some more fine-grain details that I've overlooked given how the stateful nature of mixer makes this challenging, but I wanted to put the idea forward to see what other folks thought, and see if it might lead to a better mixer CLI experience for upstreams.