Use brickstrap.conf to persist options between invocations of brickstrap.

[dlech]

This conversation started in PR #23, so here's a copy...

@dlech:

I know we discussed earlier about saving things like which qemu in the rootfs, but looking over the commands here, I am seeing a need to keep this info outside of the rootfs.

First, we need to expand the usage description to show which options are valid with which commands. Here is what I propose

Usage: brickstrap <options> <command>
Creation:
brickstrap -p <project> -c <component> [ -c <component> ... ] [ -d <dir> ] [ -f ] { create-conf | create-rootfs | all }
Incremental build:
brickstrap -d <dir> [ -f ] { run-multistrap | copy-root | configure-packages | run-hooks | create-tar | create-image | create-report }
Debug:
brickstrap -d <dir> { simulate-multistrap | run-hook <hook> | shell [ <shell> ] }
Destruction:
brickstrap -d <dir> delete

The idea here is only have to specify the configuration options once using one of the "creation" commands. Since the most basic creation command is create-conf (which currently just creates multistrap.conf), we need to create a brickstrap.conf as well. It would just need to save the absolute path to the project and each component that was passed in. All other commands source brickstrap.conf (and fail if it does not exist).

Sanity checks are as follows...

• Creation commands:
  • Checks for valid project and component source directories
  • Checks that <dir> does not exist
  • If no -f option, fails with error.
  • if yes -f option, runs delete command before proceeding
• Incremental build
  • Checks that <dir> is sane, e.g. has brickstrap.conf
  • Checks that project and components in brickstrap.conf are still valid
  • Checks that output of command does not already exist.
  • if no -f option, fail with error
  • if yes -f option, delete previous output and then proceed
• Debug:
  • check that <dir> is sane. e.g. has multistrap.conf for simulate-multistrap or has rootfs for run-hook/shell
• Delete:
  • check that <dir> exists

[dlech]

@cmacq2:

Let's leave these kind of behaviour changes out of the scope of this PR. There's a lot of scope for bugs and unintended consequences:

• what are the semantics of the -f flag?
• what if you do want a new config?
• how do you know all settings are preserved and not just the ones which are hit by the code path of the current command?
• how do you avoid the 'history' simply growing and growing endlessly, leading to worse and worse start up time and needless waste of disk space?

Perhaps you could setup an issue for exploring this/tracking these changes?

[dlech]

what are the semantics of the -f flag?

Existing files and/or directories are not overwritten/replaced unless the -f flag is used.

what if you do want a new config?

You start a new directory (-d <dir>) with one of the "Creation:" commands.

how do you know all settings are preserved and not just the ones which are hit by the code path of the current command?

Only certain commands (the "Creation:" commands) save these settings and these commands can only be run once per -d <dir> (or run with the -f which wipes out the existing <dir> and starts from scratch.

how do you avoid the 'history' simply growing and growing endlessly, leading to worse and worse start up time and needless waste of disk space?

Not sure what this "history" is you are talking about. Files and directories are create only once or wiped out and replace with the -f option. Running some "Incremental build:" commands multiple times will result in undefined behavior. The "Incremental build:" commands should be advertised as "for advanced users who really know what they are doing" and not for normal use.

[cmacq2]

Not sure what this "history" is you are talking about.

I thought this discussion was in part a continuation of much the same line of thought in PR #21 about remembering QEMU binaries. See: https://github.com/ev3dev/brickstrap/pull/21#issuecomment-170293944

[dlech]

It is. The -Q option would only be allowable once at the configuration of the brickstrap.conf file. After that, the value in brickstrap.conf would be used.

I'm also thinking that after we implement #26, we could actually store all of the cated source files and directories in the generated <dir>. This way, everything is parsed and generated only once. You can manually tweak files and such. But after the initial "creation" command, brickstrap only pulls information from <dir>.

[cmacq2]

You start a new directory (-d <dir>) with one of the "Creation:" commands.

I'm not convinced by this. It makes testing things out and experimenting with brickstrap builds harder, because you cannot easily go back and forth without discarding build directories... The pathological example would be a misconfigured multistrap.conf resulting from the interaction between mutually incompatible component sets...

But after the initial "creation" command, brickstrap only pulls information from <dir>.

I'd prefer to be able to 'update' the configuration set by the creation commands somehow. Either implicitly because we detect new settings, or explicitly because I pass some kind of option (e.g. -f).

[dlech]

Since most steps in brickstrap can only be run once, I find myself starting from scratch quite frequently to ensure that I have valid results. My workflow when something doesn't work out the way I want tends to be manually editing things via brickstrap shell until I get the results I'm looking for and then reproducing it in the config files. Then, like I said, I always run brickstrap all again to start from scratch to verify my changes. Updating the configuration on an existing <dir> would not be useful to me because by the time you can only run each incremental build command once. Copying rootfs files may overwrite files that were modified by package install scripts or hooks, running hooks more than once is already know to cause problems (#7) and so on. I also find it useful to do a directory diff on a <dir> that I have manually changed vs. one that was automatically generated to ensure that I made the correct changes to the config files to ensure the same results as my manual editing.

[cmacq2]

Makes sense.