Summary

I need to get some ideas onto "paper", and might as well do it in a location where other people can comment. This expands on Option 2 from #352 .

High-level idea: Interfaces should be polymorphic across different server implementations (#352), and the runscript should guide the entire process. Interfaces will expose a fixed functional prototype (in their .h file), but the stubs that implement the glue between components can differ. Additionally, in some cases, there will be no "server" component, and the stubs themselves will be library implementations of the interface. In this case, the interfaces effectively organize our equivalent of DLLs. It follows that many different libraries should be pushed from the lib/ directory into an interface implementation (except for those that are included into every component).

Some preliminaries:

"Runscripts" should be renamed to have higher relevance as "Composition Specifications" (or compspecs for short).
compspecs include all of the information that current runscripts do, but in a standard format (TOML) that can grow with far less pain. I still think that in the long-term this specification will be complicated enough that we want to make it an intermediate step in system specification, but we need a firm intermediate stage before getting to the highest-levels.
Interfaces will need additional specification information: notably, since some stubs might depend on additional interfaces, specific implementations need to be able to include and use other interfaces. Currently this is manual and hidden from the broader system (i.e. a hack).

Compilation and build life-cycle V2

The build system will be broken into the following stages (changes within this proposal are denoted with New):

Interface compilation

Compile any "prototype" interface code into shared libraries (interface/<ifname>/{*.c, *.h})
New: Compile the existing stubs, and any additional stub implementations that I'll call "variants" in non-stubs/ subdirectories.
New: When compiling stubs, add DEPENDENCIES support to the interface Makefile specifications, to support using other interfaces. At this point, simply -I the relevant prototype code and load the shared libraries, but don't follow through with using any specific interface stubs.

Component compilation

Including each interface's .h files to understand shared prototypes and structures
New: Compile with a shared .a library of all of the interface's helper functions (in interface/<ifname>/{*.h, *.c})
Compile and include all the normal include/ and lib/ files (as the current build system does).
New: Note that we have not compiled any stubs into the components at this point.

Composite Specification-driven, whole-system linking

At this phase, we know
1. the components to be included in the system,
2. the dependencies between them,
3. New: which of the variants to use to resolve the dependencies (which concretely is which versions of the stubs for each interface to use to hook two components together, where no variant specified means "the default variant"), and
4. New: initialization strings both to be sent to a specific component, and to another component from a specific component (i.e. "for this component, please send scheduling parameters to the @sched component's initialization")
New: Thus, we trigger a linking phase for each component with the client stubs of the specific variants specified for each of its dependencies, and for each component with its specified variant's server stubs.
New: At this point the __cos_ucap_* capabilities exist, as do the __cos_sstub_* and __cos_cstub_* symbols all exist in each component and there should be no unresolved symbols.
New: compile the component with the final linker script that adds in the ELF program headers to make loading fast and simple in the booter.
New: default values for which booter to use

Image Generation

At this point, we have a collection of components that are fully linked into ELF programs, and we know the address of the user-level capability structures, and the server and client stubs, and which components depend on which other components.

New: components are serialized into a total order by the dependencies to provide proper initialization ordering
New: a table is generated that points to start/end of each elf object
New: a table is generated which includes a list of all synchronous invocations including the stubs and user capabilities for each. Components are referenced by their offset into the previous table so that ID generation can be left to components.
New: the booter with all of this information inlined into it, is compiled into the kernel image, and it loads it using the elf program headers.

Limitations and Warts

As interface variants must be polymorphic to the overall prototype of the interface, the memory representation for data-structures of the interface must be shared between all variants, thus must be either concrete and fixed (i.e. one struct example {...}; version across all variants), or abstract (i.e. using only struct example; and pointers). This will cause troubles if we want an interface that abstracts across the capmgr and the cos_kernel_api as the latter wants a concrete representation that is allocated by the client.
The compspec must resolve all dependencies, including those created by inter-interface dependencies. In the longer term, the only way I can see to resolve this is with strong "composition errors" with suggestions for how to resolve this.
I'm avoiding going into the cases when a single component relies on two separate components for the same interface, or when a component that exports interface A also depends on a component exporting interface A. I believe this are somewhat trivial, so I'm leaving them till later.

Future

The main feature that will be added to the system in the future will be abstraction in compspecs. We want to have a compspec be able to export an interface that can be depended on by other compspecs. In other words, compspecs should be treated as a component to remove replication in "system-level" collections of components.
Each component should be able to have a file easily passed to it in its memory image. Add into the build system a simple means of doing so via specification of that file in the compspec.
We will need a new kind of interface that mimics another. This will allow components to interpose on calls to other components, filter, adding functionality, or redirecting (as in @phanikishoreg's current case of turning sync/async invocations into async/sync, respectively).

Conclusions

This infrastructure will allow a level of polymorphism for the interfaces with an integration of libraries that will allow us to avoid a lot of the cos_kernel_api vs capmgr mess that we have now. It will enable the composition specifications quite a bit of power to pull information out of the objects, and to add information in that will, long-term, enable us to pass information where we need it in a much simpler manner. The cost is in the development of the new booter, composition specification interpreter, and build system changes. This seems like a good trade-off. I believe I've addressed most of the annoying edge-cases, but if I've missed something, please let me know now!

@Others @phanikishoreg @ryuxin @hungry-foolish

To codify this, and make it a little bit more transparent, see the loader branch.

`mkimg`

[x] toml specification and parser
[x] elf file parsing and symbol acquisition
[x] elf acquisition of the relevant interface functions to enable them to be hooked up via synchronous invocations.
[x] logic to integrate the mkimg processing with the build system via make component (see below)
[x] generate a binary file including component and sinv capability information including the addresses of client and server stubs so that the sinvs can be built at runtime
[x] tar all objects (including the binary file) and link them in with the llbooter
[x] add link-time support to any component for linking in an external "data" object to ease getting data into the system
[x] initial arguments support for the booter
[ ] initial arguments support for any component
[ ] integrate initial arguments into component creation along with @ init args that go to a different component

FS hierarchy updates

[x] new mkimg directory to create the new image guided by the system specification
[x] new build directory to store the temporary component executables to populate the tar file, and hold the elf files to be interpreted
[ ] shell scripts to guide the system specification -> system execution process

Build system updates

The build system is being changed to do compilation and some linking at normal compile time (make), and to move the final linking stage (including the generation of the executable) to the mkimg phase. To understand why, see Option 2 in #352 for an overview of the highest-level goal of allowing different variants of interfaces. Also see #364 to understand how this will also be used to help guide the library compilation and linking.

[x] separate the compilation + linking phases of component construction from the linking done with dependencies
[x] create a new make directive (component) to enable a specific component to be linked, and pass in as make variables all of the characteristics of that that component (name, target, interface and library dependencies).
[x] separate the compilation and linking phases for component-specific data, from the linking with interface dependencies
[x] clean up a lot of the makefiles to remove redundancies and kludges
[x] update all addresses and functions that are needed in any of the cos_kernel_api functions to use the __cosrt_* prefix and update the build system to link them all in when the component is created
[ ] separate out the linking with library dependencies from some of the normal compilation process. This likely relies on #364.
[ ] Integrate library and interface dependencies on libraries and other interfaces into the linking process. Until this is done, interface code code cannot rely on interfaces other than their own, and library dependencies must explicitly be listed in both the components that include them, and the system specification
[x] update the link process to include a strict linker script to ensure components are placed at expected addresses

Runtime booter updates

[x] tar parsing support to get at the executables
[x] access and parse the binary file including component and capability/dependency information
[x] loading of executables into fresh memory and into new components
[x] create the capability image for each new executable
[x] properly link with sinvs different components using the binary file data
[x] update the booter code to use the tarball/binary file specification of the system and components
[ ] integrate scheduling of components into initialization

gwsystems / composite

System life-cycle driven by run-scripts #353