Remove the CLI from the kernel

[IanSeyler]

The Command Line Interface should be removed from the kernel binary. It should be treated as an application instead of a kernel component. This could be a good opportunity to re-write the CLI in C.

[CtrlC-Root]

I'd like to take a stab at this. Should the CLI only use the OS API or can it depend on newlib? It's obviously doable either way (ex: assembly version only uses OS API) but if we can use newlib that might cut down on code duplication.

[IanSeyler]

I have no issues with the CLI being written in C and using NewLib. It would make it easier to add features anyway. Thanks for taking a look!

Sent from my iPhone

On Nov 15, 2013, at 7:33 PM, Alexandru Barbur notifications@github.com wrote:

I'd like to take a stab at this. Should the CLI only use the OS API or can it depend on newlib? It's obviously doable either way (assembly version is written this way) but if we can use newlib that might cut down on code duplication.

— Reply to this email directly or view it on GitHub.

[CtrlC-Root]

So I've made some progress, you can see what I have so far in my master branch here: https://github.com/CtrlC-Root/BareMetal-OS/blob/master/programs/shell/shell.c. However, I think I may have run into a slight problem. The current CLI is part of the kernel but my shell program is already loaded at 0x200000 (I think this is correct). How am I supposed to load a different program into memory? I take it you can't simply pick a different location without fixing the program's instructions, right?

Also, while the current interface for managing (argc, argv) is acceptable, we may want to consider beefing it up or improving it a bit. It won't increase the code size by much but it will make starting other applications much easier.

[CtrlC-Root]

I've thought about this a bit and I can see the following options. 1) We compile the shell application with a different base address. We modify the CLI code to look for and launch the shell.app application if it's there. The shell can then load and run other applications at 0x200000. 2) We implement a loader in the kernel that can relocate applications. 3) The shell relocates programs as they are loaded into memory.

My assembly and knowledge of operating systems is a little lacking, so I may have misunderstood the problem or missed an obvious solution. In any case, this is your project, so I'll leave the decision up to you. Personally I think (2) is the best choice. I know this isn't a general purpose OS and it would take some effort to implement, but it would be pretty useful for more than just the shell. If it's easy to load and run applications (maybe an execvpe equivalent?) then larger problems could be split up into several different applications that run/call each other. Otherwise (1) seems the most reasonable. The OS will need some way to start a default application anyways. It may as well be the shell and we can load it somewhere other than 0x200000 while we're at it. Just my 2 cents.

[davidtsulaia]

If this system is to become a multiuser one at some point in history, then the CLI should be initiated by user log in and just malloced instead of statically loading into specific address. The "default app" mentioned above should be log in session manager of some sorts, with or without netwotking support. If the system will remain single user one, then I don't see any problem why should the CLI not be a part of the kernel and be what it is now.

P.S. CLI should be using only OS Kernel API and that's it. This is my opinion. Using sxternal library is extra secutity risk + big PIA at that. This not only my opinion, but my experience as well.

[CtrlC-Root]

I'm willing to agree on limiting it to the OS API on principle but compiling and using newlib is really not that difficult. I'm actually almost done setting up the waf build system to do it automatically (for me; might do a pull request down the line). Also, I would say that even if it remains a single user system (as it should, it's for computation after all) there are still benefits to moving the CLI into a separate shell application. 1) More room in the kernel for more important features. 2) Not everyone will want/need to use the shell in production (ex: me) nor will it always be running.

Still, none of this matters if we don't have the ability to load and run arbitrary programs.

[davidtsulaia]

If you ask me simplest solution for single user system CLI is to tie the shell program to keyboard interrupt callback. This way shell will not be running all the time. Right now size of kernel is not an issue, because it is extremely small and we live in the century of insane amounts of memory for really low cost and small sizes.

I have more purist approach than others, it might seem old fashioned, but it has it's own benefits. I like to know the concrete mix I use to build the house and writing OS is really quite like building a house :)

[IanSeyler]

I think it may be time to implement a foreground/background system. Let me think on this a bit more. I don't want to go multi-process or multi-user but it would be useful to have the user interface running while the app is.

[CtrlC-Root]

Well, whenever you reach a decision, I would be more than happy to help implement this issue (and probably more in the future). I'm using BareMetal OS as the basis of one of my projects and want to help make it better in any way I can.

[xieyuheng]

based on my language's interpreter, I can design an user interface for you, I can load a program as a FORTH function from BMFS, and call it from my interpreter.

[primoze]

Is anyone currently working on this?

I played around with the code and came up with a setup like this:

• instead of executing os_command_line here the code calls os_initproc, which loads a file named initproc from disk and jumps to that location (0x200000)
• app.ld now links binaries to 0x300000, added initproc.ld which links them to 0x200000
• initprocc.sh, which is basically appc.sh except it uses initproc.ld and produces a file named initproc - this way any app can be rebuilt as an initproc

Another idea was to have an initproc.confwhich would contain the program name to run on boot (not have it hardcoded), but using 2 megs of disk space for 1 word of text seemed excessive.

Does this seem OK, or is it too hacky?

[CtrlC-Root]

@primoze I haven't touched it since my last comment. The issue isn't that we can't move the CLI out of the kernel. I see I've deleted by branch but I had a solution for that as well. The actual issue, as best as I can remember it, was that there is no way to run more than one process at a time outside of the kernel. Yes, we could have some mechanism where when all processes are done we look for a file or a specially named application and run that. I feel like that's a bit of a hack but I wanted to get the author's opinion on it.

EDIT: And as you can see three comments above the author @IanSeyler said he would think on it.

[CtrlC-Root]

@primoze A related issue I just remembered is that many of the facilities used by the CLI such as argc and argv are hard-coded in such a way that make it difficult to use them with more than one process.

[Roxxik]

Before removing the current cli from the Kernel the functionality from it must be accessible by system calls and right now i don't see any way from an application to print the version number(without knowing the exact memory location where it is saved) or clear the whole screen(without just outputting a ton of spaces/returns)

I'd recommend to let the cli sit there for the moment and rewrite the Kernel API to make it possible to write a basic shell without hardcoding the location for the interesting informations(except for the systemcalls)

[ghost]

@CtrlC-Root "3) The shell relocates programs as they are loaded into memory." really right approach, also we're terminal cli filling into user space in e.g 0x3000 - 0x4000 around memory space.

[ghost]

I can see two ways of doing this.

One way is to compile the applications with a higher base address. The only clean way to load the application is to implement a memory allocation function that specifies an address to allocate the memory, similar to the first parameter in mmap in the POSIX specification.

The other way is to implement a function like the execve system call in Linux, which replaces the current process image with another. Although that would probably require more work, you would no longer need to compile the CLI or other applications with different base addresses.

[ohnx]

One other option is to use ELF and just load the application whereever ;)

[ghost]

@ohnx Using an elf header doesn't solve the problem.