search

MightyPirates / OpenComputers

Home of the OpenComputers mod for Minecraft.
https://oc.cil.li
Other
1.59k stars 431 forks source link

Built-in persistent memory #172

Closed fnuecke closed 10 years ago

fnuecke commented 10 years ago

All right. This is a long one. Let's get this over with.

One of the bigger hurdles when getting started with OC right now is that computers don't have any "persistent" built-in storage. There's /tmp, but that vanishes after a reboot. / is read only, which confuses people. I'm against computers having an overall persistent storage though.

@SpiritedDusty suggested having an inbetween - call it NVRAM and only have it persist while the computer sits there. Once the player breaks it, the memory is lost. It would persist across reboots or being powered down, though, which is a compromise I could live with.

The big advantage of some kind of guaranteed persistent memory, besides being easier to use / get started, would be that the OS an rely on that, and store state information that is currently lost when rebooting. For example, whether autorun is enabled or not, mount points and symlinks.

Now this poses the question exactly how that NVRAM would be made accessible. Having it as a sub mount (/etc, /home, ...) would have the same issues /tmp has now. It is not obvious. People won't find it easily. Few people would use it. This is undesirable. So the alternative is to mount it at root level. But then what - have the ROM and NVRAM be a 'hybrid' file system, where new files can be written but what's in ROM cannot be edited removed? What if files are added to the ROM ([world]/opencomputers/rom/Lua) that already exist in NVRAM?

So taking the next step, what about the NVRAM 'replacing' the ROM? Make i so users can completely wipe the built-in OS and roll their own. This is what I'd like it to be. What I'm not decided on yet, is how to get the built-in OpenOS onto computers in the first place, and how it would update.

While number two would be really cool, IMHO, to avoid putting the bar too high, I'm leaning towards number one with an addition: have a button in the GUI: 'Reset NVRAM'. It'll bascially do the same thing as breaking and re-placing the computer.

What will probably die if this gets implemented is /tmp. It won't really have a raison d'être any more, since it can be fully emulated using NVRAM and the OS deleting anything in that folder on boot.

Now with all this said, there is one major problem. Currently, I'm cheating. A few libraries in the /lib folder (file system related stuff) get loaded by the kernel directly, to a) speed up booting (a lot!) and b) make it so that these (big!) libs don't contribute to the overall memory consumption. 64KB would not be enough, otherwise. So here are two solutions I can think of: have a 'soft' memory limit in place while bootstrapping, so that manipulated filesystem librares cannot use arbitrary amounts of memory, or re-implement the file system stuff in Scala. I don't really want to do the latter, since that could break a lot of stuff, and persisting that is work that gets done for me when I keep it in Lua - which is the reason it is in Lua in the first place. A third possibility might be to move those file system libs out of the ROM and keep them magically out of scope, much like the kernel itself.

Comments, feedback and ideas are very much welcome!

rmellema commented 10 years ago

I think it might be a good idea to have something similar to the NVRAM, that gets filled with programs from the ROM if they are missing. So when you make a new computer, there is nothing in NVRAM, the ROM gets pasted over the NVRAM.

When you edit a file(a new one or from ROM), it gets saved to the NVRAM. So If you for example change edit by typing edit /bin/edit you are editing a copy of edit on the local machine, and your changes get saved to the NVRAM. When you reboot the computer, there is already a file called /bin/edit, so it won't get overwritten by the OS, and you can still use your changed edit. This will also solve the issue of files that are added to ROM after the computer gets created.

And I guess it would be better to move all the file system stuff out of the ROM, like the kernel. Or move them to the scala side (maybe the computer API?)

fnuecke commented 10 years ago

So basically: copy everything from ROM into NVRAM if it isn't there already? Might work, would be a little unintuitive, though I think (I can already see the reports "bugged, I deleted that file and now it's back"). Having the user manually update sounds like a safer method. A message when the computer can't find an init.lua script to explain should be enough to explain that, too, I think.

I'm afraid as much as I don't want to, I'll have to reimplement what's currently in filesystem.lua in Scala, yeah... oh well. As mentioned in another issue, when I have the time, which will be in roughly two weeks. Until then I won't have time to do anything but very small things.

GopherAtl commented 10 years ago

Just throwing this alternative out there, as I was thinking about the same issue yesterday.

instead of nvram, you could use a the hard drive. With no hard drive, they would be as they are now, nowhere but /tmp to save files and a reboot completely resets everything, with not even any settings that persist. Add a hard drive, though, and on bootup the bios program will recognize the hard drive. If the primary (1st slot) HD is blank, it will install the os from ROM to that hard drive instead of booting from ROM. Thereafter, if the primary disk has a "boot sector" the computer will detect it and boot from it directly, rather than booting from memory.

This would avoid what I see as a bit of "magic" to the nvram approach, and eliminate any issue deleted files reappearing. The main downside is that any security people implement into their os could be bypassed by moving the primary disk to a secondary slot, in this or another computer, and booting from another, clean hd in primary, or potentially even just booting from ROM and re-inserting the hd while it is running. This con could be mitigated somewhat if the primary hd slot were "locked," requiring you to craft a HD into/out of the slot (or possibly swap it in a right-click gui on the held case like server blades) rather than allowing it to be hot-swapped from the case gui, but rather than requiring breaking the computer to update the OS, it only requires breaking the computer to, well, break in to the computer, or to recover from a damaged/corrupted boot sector.

With or without the locking tweak, this would remove the privileged status of the built-in OS while keeping the convenience and user-friendliness of it.

fnuecke commented 10 years ago

There's one problem with that: you can't tell which file system is in which slot. If there are multiple "bootable" disks (say, they have an init.lua script) the one that is actually booted from would basically be random. NVRAM would always be NVRAM. However, it may, in fact, just be very tiny, too small to contain the actual OS, so the kernel would just save the address of the file system to boot from on there. Maybe.

As for the locking, computers can be owned, so that's that, but attaching an additional disk drive with a bootable medium could still be used to potentially "hack" a computer. I think that's fair, though. Use WorldGuard or some other permissions mod/plugin if you need to go further than that.

skyem123 commented 10 years ago

What about a ROM chip you have to craft? You could then make custom ones with an EEPROM programmer. The NVRAM could be used to store boot data. (eg: Boot disk order, kind of like a real computer.)

fnuecke commented 10 years ago

It's interesting, but in this context I don't see it being much different from a floppy disk, really, just different names and machinery for it.

I've been doing some initial testing with moving more stuff out of the kernel onto the ROM, and the slow-down when booting wasn't nearly as bad as I remembered it to be. The main problem here really is the RAM. For this to work I had to shift the scale quite a bit to 192, 256, 384, 512, 768, 1024. Which kind of annoys me because I like powers of two for obvious reasons. But it makes for a much cleaner system, and is a lot fairer to custom operating systems (since parts of OpenOS are no longer loaded by the kernel, thereby cheating in the memory department). What this means is that the kernel only would contain the "native" APIs (Lua standard stuff except os and io, computer, component and unicode). It then simply runs 'init.lua' on the boot device, which can be set using new API methods in the computer API (get/setBootAddress). This basically eliminates the need for the NVRAM, because nothing "special" happens before a completely user-definable script is run, which can lie on any kind of file system - floppy, hard disk, some other component's companion file system. As a fallback, if no usable init.lua is found on the configured boot device, it'll always go back to running from the ROM.

I think this is relatively solid, and gives a ton of flexibility (and essentially full control over what you do with the RAM - running a tiny init script just printing the free memory I confirmed that all of it was available, minus the bit for that script itself of course). And as mentioned, it avoids the nasty memory cheating (except for the kernel, which is still "free", but since that isn't user configurable that's OK I think).

So if no-one has any valid concerns with this design, I think I'll go ahead with this for 1.3. Any further feedback is of course always welcome, in particular if you can come of with scenarios that cannot be covered with this approach.

PS: OpenOS would then come with a simple 'install' script, that could copy the whole OS from ROM to some other file system, and set that file system as the boot file system. There all the files could then be edited.

rmellema commented 10 years ago

So if I understand correctly, the kernel loads the basics, and then it selects an init.lua to run? How does it select this script? Like a boot manager? If I understand your idea correctly, then I think that this is the way to go. The only problem I have with it is the magic ROM that appears out of nowhere.

fnuecke commented 10 years ago

That's the idea, yes. The script itself would just always have to be named init.lua, the device / filesystem it lives on is set via the mentioned new methods (the address of the boot fs would be saved "in the computer", super tiny single-purpose NVRAM, basically).

And yes, the ROM is somewhat magical. I'm torn here, honestly. Sure, it'd be nice to have OpenOS on a floppy and all that, make it craftable to avoid people making the computers in their world utterly unusable, but on the other hand that feels like unnecessarily complicating getting a computer to work. It would certainly need a proper setup wizard then (with a 'live boot' option, of course). I do tend to keeping it around, though, aside from it being easier to get started (and not invalidating all. the. tutorials. again), handling the 'ROM override' would become a bit more... fiddly.

skyem123 commented 10 years ago

What if there was the ability to make your own custom rom (but the defualt rom is still accessible) as it would allow someone to create their own os in rom. The equipment for making a custom rom would be very expensive but it could pay off with faster boot times. (I also like the fact that it is another thing that you can customize.) The NVRAM could then be accessed as a table with a limited size (so people can use it for storing settings but not mutch else).

GopherAtl commented 10 years ago

fnuecke, everything non-essential moving out of kernel sounds good, but re: the current setup, wasn't it said earlier that it is not predictable which disks will be found first? Meaning if you have multiple disks with multiple OSs, it'd be a coin toss which which one booted? I thought at a minimum we would need enough nvram to hold some equivalent of bios config settings, things like the addresses for which of each device is primary (which would control boot order, which monitor gets the terminal if there are more than one, etc)

As little as 1K of nvram would seem more than adequate for this as well as allowing some user space for user-created OSs to use for, well, whatever purposes.

fnuecke commented 10 years ago

skyem123, the boot times even from normal file systems aren't that bad (a few seconds, as opposed to "instant" like now). This did give me the idea for the robot-rework, though. One of the reasons to keep ROM are that robots would otherwise have to sacrifice their sole disk slot for the OS disk. Once the robot build process becomes more customizable, though, a floppy disk might be built into the robot as ROM. Making me tend towards the 'no ROM' side again...

Gopher42, there'd be a new API entry to allow getting and setting the address of the file system to boot from, making that deterministic. It falls back to the ROM if that address is invalid (no component or no init.lua). If the ROM were to go, it'd have to iterate all disks to find a working init.lua, and could then remember that using mentioned API methods. The primary boot fs would be user changeable, of course.

To recap, here are the current ideas and what I'm leaning towards:

Right now I'm actually leaning towards the latter, completely getting rid of the ROM, as breaking as that may be. It just feels so much... cleaner. I'm also preferring the simple "boot fs address" API entry over NVRAM, because... well, it's much easier to implement and a lot less error prone.

Techokami commented 10 years ago

Making an OS install disk shouldn't be TOO hard to make available. Why not do it like how RedPower2 did it and just craft a completely blank floppy disk with one redstone?

Alzurana commented 10 years ago

I have to agree with fnuecke. No ROM at all with a little NVRAM for BIOS settings (where to load from) is the best approach. It's actually a feature I was hoping for, since I'd like to make some modifications to the shell without copying it and building an autorun script which hacks the current shell to load my own on startup. Getting the FORTH boot disk on RedPower2 (Rest in piece ;-;) wasn't so hard, either. it's just one additional crafting step (among 50 others before you actually have your computer). ->Maybe make the computers fall back to boot from floppy if no OS is found. That would make it possible to auto run the setup or just execute OpenOS on the floppy itself. (there's your live system)

skyem123 commented 10 years ago

What about a boot system in a hardcoded ROM that:

  1. Does native startup tasks.
  2. Checks for custom ROM (If found it runs it with the hardcoded ROM files still availible, the custom ROM can either take over the boot process or just add a possible item to the boot order).
  3. Checks NVRAM for boot order.
  4. Checks for keypress to enter manual boot selection.
  5. Tries to boot items in that order (Exactly like a real PC). The NVRAM could be a table (computer.nvram?) and the boot order could be another table (computer.nvram.bootOrder?). One of the boot items could be an internal LUA prompt (Also explain that you need to make a boot disk to get OpenOS when it loads so people know what to do). Another boot option could be a way to change the boot order.

Also, Why not make hard drives and floppy disks slower?

rmellema commented 10 years ago

I Almost agree with skyem123, but I don't like the idea of a full blown lua prompt at boot. I would prefer a special boot prompt with commands like listDrives and bootFrom and maybe some more. Kinda like loader in FreeBSD.

fnuecke commented 10 years ago

Yes, if ROM were to go (and it probably will), I was thinking making the OpenOS disk craftable - something like Floppy + Microchip or so. And make it spawn in dungeons chests, for example.

The startup system skyem123 describes is pretty much what the kernel would be doing, although somewhat less... elaborate, maybe. I wasn't really planning on adding a boot selection menu... that's something that should go into init.lua scripts, IMHO. The kernel would just try to start from the "primary" boot device, and if that fails try to find another bootable device.

The reason I'm not too keen on adding a customizable ROM is that there is a sufficiently reasonable system in place already - floppy disks/hard drives - that would allow for just as much freedom. Adding a new item for a customizable ROM just feels redundant to me. The one area where I'd agree it would make sense is for robots, but allowing people to install a floppy as the ROM in the (planned) robot assembler (I really need to write up what I'm currently planning for the robot rework...) should be good enough, IMHO.

As for the NVRAM, making that a Lua table doesn't make a lot of sense to me, tbh, and would also be weird to implement in a 'space limited' fashion. It would either have to be a "block" of memory (i.e. a string of fixed length) or an actual file system. The first would be an utter pain to work with. And making that an actual file system has several disadvantages (implementation-wise ) and offers little over the more simplistic 'primary boot device address in API' approach, in particular when ROM is removed, where having a hard drive is much more encouraged than before, anyway. ) For one, it would also need external saving, like the kernel, to avoid MC derping when saving chunks with lots of computers in them. The other thing is that accessing the fs is a lot more annoying from the kernel, because there is no wrapper filesytem API at that point, just raw fs component access.

So. Unless someone can point out something (reasonable) that's not possible this way, this is what I'd like to do: kernel checks computer.getBootAddress, tries to run init.lua on that filesystem, if it fails try running init.lua on other file systems until success. Error with 'no bootable media found' if necessary. init.lua scripts could then be used for boot selections, simple OS initialization (as it would for OpenOS) or the actual OS itself, if it's very simple - just a Lua interpreter, or some other very specific program, for example. OpenOS would come with a simple "installer" to copy it from floppy to an actual hard disk. This way the options for customization are, I think, sufficient?

SpiritedDusty commented 10 years ago

I think that this implementation is great but how will computers in existing worlds get converted over? Will they all automatically get an OpenOS disk installed on update or will the user have to craft an OpenOS disk on update?

fnuecke commented 10 years ago

They'll crash and burn. It will be a breaking update in that respect.

SpiritedDusty commented 10 years ago

You should display this on the computers after updating to 1.3:

classic_windows_3_1_bsod_by_rhf

fnuecke commented 10 years ago

Heh. Well, depending on what programs they are currently running, computers may actually continue to work for a while - until the ROM has to be accessed. At that point either the running program will error or the computer crash altogether (resulting in a bluescreen, although it doesn't look quite like the Win95 one - some people already pointed out that the new OC bluescreens look to Windows-ish for the Unix-ishness of OpenOS :P).

skyem123 commented 10 years ago

How about a way to "flash" a ROM like you can do with a PC? That would allow people to customize their ROM and it can also solve the fact that the update will break programs by allowing people to do a ROM flash when they are ready to update. The ROM could stay when the computer is broken (use a hash to check if computers should stack?) and the NVRAM could be a fixed length string (4KB? 2KB?) and be wiped when the computer is broken.

fnuecke commented 10 years ago

The argument that this would allow computers to continue working after the update is invalid, I'm afraid, because the kernel will have changed, and that isn't part of the ROM / boot filesystem (and never ever will be because it contains the sandbox - it must not be user-editable).

What is/are the big arguments in favor of a built-in storage aside from making it easier to get started? Because making OpenOS clearly state it has to be installed onto a hard disk for persistence of some settings (autorun, mounts, symlinks), or even show a small boot menu when run from disk, would be sufficient I think. So if that's the only argument for it... I'd prefer to stick to less special cases = less room for error + less work ;-)

skyem123 commented 10 years ago

I just like the idea of being able to change a ROM for a computer ingame, It feels more complete to me. I could probably do something interesting with an editable ROM (Boot process I was talking about earlier)...

GopherAtl commented 10 years ago

unless I misunderstood soemthing, fnuecke just said a few comments back that rom is going away completely, so the only thing that will not be editable by users for custom OSs is the kernel, which will only implement the lua sandbox and a very primitive bootstrap to identify and launch an OS, and at least the sandbox cannot, in any case, be made editable in-game, regardless of the scheme involved. So what are you asking for exactly, skyem123?

skyem123 commented 10 years ago

The layer after the kernel, the bit that looks for the OS to load. The kernel will load the file from the ROM.

fnuecke commented 10 years ago

OK. But... why? What's the advantage over having it on a disk (that justifies the added complexity)?

skyem123 commented 10 years ago

Custom boot firmware. I guess that's not much of an advantage though... (Can you edit the ROM for an individual computer outside of the game? Just curious...)

fnuecke commented 10 years ago

Right. I don't see it as adding enough value to be worth the complexity it adds to keep it around, then :/ And no, right now ROM cannot be changed per computer - but it'll go away anyway, as Gopher correctly mentioned, so that point is kinda moot :-)