ascpixi commented 1 month ago

Historically, Cosmos has had 2 major revisions: gen1, and gen2. I'd like to outline the need for a new major version, and potential major improvements that can be done with a rewrite.

I'm also looking for the feedback and approval of the maintainers (coredevs).

The need for a rewrite

In the past, Cosmos has been developed with actual, standalone kernels in mind - developers could use the toolkit to quickly develop domain-specific kernels. Nowadays, Cosmos is primarily used by people wanting to learn systems programming via a high-level language.

Cosmos, in its current state, is not viable for its original use-case - that is, domain specific kernels. The reasons for this are the following:

lack of stability. Kernels using Cosmos experience a high amount of unintended behavior, with the cause originating from Cosmos itself, and not the consuming projects.
lack of performance. The main reason for this is the IL2CPU compiler, which does not perform any kind of optimizations.
strict reliance on x86. Cosmos assumes that it is going to be compiled for x86 - not x86-64,nor ARM64. For domain specific kernels, architectures like ARM64 are dominant.

The current development efforts focus on the wrong thing - instead of improving the core design of Cosmos, or working on its stability issues, most of the manpower is dedicated to new features.

Issues with the current code-base

A rewrite, which would reuse select components of Cosmos, would take less time than to create several, small PRs to slowly bring the codebase to an acceptable level. I'd like to highlight some of the issues with its current state, namely:

Lack of thread safety

The entirety of Cosmos isn't designed for multi-threading in mind - no code has any kind of synchronization, and some methods even use hlt to wait:

https://github.com/CosmosOS/Cosmos/blob/e42447d958753edacc9313fc89aae6b7e82da981/source%2FCosmos.HAL2%2FPIT.cs#L220-L230

In order to make such parts multithreaded without the use of a "big kernel lock", some portions of the project would have to be redesigned.

Poor API design

The public API surface of Cosmos could be redesigned for more extensibility, as well as to correct some OOP quirks, such as the Device class:

https://github.com/CosmosOS/Cosmos/blob/e42447d958753edacc9313fc89aae6b7e82da981/source%2FCosmos.HAL2%2FDevice.cs#L6-L9

If the API would be largely changed, it would require all users of Cosmos to update their projects to use the new, updated API. Such breaking changes should be limited to major versions - in our case, gen 3.

Inconsistent code style and readability

There hasn't been any standardized code style for the repository, and as such, different portions of the codebase use different code styles. The most prominent example of this are some locals using system Hungarian notation, which does not benefit C# code - the types are already apparent, as opposed to e.g. C. Some other locals, however, use the .NET code style.

A large amount of code also has been designed with older C# versions in mind - resulting in the lack of use of fluent APIs and poor code quality. Principles like DRY and composition over inheritance are also often overlooked.

The public API documentation is also in a poor state - for example:

https://github.com/CosmosOS/Cosmos/blob/e42447d958753edacc9313fc89aae6b7e82da981/source%2FCosmos.HAL2%2FBlockDevice%2FPartition.cs#L71-L78

Such poor documentation is widespread throughout the repository.

Reliance on IL2CPU

Cosmos currently heavily relies on its compiler, IL2CPU. However, it is the largest cause of odd bugs, lack of debugging support, and performance problems. It also does not follow conventions set by other compilers.

Poor Linux support

Currently, Linux support is quite poor - while one can compile Cosmos kernels under Linux, debugging them is extremely hard. On Windows, Cosmos's Visual Studio debugger is meant to solve this problem, but currently, the debugger isn't fully functional - and works on only one IDE.

Megalithic

Cosmos has no modularization. This means that every kernel, no matter if it uses it, has e.g. the network stack. Additionally, executing binaries from inside the kernel is made quite hard.

No defined ABI

While Cosmos does follow cdecl as its calling convention, it does not follow any ABI. This makes interoperability with native libraries (e.g. zlib) especially difficult.

Potential improvements in Gen3

First, let's address how some of the problems outlined in the previous sections can be avoided in the rewrite.

Design with thread safety in mind

All code that could access potentially contended resources would need to make use of some sort of synchronization - be it atomic operations, locks, etc. This could require specialized algorithms for concurrent operations.

An IPL system could also be introduced. Interrupt controllers like the APIC already support such mechanisms out-of-the-box.

Modularized approach

Instead of providing code that might not be of interest of all users to all kernels, Cosmos could split these components to NuGet packages - e.g. the network stack could be seperated.

New ILC-based compilation pipeline

This change would replace IL2CPU with ILC - the compiler used for Native AOT compilation. The pipeline would integrate with the existing plug concept.

1. Transform standard library assemblies according to the plug set

The plug set refers to a collection of assemblies with classes and methods marked with attributes recognized by the plug weaver. The plug weaver is responsible for transforming one or more target assemblies by replacing the method bodies and adding/modifying fields according to one or more plug assemblies.

Private fields and methods can be accesses from plugs via the [Expose] attribute.

2. Compile main kernel code

ILC is used to compile the main kernel code to an intermediate object file. The consuming assembly is set as the main one, and the plugged standard libraries are provided (alongside any other references defined by the author).

3. Link

Plugs should also be able to specify that methods should be included as imports. With the compilation process, the user can also specify other object files to link alongside the main one.

[!NOTE] The ability to automatically compile assembly files to object files and link them could be implemented with an optional NuGet MSBuild extension, named e.g. Cosmos.AsmCompiler.

Plugs would not provide native code directly, like it's the case now - the default MSBuild rules would include the appropriate native object file depending on the architecture, and include the right plug set for the architecture.

Addendum A. ABI

Using ILC for compilation also makes Cosmos use a popular, solid ABI - the System V ABI. This means that interop with Unix libraries is greatly simplified.

Addendum B. Compiling on non-Linux hosts

The ILC compiler emits code with a different binary format, ABI, and dependent symbols, depending on the RID. From my experience and past research, linux-(arch) is the most documented, extensible, and solid compilation target. However, as ILC doesn't support cross-compilation, we would need authors to install tools like WSL on Windows.

Addendum C. Debugging

Compiling for a Unix target also unlocks the ability to use QEMU and GDB to debug Cosmos kernels. This means that the following features would be supported with no extra effort:

breakpoints on line numbers
backtracing
viewing local variables
catching triple faults
stepping line-by-line

Given GDB also defines a protocol, this can further be integrated with IDEs. A large amount of IDEs have GDB support.

Making Cosmos more like scaffolding, rather than a ready kernel

Currently, Cosmos defines an extremely large part of what a kernel should do. Authors that use Cosmos to learn cannot learn how to make systems like a VFS or a thread scheduler by themselves, and users already experienced with these concepts are limited by Cosmos's ready implementations. Crucial design decisions that define a kernel are already taken by Cosmos, depriving authors from choosing the design of their own operating system.

As a consequence of modularization, the base Cosmos kernel would not include all of these ready components. For those using Cosmos for education, interactive guides could be written to teach systems programming concepts. This is enticing to beginners that view lower level languages like C as "too complicated". Experienced developers, on the other hand, could write their own, custom, components.

All existing APIs, for VFS and whatnot, would be present as optional NuGet packages, for those looking to migrate or to use pre-made components.

cosmicdaman commented 1 month ago

o ma gawd Cosmos.System3

valentinbreiz commented 1 month ago

yeeey ilc

PS: I'm writing a POC for multiplatform C# compilation using https://github.com/xiaoyuvax/MOOS/blob/master/MOOS.bflat.md. My objective is to run at least Cosmos.System under x64 and arm64

cosmicdaman commented 1 month ago

sysV abi, VFS and stuff being optional, possible x64, holy cow

zarlo commented 1 month ago

i have been looking in to porting cosmos plug system to work with any .net project (build time patching of .net dlls)and to make a X# emitter for NativeAOT https://gitlab.com/liquip/nativeaot-patcher

valentinbreiz commented 1 month ago

i have been looking in to porting cosmos plug system to work with any .net project (build time patching of .net dlls)and to make a X# emitter for NativeAOT https://gitlab.com/liquip/nativeaot-patcher

this is incredible

Guillermo-Santos commented 1 month ago

I was thinking on trying to implement something like the 'WebApplication' builder of ASP .NET, but for cosmos, an object where you could add the drivers and devices that needs to initialize, configure logging and maybe Dependency Injection (that is Source Generated in case of NativeAot), maybe could be done for gen3?

Gabolate commented 1 month ago

Currently, Cosmos defines an extremely large part of what a kernel should do. Authors that use Cosmos to learn cannot learn how to make systems like a VFS or a thread scheduler by themselves

I think we also need more links that redirect to websites with useful information so people can learn how to implement stuff by themselves

Epiczhul commented 1 month ago

Poor Linux support Currently, Linux support is quite poor - while one can compile Cosmos kernels under Linux, debugging them is extremely hard. On Windows, Cosmos's Visual Studio debugger is meant to solve this problem, but currently, the debugger isn't fully functional - and works on only one IDE.

what if there were plugins for jetbrains rider? it can use nuget packages

ascpixi commented 1 month ago

Poor Linux support Currently, Linux support is quite poor - while one can compile Cosmos kernels under Linux, debugging them is extremely hard. On Windows, Cosmos's Visual Studio debugger is meant to solve this problem, but currently, the debugger isn't fully functional - and works on only one IDE.

what if there were plugins for jetbrains rider? it can use nuget packages

I'd say that the goal for Gen3 would be to make Cosmos as IDE-agnostic as possible. As far as IDE integration goes, while plugins definetly would be nice, the user should be able to compile, debug, and test their Cosmos projects without an IDE at all.

From my experimentation with Rider, making plugins that use its PSI layer is extremely cumbersome. You can make plugins that use the regular IntelliJ API set, with Kotlin, for example, but we definetly don't have the manpower to make and maintain plugins for different IDEs.

ascpixi commented 1 month ago

Currently, Cosmos defines an extremely large part of what a kernel should do. Authors that use Cosmos to learn cannot learn how to make systems like a VFS or a thread scheduler by themselves

I think we also need more links that redirect to websites with useful information so people can learn how to implement stuff by themselves

I agree - but I also think that developing the educational aspect of Cosmos would be great for Gen3. There is an interest in Cosmos from developers only experienced with higher-level languages that still want to make an operating system. While I do believe that C is the best language to teach people low-level concepts (because of its relatively low amount of abstraction), telling someone "learn C" is definetly demotivating. C# offers a lower barrier of entry that could transition newcomers to C - later on, after they get more experience and gain more knowledge, they may return to Cosmos if they are looking to create a domain-specific kernel.

ascpixi commented 1 month ago

@valentinbreiz @zarlo, is there any news from other maintainers/coredevs? Given that this would be a massive undertaking, and a huge change to the entire project, I feel like at least most of the maintainers would have to give this concept the green light.

cabfile commented 1 month ago

with how slow the progress already is its going to take several years to finish this sort of thing

zarlo commented 1 month ago

@valentinbreiz @zarlo, is there any news from other maintainers/coredevs? Given that this would be a massive undertaking, and a huge change to the entire project, I feel like at least most of the maintainers would have to give this concept the green light.

nothing re this, might need to give it a week

cosmicdaman commented 1 month ago

with how slow the progress already is its going to take several years to finish this sort of thing

all it takes is a big team and dedication :)

9xbt commented 1 month ago

what

cosmicdaman commented 1 month ago

what

he is the flabbergasted

ascpixi commented 1 month ago

We definetly appreciate the enthusiasm, but let's keep the discussion here more akin to a forum, please :)

Arawn-Davies commented 1 month ago

We definetly appreciate the enthusiasm, but let's keep the discussion here more akin to a forum, please :)

What I'm about to say might be a bit controversial, but from someone who has seen Cosmos mature for the past 12 years, I hate to say this but a significant amount of Cosmos users aren't very mature. I stopped using Cosmos and the discord server mainly due to the rampant racism/sexism/etc. which never seems to go away.

While I love Cosmos and it's always had a place dear to me (hell, I was 12 when I first used it and now I'm 24), IMHO it's remained a research project for this reason.

That being said, a rewrite/rebase would be a good idea, help bring the source into the 2020s 🙂

selkij commented 1 month ago

Wow, I didn't see that coming. You're, if not, goated. I have a question, how would we handle concepts that should be customizable but is required to run (like the bootloader for example)? We would have to quickly look into that if the project is accepted so that education-wise and for experienced users, everyone is happy. Even in it's current state, Cosmos doesn't offer proper personalization concerning config files, which, quite counter the goal of cosmos.

9xbt commented 1 month ago

We definetly appreciate the enthusiasm, but let's keep the discussion here more akin to a forum, please :)

What I'm about to say might be a bit controversial, but from someone who has seen Cosmos mature for the past 12 years, I hate to say this but a significant amount of Cosmos users aren't very mature. I stopped using Cosmos and the discord server mainly due to the rampant racism/sexism/etc. which never seems to go away.

While I love Cosmos and it's always had a place dear to me (hell, I was 12 when I first used it and now I'm 24), IMHO it's remained a research project for this reason.

That being said, a rewrite/rebase would be a good idea, help bring the source into the 2020s 🙂

I stopped using Cosmos because of just the community being 99% kids and 1% developers that actually contribute to the project, and also because I wanted to learn real OS development (and cosmos wasn't getting much progress lately either)

The "what" that I said is me just being surprised there's actually still people putting effort into this

ascpixi commented 1 month ago

Wow, I didn't see that coming. You're, if not, goated. I have a question, how would we handle concepts that should be customizable but is required to run (like the bootloader for example)? We would have to quickly look into that if the project is accepted so that education-wise and for experienced users, everyone is happy. Even in it's current state, Cosmos doesn't offer proper personalization concerning config files, which, quite counter the goal of cosmos.

Thanks! This depends on if we are going to use a custom kernel module loader for select modules.

If we were to use a built-in loader...

...we would include some modules as IL assemblies directly via ilc, and some would be regular relocatable file kernel modules. This is similar to the concept of static and shared objects. Bootloader support modules would be "static" modules. I can confirm that ILC, having the --preinitstatics flag specified, can use Limine's executable scanning functionality.

If we were to not include any loader...

...we would need to provide some way for the user to transform "static" modules to "dynamic" ones. We also would provide a NuGet package providing a pre-made kernel module loader, which could include build files to quickly create dynamic modules from plain CLR assemblies (which would be considered "static").

Using APIs from dynamically loaded modules

Using the public API surface of a dynamically loaded module originating from a CLR assembly would be tricky - we would have to somehow defer certain relocations until we load the module in question (plain impossible with e.g. Limine), or use stubs/thunks for every method originating from a module. The stub would work similar to a relocation-based trampoline - e.g., for x86-64:

_ModuleDispatchStub_ExampleMethod:
  mov rax, qword _InvalidDispatchStub       ; pending relocation
  jmp rax

At runtime, we would replace the qword 0 here - just like a relocation. If the relocation isn't yet performed (the module hasn't yet been loaded), we jump to _InvalidDispatchStub, which would be a special function which would raise an exception, e.g. InvalidOperationException: A method call to a module is invalid, as it hasn't yet been loaded.

I feel like this would need a dedicated discussion, with the maintainers weighing in. I apologize if I sound pedantic, but I stress that maintainer activity is crucial for this to go anywhere.

Epiczhul commented 1 month ago

why not rewrite cosmos to work with NativeAOT? Zarlo already mades something to use Cosmos Plug system with NativeAOT. and x86_64 would be nice (possible with NativeAOT) because most CosmosOS's use a lot of ram and 32bit x86 can only adress up to 4GB of Ram.

ascpixi commented 1 month ago

why not rewrite cosmos to work with NativeAOT? Zarlo already mades something to use Cosmos Plug system with NativeAOT.

ILC is a component of Native AOT compilation. It's practically the main one - the compiler. The other components of Native AOT - mainly, the specialized versions of the standard libraries and MSBuild specification files - could also be used, but we would probably have to customize them to the point that it'd just be easier to use the components selectively rather than relying on Native AOT's regular deployment methods.

and x86_64 would be nice (possible with NativeAOT) because most CosmosOS's use a lot of ram and 32bit x86 can only adress up to 4GB of Ram.

x86-64 is not only possible w/ ILC, but required for non-Windows ABI targets. There is no linux-x86 RID support for Native AOT/ILC. The rewrite should fix the "most CosmosOS's use a lot of ram" concern, but a bigger virtual address space would be much nicer for other reasons (e.g. the ability to define a direct physical-to-virtual mapping area).

Epiczhul commented 1 month ago

oki

LeonGamerPS1 commented 1 month ago

oki

piss off from the cosmos discord -sammy

cabfile commented 3 weeks ago

x86-64 is not only possible w/ ILC, but required for non-Windows ABI targets.

wait, so when gen3 happens, it wont be possible to make 32-bit oses anymore?

ascpixi commented 3 weeks ago

x86-64 is not only possible w/ ILC, but required for non-Windows ABI targets.

wait, so when gen3 happens, it wont be possible to make 32-bit oses anymore?

It technically is possible (see https://github.com/ascpixi/smolsharp/pull/3), but it's very much not officially supported. I don't see why we would include support for 32-bit compilation anyways, as that is practically obsolete in the current era.

cabfile commented 3 weeks ago

a few reasons:

some people specifically want to make an os that looks more or less like it came from the 90s, so they wouldnt like the fact that there's only 64-bit compilation
an "os" that is compact and designed for managing, say, disks or any other devices. usually you'd want such an os to be 32-bit (or both 32 and 64-bit)
there could possibly be issues with 32-bit windows/unix executables (in case someone writes something to run them)
it would be nice :)

ascpixi commented 3 weeks ago

a few reasons:

some people specifically want to make an os that looks more or less like it came from the 90s, so they wouldnt like the fact that there's only 64-bit compilation

an "os" that is compact and designed for managing, say, disks or any other devices. usually you'd want such an os to be 32-bit (or both 32 and 64-bit)

there could possibly be issues with 32-bit windows/unix executables (in case someone writes something to run them)

it would be nice :)

I'm not quite convinced - supporting legacy systems is always added workload on the backs of developers.

re. 1. You can just as well make an OS that has a 90s-like design running on 64-bit code. Furthermore, if you'd want to make an OS that's designed like the ones we had in the 90s, wouldn't you want to use legacy x86 real mode...? And good luck with finding a modern compiler that still supports that, haha 😅

re. 2. This implies that 32-bit operating systems are "more compact" that 64-bit ones - that doesn't really make much sense.

re. 3. These are already considered legacy/obsolete. Additionally, most architectures support running 32-bit code under their 64-bit ISAs. The compiler does not matter here - the legacy binaries you'd running would probably come from another compiler altogether, and wouldn't come from Cosmos compilation.

re. 4. Ehhh... supporting legacy technology? If we'd have a good use for targeting 32-bit CPUs, then sure, with enough manpower, we could try to target that as well. But given that none of the maintainers want to take the initiative to actually lead (or even greenlight!) this concept, I'm not sure if we will even have a re-write in the first place.

Given that we still haven't heard anything concrete from any of the maintainers for a month now, I'm afraid that the only way to advance this project is with a fork, with stricter moderation, and with a more active and professional team. :(

cabfile commented 3 weeks ago

This implies that 32-bit operating systems are "more compact" that 64-bit ones - that doesn't really make much sense.

i meant that 32-bit oses can run on both 32-bit and 64-bit machines. imagine you want to repair some old computer using your own os, but cant because its 64-bit. i am aware you can just use something else that actually works on 32-bit machines, but what if you really want to try your own thing on practice?

zarlo commented 3 weeks ago

if we support 64 bit we will not support 32 bit (does not mean we will prevent people but you will be on your own) this goes for ARM and x86

like if you ask me base line support should be x86-64-v2 or x86-64-v3

9xbt commented 3 weeks ago

Dude x86-64 was introduced in the late 90s and the first 64-bit x86 CPU appeared in 2003. I genuinely do not see a point in supporting i386 along with x86-64

9xbt commented 3 weeks ago

Dude x86-64 was introduced in the late 90s and the first 64-bit x86 CPU appeared in 2003. I genuinely do not see a point in supporting i386 along with x86-64

Your average $7 laptop you got from your neighbor is very probably gonna be 64 bit

ascpixi commented 3 weeks ago

This implies that 32-bit operating systems are "more compact" that 64-bit ones - that doesn't really make much sense.

i meant that 32-bit oses can run on both 32-bit and 64-bit machines. imagine you want to repair some old computer using your own os, but cant because its 64-bit. i am aware you can just use something else that actually works on 32-bit machines, but what if you really want to try your own thing on practice?

That seems like an extremely specific use-case. I wouldn't recommend using Cosmos for that purpose, then - just as you shouldn't use Cosmos for e.g. creating a domain-specific OS for PA-RISC. x86 simply wouldn't be a supported architecture (x86-64 and ARM64, however, would be).

It would be nice to be able to target as many architectures as possible, but as it stands now, we should target only the most important architectures first. The difference between 64-bit virtual address spaces and 32-bit ones is massive - it greatly simplifies memory management with things like a direct map. For R/W mappings without any special page table attributes, all it takes to create one is just phys + directMapOffset.

We would be writing the code to be architecture-independent, and that includes using e.g. nuint and nint for places where the native bit-width matters. It's just that we wouldn't focus on 32-bit from the get-go - that could be a candidate for some future PR, maybe...

Gabolate commented 3 weeks ago

All these things are interesting to hear, but i think we also need some kind of structure of how gen 3 could be made, like, should x thing be done first or stuff that we should mainly focus on to discuss, etc.

RecursiveDescent commented 2 weeks ago

Another improvement that could be integrated into gen3 is an improved plug system, and in general going about plugs a different way. As it is now, cosmos is only focused on plugging the lowest level of every api. That's nice and it means a lot of things will work out of the box with no deviation from standard, but... Some things might be better to simply reimplement instead of plug to avoid what I call plug whack-a-mole. It only works until you run into the next thing that needs to be plugged. At the very least I have a suggestion to make plugs much easier to develop.

Anyway, that's not to mention how the contributions might actually go about writing current plugs. Here is an example from UnmanagedMemoryStreamImpl:

byte* pointer = null;

try
{
     for (int I = 0; I < nInt; I++)
     {
          aSpan[I] = *(pointer + aStream.Position + aStream.Capacity);
     }
}
catch { }

...You see the problem here, right?

Plug system improvement

Now, how can the plug system be improved? The first thing is move away from it being a simple patch operation, and more of an assembly merge. My suggestion is that non native plugs would look closer to something like this:

namespace _PLUGS.System.IO {
    public abstract class FileStream : Stream {
        [NativePlug(/* ? */)]
        public override extern void WriteByte(byte b);

        public override long Seek(long offset, SeekOrigin origin) {
            // ...
        }
    }
}

[^1]

Notice how this naturally expresses the members as overriding the original methods without the need for messy attributes. Of course, an alternate way to do this with Attributes is required in case you need to plug a class that cannot be inherited or for other edge cases. And, of course there could even be a way to completely replace a class with your own in case you deem it easier to just implement the whole thing your own way (even if cosmos itself opts to stay closer to the lower level apis).

(Fun fact: You could also probably put this directly in System.IO too! It would work, but having to suppress the warning of a conflict would be uncomfortable)

(EXTRA fun fact: This doesn't mean all the definitions have to be in the same place. If you mark the class as abstract partial you can shard the definition across different places in the source.)

And this is where the implementation aspect of my proposal diverges.

Instead of doing this on the fly as the project is being compiled, you would do this initial step before starting to compile the OS. The general idea is instead of patching as you compile, you merge the original assemblies with the plugs to create a new assembly first, which the project would use instead of the original. This can help compilation speed since depending on how you structure, only native plugs need to be rebuilt.

As for native plugs, the merging tool could compile extern methods marked as native plugs into a stub function, which can later be linked to native code by a later step.

Extra benefits of this are that the native compilation itself only has to worry about compiling the IL into native, not weaving plugs and having to compare signatures as it goes (Signatures are much easier for the merging tool to resolve as well, I personally have an issue with the cosmos plugs for FileStream.Seek failing to find their target method). [^2] As well as allowing plugs to easily be created in the OS project itself, without the need for creating another project and then adding a plug reference. (The merging tool can simply compile the OS IL and merge it against the assemblies as well)

This also makes it easy to statically analyze what methods aren't plugged. The merge tool can generate error stubs with metadata for unplugged methods.

[^2]: This is another issue the current plug system can have. A lot of them target private internal functionality of .NET, which can be subject to change across any version without warning.

[^1]: This method might need some revision, as though it might not be necessary, having it directly inherit the class you want to plug is a problem with this because of namespace scoping, meaning it would try to inherit itself instead of the original class. This might be a desired effect, though, it's just that your class might not need to derive from anything, and might not need to use override keyword.

Guillermo-Santos commented 2 weeks ago

Well, an improvement would be that it warns you if native code is encountered on any type that you use (as long as you are not using that member, if used, then error as usual), it could be a parameter you set on the project file or something, so that if you are working on plugin something then you get to know all of the places that possible needs patching. This could be useful also when changing .NET Versions as you can compare with an old scan to see what plugs are no longer needed and what new plugs you need to implement. I think that is may be even possible with today's implementation too.

zarlo commented 2 weeks ago

Well, an improvement would be that it warns you if native code is encountered on any type that you use (as long as you are not using that member, if used, then error as usual), it could be a parameter you set on the project file or something, so that if you are working on plugin something then you get to know all of the places that possible needs patching. This could be useful also when changing .NET Versions as you can compare with an old scan to see what plugs are no longer needed and what new plugs you need to implement. I think that is may be even possible with today's implementation too.

so a plug stub generator? if so that would not be to hard to make

Guillermo-Santos commented 2 weeks ago

so a plug stub generator? if so that would not be to hard to make

Yes

Samma2009 commented 2 weeks ago

are you guys actually working on this or it is just talk lol

Gabolate commented 2 weeks ago

All these things are interesting to hear, but i think we also need some kind of structure of how gen 3 could be made, like, should x thing be done first or stuff that we should mainly focus on to discuss, etc.

@Samma2009 i sent a message here with what i think its needed before coding, but nobody seems to do what's in there ;_;

ascpixi commented 1 week ago

I mentioned this before, but first and foremost we need activity from the maintainers. We definetly would want to work on this in the CosmosOS organization, and API changes would need to be discussed. The only activity I've seen was from @zarlo's screenshot of an internal WhatsApp (?) group, but that's about it.

In order to effectively coordinate this, we need:

manpower. At least 6 active contributors to gen3 experienced with both Cosmos and systems programming. Please note that while it's definetly nice to see new contributors wanting to contribute to the project, we do need people with enough experience on this project. Of course, this is an open source project, and the active contributors change, but we do need an initial group of contributors to spearhead the project.
a centralized knowledge base. A Discord channel, a Slack team, a Trello board - you name it. We need some kind of authority (in our case, a maintainer) to create a good way for us to discuss the project.
maintainer activity. It's a bit hard to merge pull requests without maintainers actively looking into the project. A rewrite is an extremely demanding project that requires discussion with the coredevs. There are definitely going to be disagreements between different parties when it comes to technical matters, and so we do need a reliable communication channel with the maintainers.

Adding some active contributors to the maintainer team would also be welcome. There is, however, an issue of trust on this part, and so take that proposition with a grain of salt.

ascpixi commented 1 week ago

re: the plug system improvements suggested by @RecursiveDescent

In general, I agree with this new system, except for using a specialized namespace name to place all plugs into. Instead, I recommend simply marking plug classes with a PlugAttribute - like so:

using System.IO;

namespace Cosmos.Plugs;

[Plug(typeof(System.IO.FileStream))]
public class FileStream
{
    // ExposeAttribute exposes private members, making a "pseudo-member". This results
    // in a member that only exists as metadata - all references to it will be replaced with
    // the actual member.
    [Expose] ulong _privateField;
    [Expose] extern void PrivateMethod();

    [NativePlug(/* ? */)]
    public override extern void WriteByte(byte b);

    [Plug]
    public override long Seek(long offset, SeekOrigin origin)
    {
        SomeUtilityMethod();
        PrivateMethod();
        // ...
    }

    public void SomeUtilityMethod()
    {
        Console.WriteLine("Hello, Cosmos!");
    }
}

Another benefit from generating plugged standard library assemblies ahead-of-time is better IntelliSense - methods that use unplugged methods anywhere in the chain would be dropped from the final output.

I'd also like to underline that when we'll be doing plugs for the new plug system, we should never target platform-dependent methods. That is, if a method mentions either Windows or Unix, methods higher in the chain (that call such methods) should be plugged instead - even if that would result in more plugs or in duplicate code.

Samma2009 commented 1 week ago

re: the plug system improvements suggested by @RecursiveDescent

In general, I agree with this new system, except for using a specialized namespace name to place all plugs into. Instead, I recommend simply marking plug classes with a PlugAttribute - like so:
using System.IO;

namespace Cosmos.Plugs;

[Plug(typeof(System.IO.FileStream))]
public abstract class FileStream : Stream
{
    // ExposeAttribute exposes private members, making a "pseudo-member". This results
    // in a member that only exists as metadata - all references to it will be replaced with
    // the actual member.
    [Expose] ulong _privateField;
    [Expose] extern void PrivateMethod();

    [NativePlug(/* ? */)]
    public override extern void WriteByte(byte b);

    [Plug]
    public override long Seek(long offset, SeekOrigin origin)
    {
        SomeUtilityMethod();
        PrivateMethod();
        // ...
    }

    public void SomeUtilityMethod()
    {
        Console.WriteLine("Hello, Cosmos!");
    }
}
Another benefit from generating plugged standard library assemblies ahead-of-time is better IntelliSense - methods that use unplugged methods anywhere in the chain would be dropped from the final output.

I'd also like to underline that when we'll be doing plugs for the new plug system, we should never target platform-dependent methods. That is, if a method mentions either Windows or Unix, methods higher in the chain (that call such methods) should be plugged instead - even if that would result in more plugs or in duplicate code.

Why you guys keeping the plug system.... Wouldn't it be easier to have a corelib-based system... Specially with NativeAOT

Also I asked kuduzu and yes he would like to come back to cosmos in the future

ascpixi commented 1 week ago

Why you guys keeping the plug system.... Wouldn't it be easier to have a corelib-based system... Specially with NativeAOT

Also I asked kuduzu and yes he would like to come back to cosmos in the future

Mainly because the previous maintainers are comfortable with this system - I'm all for a custom standard library, but that will affect compatibility with existing NuGet packages (they simply wouldn't be using the same assemblies - we could make some migration tool that would automatically replace references, though).

CosmosOS / Cosmos

🚀 Cosmos Gen3 re-write proposal #3088

The need for a rewrite

Issues with the current code-base

Lack of thread safety

Poor API design

Inconsistent code style and readability

Reliance on IL2CPU

Poor Linux support

Megalithic

No defined ABI

Potential improvements in Gen3

Design with thread safety in mind

Modularized approach

New ILC-based compilation pipeline

1. Transform standard library assemblies according to the plug set

2. Compile main kernel code

3. Link

Addendum A. ABI

Addendum B. Compiling on non-Linux hosts

Addendum C. Debugging

Making Cosmos more like scaffolding, rather than a ready kernel

If we were to use a built-in loader...

If we were to not include any loader...

Using APIs from dynamically loaded modules

Plug system improvement