Multi threading in compiler..

[mjarkk]

As we now get to a point were we can start to support importing other files yay we might also want to also look into multi threading because the more things we start to support and do the slower the compiling progress becomes.

Rust perfectly fine support multi threading so no issue there but because i have never written a compiler i can only go after my guesses to how this should work.
In the bases i think we should defensively use a thread pool (i think this is whats it is called) with x works for the amount of threads the system has or user defined.

From there we have a lot of options to how we should use those threads and here are just some random ideas we could use and mix if we want to.

1. reading files in their own job
2. file tokenization is it's own thread job
3. function bodies get parsed in their own job
4. parsing stage 2 into it's own jobs
5. parsing stage 2 separate everything into jobs
6. only separating files into their own job
7. having some kind of requirement to parsing things and based on that requirement parse things in order.
8. only spawn threads once we need them
9. using that new rust async thing. I might add more later but this is it for now.

Note 1: Moving things between threads can take up time so the job compute size must be worth waiting for it moving it around threads.
Note 2: IO in rust is non-async by default and so the full thread is blocked and this is a lot of wasted time if only dune in 1 thread.

[TheOtterlord]

This does sound good. Sadly I have no experience with threading and minimum experience with asynchronous programming, so I may not be able to help right away.

From what I can tell reading your note, we are better off starting by reading the files in their thread. Once we have this it should be easier to continue. It may also allow us to start thinking about our file/package organisation and how we wish to implement it.

[TheOtterlord]

Personally, I quite like the organisation used in NPM packages, but I realise that since we are compiling into may languages it won't be that simple.

When we start to import files, we are going to need to decide how people implement native code into a library. Hopefully, we can deal with some of this by offering built in libraries that deal with some of the issues.

[TheOtterlord]

I've just done some research into threading, and agree with the thread pool approach.

I don't really use C-like languages very often, but I remember compiling something last year where my computer was unable to compile it without overheating. To fix this, I had to pass an optional argument that limited the number of threads that were used when compiling. I think this is an option that our programming language should implement.

In chapter 20, section 2 of the rust online book, they implement a thread pool that only allows a finite number of threads. I think we could modify this to work for the optional argument.

Edit: Also, what editor are you using for this project? I am currently making a simple grammer extension for our language that works in vscode and atom.

[mjarkk]

When we start to import files, we are going to need to decide how people implement native code into a library.

I think after compiling we should output files as something the end user can use as library.
What we don't want to do here is output the full library as in also a package.json for an npm package we only want to output native code files that try to follow the general way of making a library in that specific language.
This allows the user to also choose not to use it as a literal library but also via normal imports or whatever the language has.

In chapter 20, section 2 of the rust online book, they implement a thread pool that only allows a finite number of threads. I think we could modify this to work for the optional argument.

Jup we will need to create something similar

Edit: Also, what editor are you using for this project? I am currently making a simple grammer extension for our language that works in vscode and atom.

Cool, i'm using vscode

Also one last thing, from what i can remember rust tests don't support threads so we will have to also make sure everything still works when using no threads.

[TheOtterlord]

I think after compiling we should output files as something the end user can use as library. What we don't want to do here is output the full library as in also a package.json for an npm package we only want to output native code files that try to follow the general way of making a library in that specific language. This allows the user to also choose not to use it as a literal library but also via normal imports or whatever the language has.

Yeah, this makes more sense. I'm just trying to get a sense of how native written libraries will be supported if they are written for more than one supported language, so the end user can compile it without having to change imports depending on the target platform. For example, our built-in system libraries, such as the console logging one will be written natively, but will be written multiple times, each for a different supported end language. I suppose that as long as there was some form of platform detection for libraries, that could solve the problem.

Cool, i'm using vscode

Awesome, me too! It's not done yet, but as we implement new coding features and I update my example file, I will continue to add support for the language. Once it's all finished, I can put it in a repo.

Also one last thing, from what i can remember rust tests don't support threads so we will have to also make sure everything still works when using no threads.

If we create an optional argument in Parser:parse that specifies the number of maximum threads, we could account for it being 0 (as it would be in the tests) and just compile without threading.

[mjarkk]

such as the console logging one will be written natively

My idea was not to support native function but rather let the user define these functions.
When someone wants to use native functions like console.log they should add define log = fn (string) or something like that and once it's compiled the user must define these functions somewhere to use the created library.
Because the goal for every language is to create a kinda library there should be some init function i guess and there the user should define these functions in there.
I haven't thought this to much trough i only have some ruff ideas to how i think it should work

I can put it in a repo.

Once you can create a repo should we also move everything over to a github org because then we already have 2 repos to keep track of?
But now i'm saying this we should maybe also start thinking of a better name for this project because the org also needs a name.

If we create an optional argument in Parser:parse that specifies the number of maximum threads, we could account for it being 0 (as it would be in the tests) and just compile without threading.

I guess so, but i'm thinking of the complicattion this will add i'll postpone the multi threaded work a bit i think :)

[TheOtterlord]

My idea was not to support native function but rather let the user define these functions.

Ok, this works.

Once you can create a repo should we also move everything over to a github org because then we already have 2 repos to keep track of? But now i'm saying this we should maybe also start thinking of a better name for this project because the org also needs a name.

I think that before creating an organisation, we should at least have a 0.1 version for our language that can compile into at least one language. I think that we should start thinking about names now as with the rate the project is going, hopefully it won't be long until that 0.1 version.

I guess so, but i'm thinking of the complication this will add i'll postpone the multi threaded work a bit i think :)

Yeah, we should probable plan this out properly before implementing anything. I'd keep the issue open to store any notes/plans.