search

vnmakarov / mir

A lightweight JIT compiler based on MIR (Medium Internal Representation) and C11 JIT compiler and interpreter based on MIR
MIT License
2.33k stars 150 forks source link

Awk has a syntax error in the benchmarks #382

Open rempas opened 1 year ago

rempas commented 1 year ago

Specifically, the line shows: awk: cmd. line:1: BEGIN {if (2,88==0) print "Inf"; else printf "%.2f\n", 2,88/2,88;} awk: cmd. line:1: ^ syntax error.

vnmakarov commented 1 year ago

It works for me on Linux. It seems 2,88 is taken from some time utility which outputs time with comma. Probably LC_NUMERIC was set up on your system.

Could you give more info how to reproduce it.

rempas commented 1 year ago

It works for me on Linux. It seems 2,88 is taken from some time utility which outputs time with comma. Probably LC_NUMERIC was set up on your system.

Could you give more info how to reproduce it.

Well... I truly haven't changed anything! I could give you info about my system!

Linux: 6.6.2.arch1-1 Awk: GNU Awk 5.3.0, API 4.0, PMA Avon 8-g1, (GNU MPFR 4.2.1, GNU MP 6.3.0)

iacore commented 1 year ago

Have you tried removing all LC* and LANG* environment variables before running the benchmark?

rempas commented 1 year ago

Have you tried removing all LC* and LANG* environment variables before running the benchmark?

@vnmakarov said the same, but I'm apparently blind :facepalm:

Yes, "LC_NUMERIC" is set. Unsetting it solves the problems.

Now, before I close that. Should the script unset it manually?

rempas commented 1 year ago

@vnmakarov

To add on my previous reply (I don't edit in case someone is reading it right now), I run a script and when you unset it inside a script, it will not affect the shell when the script is over. So I would suggest you make the script unset it.

Btw, I did run the benchmarks compared to clang: -O2, -O3 and -O3 -finline -finline-functions and the geomean of c2m -eg was 0.75x of clang -O2 so good job!

I would also suggest that you show the compile times for each test and then the total compile time of all as this is also a metric!

iacore commented 1 year ago

Now, before I close that. Should the script unset it manually?

It should, although that's a work around.

What was your LC_NUMERIC set to?

rempas commented 1 year ago

It should, although that's a work around.

What was your LC_NUMERIC set to?

It was set to: el_GR.UTF-8

iacore commented 1 year ago

I cannot reproduce this.

What I ran in fish:

set -x LC_NUMERIC el_GR.UTF-8
make --debug bench

what command did you use to run the benchmark?

vnmakarov commented 1 year ago

Thank you for the discussion. I addrd clearing LC_NUMERIC to the script.

Btw, I did run the benchmarks compared to clang: -O2, -O3 and -O3 -finline -finline-functions and the geomean of c2m -eg was 0.75x of clang -O2 so good job!

If you use master branch, you should check bbv branch. It generates a better code (by 5-10%).

https://developers.redhat.com/blog/2021/04/27/the-mir-c-interpreter-and-just-in-time-jit-compiler#how_the_mir_c_compiler_compares_with_other_c_compilers might be interesting for you as it compares a lot of different C compilers although the data are not fresh.

rempas commented 1 year ago

I cannot reproduce this.

What I ran in fish:

set -x LC_NUMERIC el_GR.UTF-8
make --debug bench

what command did you use to run the benchmark?

I am using ZSH (kernel and awk info above) and I run: ./c-benchmarks/run-benchmarks.sh

rempas commented 1 year ago

Thank you for the discussion. I addrd clearing LC_NUMERIC to the script.

Thanks. Even if @iacore didn't have the same problem (when trying to set LC_NUMERIC) and some others may not, you will save some time from the issues of people who will have that problem like me.

If you use master branch, you should check bbv branch. It generates a better code (by 5-10%). Also, does this change the compilation times?

Thanks for the info. We have talked about it, and I was using that branch for my test until now, but I thought about sticking to master as these two will probably merge one day, right? If not, what is bbv about, and should I always use this, especially when testing?

although the data are not fresh.

Thank you for the info, but 2 and a half years ago is a loooooot of time! In that period, clang (llvm) went from both compiling slower and producing worse code than GCC to basically doing an 180 degree and beating GCC on my system!

So yeah, I prefer to have the benchmark suit as a good indication of how things are going as compilers change rapidly. The only thing I would suggest, is to add compile times for every test separately and a sum in the end.

In any case, your work opens new possibility and ways to think about compilation! I have tried to compile this program (no recursion version and slightly modified) and the output file is 16.000 bytes for clang and just 360 bytes for Mir!!! While the execution takes more for Mir. And I suppose you can still do a lot of optimization for both files sizes, compilation speed and code quality compared to LLVM who is already optimized as much as possible (or at least "practically" possible). JIT is the future!!!!

I will try with the bbv branch and come I'll post update!

rempas commented 1 year ago

Update: Using the bbv branch, c2m -eg has 0.88x (average) and 0.81x (geomean) the runtime of clang -O3 (didn't use "inline" in these tests as I practically won't use them in production) while producing 40-50 times smaller file sizes and compiling 3-4 times faster!

Add shebangs, system calls and libraries, and I don't see a reason for LLVM to exist for most use cases, lol! And I won't even talk about QBE...

I would like help making the benchmark tool more advanced, but as I don't know shell scripting, I have another idea. I will create another benchmark suite written in Typescript, and it will require bun to run. This can be offered as an alternative for someone that wants something more advanced. It will also show the compile times and file sizes and will let you choose which "capabilities" you want to use. So, yours will be there (and you can modify as needed) but there will be another one. What do you say? I will do it anyway for personal use, but would you want me to make a merge request once it is done? Do you have any other idea?

iacore commented 1 year ago

I would like help making the benchmark tool more advanced, but as I don't know shell scripting, I have another idea. I will create another benchmark suite written in Typescript, and it will require bun to run.

If you want, try Janet. you can include the language itself as janet.c and bootstrap it from C compiler in source.

rempas commented 1 year ago

If you want, try Janet. you can include the language itself as janet.c and bootstrap it from C compiler in source.

You mean Janet?

If yes, then the point is to not have to spend time learning something else that won't use anywhere else and instead, spent that time to work on an easy language that I know and have the benchmarks faster.

I also cannot spend a lot of time as I have to learn Svelte and continue my work in Nemesis. So, I'm going to have to make this quick!

iacore commented 1 year ago

I don't think languages other than C or shell script fit in this project though. JS/bun is probably not a good fit.

My reasoning was, Janet script can be compiled to C. If you know C, C is better.

rempas commented 1 year ago

I don't think languages other than C or shell script fit in this project though. JS/bun is probably not a good fit.

Yeah, I would agree if we were talking about the source code, but this is about benchmarking, so I don't think that it's a big problem as long as it as SECOND choice and the shell script remains. That's why I didn't say to remove the current benchmarks.

My reasoning was, Janet script can be compiled to C. If you know C, C is better.

I understand but like I said I don't have so much time. I know C but C will make my life so much harder...