search

Mathics3 / mathics-benchmark

Benchmark tools for mathics-core
https://github.com/Mathics3/mathics-core
GNU General Public License v3.0
2 stars 0 forks source link

Rewrite compare.py #5

Closed TiagoCavalcante closed 3 years ago

TiagoCavalcante commented 3 years ago

image

Now it is working.

My idea is verticals bars with groups, and below the plot, the list of queries of each group. But this is a start.

TiagoCavalcante commented 3 years ago

I want to add an optional group option, so the plot would be more readable.

Do you think that should be done in this PR?

rocky commented 3 years ago

I want to add an optional group option, so the plot would be more readable.

Do you think that should be done in this PR?

It is your choice. I will be using this in a day or so. And I can probably use it either in a branch or master (although master is easier for me in case there are more changes elsewhere that I'd like to make)

TiagoCavalcante commented 3 years ago

It is your choice. I will be using this in a day or so. And I can probably use it either in a branch or master (although master is easier for me in case there are more changes elsewhere that I'd like to make)

I'll do it in this PR.

Also, what do you think of calling bench.py if the results doesn't exist?

TiagoCavalcante commented 3 years ago

@rocky could you review the comments?

TiagoCavalcante commented 3 years ago

A follow-on thing that we might consider is listing a table of percentage changes for each test.

This would be an output or a file?

rocky commented 3 years ago

It is your choice. I will be using this in a day or so. And I can probably use it either in a branch or master (although master is easier for me in case there are more changes elsewhere that I'd like to make)

I'll do it in this PR.

Also, what do you think of calling bench.py if the results doesn't exist?

That is a great idea!

I imagine you mean calling using the module mathics_benchmark.bench rather doing some kind OS fork.

rocky commented 3 years ago

A follow-on thing that we might consider is listing a table of percentage changes for each test.

This would be an output or a file?

I was thinking in a legend of a chart. But having an option to write the data to a file is a nice addition too.

TiagoCavalcante commented 3 years ago

I imagine you mean calling using the module mathics_benchmark.bench rather doing some kind OS fork.

What I made is calling the main function from bench.py.

I was thinking in a legend of a chart. But having an option to write the data to a file is a nice addition too.

It can be in the plot, think that adding it to the results file maybe be unnecessary data, as that would be a duplicate. I didn't understand what you meant before.

Also, about it what do you think of the following format: example bar chart

The smaller bar is in the front and the other in the back. For example: if master takes 10 seconds and a branch 8 seconds, the bar would be 80% from the color of that branch and 20% of the master branch color.

That would be better to see when there are too many bars, but I don't know if this format is so easy to understand or if it is common.

rocky commented 3 years ago

I imagine you mean calling using the module mathics_benchmark.bench rather doing some kind OS fork.

What I made is calling the main function from bench.py.

I was thinking in a legend of a chart. But having an option to write the data to a file is a nice addition too.

It can be in the plot, think that adding it to the results file maybe be unnecessary data, as that would be a duplicate. I didn't understand what you meant before.

Also, about it what do you think of the following format: example bar chart

The smaller bar is in the front and the other in the back. For example: if master takes 10 seconds and a branch 8 seconds, the bar would be 80% from the color of that branch and 20% of the master branch color.

That would be better to see when there are too many bars, but I don't know if this format is so easy to understand or if it is common.

I may be me but personally, I find combined bars harder to follow then side by side. Especially when there is % comparison.

TiagoCavalcante commented 3 years ago

Ok.

The percentage should be ref1 vs ref2 or ref2 vs ref1? I guess it would be the first as the ref2 is master by default.

TiagoCavalcante commented 3 years ago

image

Positive is bad, maybe change the sign? Or the color?

rocky commented 3 years ago

image

Positive is bad, maybe change the sign? Or the color?

Usually green is for improvement and red is for regression. If you want to get slick it could change color by the amount.

However it looks to me like the percentage are low by a factor of 10 by looking at the bar sizes.

TiagoCavalcante commented 3 years ago

The percentage difference was completely wrong.

I adjusted the colors so it is easier to see the percentages: image

I also added an option to hide the percentages and show more bright colors (with them is hard to see the numbers).

TiagoCavalcante commented 3 years ago

If you want to get slick it could change color by the amount.

This is hard to do, and if it is done it would be slow, in matplotlib the whole array of the legends is of the same color, so an array would be generated for each different color.

TiagoCavalcante commented 3 years ago

@rocky I think it is ready (maybe except for the colors). What do you think?

rocky commented 3 years ago

@rocky I think it is ready (maybe except for the colors). What do you think?

Yes, I think this is great. I think the colors are fine too. More adjusting can come later on if that is needed/desired.