Performance loss in some commits

[p6rt]

Migrated from rt.perl.org#131515 (status was 'resolved')

Searchable as RT131515$

[p6rt]

From @zoffixznet

Recently, we have had about ~2.7% overall performance loss, when running stresstest/spectest that a couple of devs noticed. Tux's bench also showed a slowdown.

Based on stresstest timings I post ( https://irclog.perlgeek.de/perl6-dev/search/?nick=&q=ZOFVM ), the slowdown happened in some commits made between 2017-05-31 and 2017-06-01. I tried narrowing down what brought the slowdown, but couldn't quite pinpoint it, because there's some randomness in timings of some tests. Here's my results of test times​:

SHA Stresstest (s) Spectest (s) aebd0fa 123
01d948d 122
2500e50 121 / 122 86 79b8ab9 119 / 118 83 b879060 117 / 119 82 / 82 3f5cc5a 117
3488914 118
fb7dd8a 119

As you can see, the middle commits is where the slowage appears to occur, with 2500e50 being the most likely culprit, even though, the code itself isn't something that would be even remotely hot.

I went and rewrote it in NQP (attached) to see if it drops the numbers at all, and... it didn't really. I got a 120s stresstest and a 85s spectest.

I gave up with this.

I think we need some extensive bench without any random sleeps or delays in it and comprehensive enough to test more than just a few text methods. Perhaps, something that can be cooked up with https://map.perl6.party/

And with such a bench in hand, it might become easier to hunt commits that slow stuff down considerably.

[p6rt]

From @zoffixznet

exception.diff

[p6rt]

From @lizmat

On 5 Jun 2017, at 19​:30, Zoffix Znet (via RT) \perl6\-bugs\-followup@​perl\.org wrote​:

# New Ticket Created by Zoffix Znet # Please include the string​: [perl #​131515] # in the subject line of all future correspondence about this issue. # \<URL​: https://rt-archive.perl.org/perl6/Ticket/Display.html?id=131515 >

Recently, we have had about ~2.7% overall performance loss, when running stresstest/spectest that a couple of devs noticed. Tux's bench also showed a slowdown.

Based on stresstest timings I post ( https://irclog.perlgeek.de/perl6-dev/search/?nick=&q=ZOFVM ), the slowdown happened in some commits made between 2017-05-31 and 2017-06-01. I tried narrowing down what brought the slowdown, but couldn't quite pinpoint it, because there's some randomness in timings of some tests. Here's my results of test times​:

SHA Stresstest (s) Spectest (s) aebd0fa 123
01d948d 122
2500e50 121 / 122 86 79b8ab9 119 / 118 83 b879060 117 / 119 82 / 82 3f5cc5a 117
3488914 118
fb7dd8a 119

As you can see, the middle commits is where the slowage appears to occur, with 2500e50 being the most likely culprit, even though, the code itself isn't something that would be even remotely hot.

I went and rewrote it in NQP (attached) to see if it drops the numbers at all, and... it didn't really. I got a 120s stresstest and a 85s spectest.

I gave up with this.

I think we need some extensive bench without any random sleeps or delays in it and comprehensive enough to test more than just a few text methods. Perhaps, something that can be cooked up with https://map.perl6.party/

And with such a bench in hand, it might become easier to hunt commits that slow stuff down considerably.\<exception.diff>

Generally, we don’t care about performance once we have an unrecoverable exception that needs to be reported. If the stresstest regression is caused by trying to do Levenstein on method names (which can be a lot on some objects), then maybe the performance of spectest/stresstest can be accounted for by *just* the tests that throw Method Not Found errors?

[p6rt]

The RT System itself - Status changed from 'new' to 'open'

[p6rt]

From @zoffixznet

On Mon, 05 Jun 2017 13​:28​:00 -0700, elizabeth wrote​:

Generally, we don’t care about performance once we have an unrecoverable exception that needs to be reported. If the stresstest regression is caused by trying to do Levenstein on method names (which can be a lot on some objects), then maybe the performance of spectest/stresstest can be accounted for by *just* the tests that throw Method Not Found errors?

Yeah, I think the nqp version of the exception is proof that's the wrong commit, otherwise it'd've got faster than even pre-commit.

Something else was the cause, but I ran out of steam hunting it.

[p6rt]

From @AlexDaniel

I'll take a look a bit later. It feels like the commit you mentioned is the right one, which makes me think that we can reject this ticket, but at that pace the spectest will get to turtle speeds even though all our changes will be justified.

Anyway, I'll bisect it first and then we'll have a look at what can be done to Make the Spectest Fast Again.

On 2017-06-05 14​:27​:28, cpan@​zoffix.com wrote​:

On Mon, 05 Jun 2017 13​:28​:00 -0700, elizabeth wrote​:

Generally, we don’t care about performance once we have an unrecoverable exception that needs to be reported. If the stresstest regression is caused by trying to do Levenstein on method names (which can be a lot on some objects), then maybe the performance of spectest/stresstest can be accounted for by *just* the tests that throw Method Not Found errors?

Yeah, I think the nqp version of the exception is proof that's the wrong commit, otherwise it'd've got faster than even pre-commit.

Something else was the cause, but I ran out of steam hunting it.

[p6rt]

From @zoffixznet

So it *is* the commit originally found, but the Exception.message call was later optimized away to being called only when needed, and not when the exception is thrown, so this don't matter anymore.

Commit timing results​: https://irclog.perlgeek.de/perl6-dev/2017-10-14#i_15303228

[p6rt]

@zoffixznet - Status changed from 'open' to 'resolved'