Evidence

[bennn]

Evidence that the contract profiler is doing a good job.

• The profiler really is measuring the cost of verifying that functions are (-> any/c boolean?). It really is that slow.
• Removing those contracts so they're never attached to values gives the expected speedups.

Also, some notes on the recent changes to MBTA and zordoz, and an interesting result with list/c vs. vector/c. No luck on adding a new continuation mark for list contracts yet.

[stamourv]

:) I'm curious about the bit re vector/c and list/c and continuation marks. Mind elaborating?

[bennn]

Yes, and I'd also like advice!

1. We had a funny result from the profiler on the tetris program. Checking (-> any/c boolean?) contracts caused 44.25% of the total contract runtime.
2. Removing all (-> any/c boolean?) contracts led to a 10-second speedup. But improving a few function signatures to export "structs containing lists" instead of just lists caused a 40-second speedup.
3. We realized that the contract profiler doesn't attribute recursive costs to the list contract that generated them. I was wondering how to add that ability.

Do you know where listof checks start in the contract library? I was hoping to add a new mark (list-contract-mark) at that location, and no antimark. So far I've poked around contract/private/misc.rkt but I haven't seen functions like listof-exercise called.

---

The "interesting result" was that on a microbenchmark, vector contracts seemed cheaper than list contracts. To explore, I changed all lists in tetris to vectors, and this improved the overall runtime from 30656ms to 150ms. Then again, doing the same thing with snake led to less time checking contracts, but increased overall runtime from 20,937ms to 252,137 ms.

Asumu pointed out that vector contracts are checked lazily (on access); I think that explains it.

[stamourv]

Re 1: Even these simple contracts involve wrapping, which is usually the expensive part. So 44% does not surprise me. I think struct contracts are also lazy, which would explain 2. Re 3: Ah, it's possible that listof contracts do not add a mark by themselves, which would be a bug. (But a minor one, as most of the costs would lie in the subcontracts, which will have marks.) I don't see why they should have a different mark, though. The regular contract mark would work; the profiler gets the contract itself from the blame object. As for where the checking code for these contracts lives, I don't know, but I'll look.

[stamourv]

Ok, try adding marks inside the λ (val)s inside listof-projection. These lambdas perform the actual checking (which can only be done once you have received the value you're checking, hence the λ (val)), and so that's where the mark should go.

[takikawa]

BTW, vector contracts can also be eager/flat if you supply the #:immutable keyword and all the subcontracts are flat. So try that if you haven't already and I bet you'll get similar numbers between vectors and lists.

[bennn]

Cool, I will try adding those marks.

Asumu, I couldn't find #:immutable in the docs, but using vector->immutable-vector did bring performance back down to what lists give. (note: I make a single vector/list and pass it to a function many times)

[bennn]

@stamourv adding marks inside listof-projection doesn't seem to have worked. Re-running the contract profiler gives me the same results as before (same low cost attributed to listof contracts -- I can send the code).

Also, when I add print statements under the 4 new marks, only the second is ever printed, and only at compile-time.

I'll think about this more seriously tomorrow, but I wanted to ping you in case you know of an obvious place to look next.

[stamourv]

@benn: I'm surprised. If the code only prints at compile time, then it's not the right place. The loops inside these lambda (val) definitely look like run-time checking to me, though.

[bennn]

In the end, I got what I wanted by changing the profile to attribute costs to the last stack frame rather than the first (i.e., give me the contract that led me to the current check, rather than the contract I'm currently checking).

This definitely loses some information, but seems like a nice alternate view. If it really is useful, I'll formulate a pull request (new parameter) for the contract profiler.

[stamourv]

That sounds reasonable. Actually, if we want to get fancy, we could go for an "edge contract profiler", kind of like what the Racket profiler does. Not sure that would make the info easy to digest, though. Could also add a level of nesting to the text view, which would show the cost of each subcontract. But again, the amount of info may be overwhelming.