refine parsnip overhead test

[simonpcouch]

Maybe every three or four times we run the parsnip overhead unit tests on the macOS-release and ubuntu-oldrel-4 runners, we see a failure. We should figure out how to make that happen much, much less often.

Starting out by making the error message more informative to see whether we're "close" to the limit.

[simonpcouch]

The hard-coded limit for fit_xy() is a factor of 3.5x in the minimal example. We see in one failure:

parsnip overhead factor (xy interface): 3.52

Sooo.. it is a close call, which made me think we may have indeed that fit time slightly in recent changes. Profiling the dev version:

This is actually the case—calls to proc.time() from https://github.com/tidymodels/parsnip/commit/1e7dad2753f8bd0629f5c49bfaca854b4f4167b9 (and before that, https://github.com/tidymodels/parsnip/commit/40524aab2cd6b4605fa4b7f65c91c3b363fc16ea) are what's causing that inflation in fit time.

[EDIT: This graph makes it appear like proc.time() is taking up a bunch of time, but it's actually because it wraps the model fitting code and messes with the stack trace in a way that profvis doesn't know what to do with. Very little time is being taken up by proc.time() itself.]

[simonpcouch]

Here are the distributions of those factor slowdowns I see locally:

library(parsnip)
library(bench)

form <- numeric()
xy <- numeric()

for (i in 1:30) {
  bm <- bench::mark(
    time_engine = lm(mpg ~ ., mtcars),
    time_parsnip_form = fit(linear_reg(), mpg ~ ., mtcars),
    time_parsnip_xy = fit_xy(linear_reg(), mtcars[2:11],  mtcars[1]),
    relative = TRUE,
    check = FALSE
  )

  form <- c(form, bm$median[2])
  xy <- c(xy, bm$median[3])
}

hist(form)

The limit for the above image is currently 3.

hist(xy)

The limit for the above image is currently 3.5.

^{Created on 2024-08-29 with reprex v2.1.1}

On average, these bench::mark() runs are running 300 to 2000 iterations per expression.

At least on this PR, I've seen 3/6 runs failing with XY overhead factors of 3.52, 3.56, and 3.501. I've yet to see any failures with the formula interface but have seen them in the past.

Alternatives from bench and ps are a good bit slower, at least on macOS. I do think the timing functionality is worth having, and it's made it to CRAN already. [EDIT: this functionality has not actually made it to CRAN.] I'm leaning towards increasing that overhead limit for both interfaces slightly.

[github-actions[bot]]

This pull request has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.