search

codewars / codewars-runner-cli

Old CodeRunner project. See https://github.com/codewars/runner instead.
GNU Affero General Public License v3.0
402 stars 141 forks source link

CFML Runner timing out for large test suites #787

Closed bdw429s closed 4 years ago

bdw429s commented 4 years ago

I'm not entirely sure if this is the best place to discuss this or not. I'm working on translating a kata that specifically processes large amounts of data for the express purpose of ensuring you've written a linearly scaling algorithm.

https://www.codewars.com/kata/5dcde0b9fcb0d100349cb5c0

The code is pretty fast (and my algorithm is linear as best I can tell), but still taking several seconds which is pretty much expected. However, I'm dealing with a lot of intermittent timeouts that claim my tests have taken over 12 seconds to run. However, the timing stats at the bottom of the test output (when it displays) tell another story. I have a feeling the tests ARE taking some time and are likely close to the limit, but I'm wondering if the spool up time of the container is being counted against my time.

image

In that screenshot you can see the tests claim they finished in well under 3 seconds, but the attempt timed out and says a 12 second limit was reached.

kazk commented 4 years ago

Is the container spool up time counted against the 12 seconds?

Yes. This is the same for all languages (e.g., compiled languages needs to be able to compile and run within the time limit). For CFML, this includes the CFML engine's start up time, starting CommandBox, then running tests, and outputting the tests.

We can increase the maximum time allowed to 16s or something.

bdw429s commented 4 years ago

We can increase the maximum time allowed to 16s or something.

Yes please. From what I can see the total time will report as right under 12 seconds but my test timers will show < 3 seconds so there seems to be 9 seconds of spin up time or so. For all the other katas I've looked at, the tests run in ms so no issues, but any kata that pushes into a few seconds for the test will be hitting that limit once you add in the spool up time. And this kata author specifically really tried to crank up the ante to use as much of that time as possible.

kazk commented 4 years ago

The maximum allowed time was increased to 16s.

bdw429s commented 4 years ago

Boom! My Longest Palindromic Substring (Linear) CF translation is humming now.