Timing difference running with include vs timeit?

[rohith14]

Hi,

Why is there a performance difference running julia code with include("") vs timeit (MACRO defined in perfutil.jl)?

For example, in the use-case below I am seeing a 13x performance difference. I run this use-case multiple times in the same Julia session, so I am not sure if it could be a first-time cost issue

julia> include("test_wrapper.jl") # Iters: 5000 include:18.649156 msec timeit: 1.409061 msec Ratio of include/timeit: 13.235166

testname = "test.jl"
iter=5000
@printf "***# Iters: %d***\n" iter
include(testname)

test.jl

using Test

function parseintperf(t)
    local n, m
    for i=1:t
        n = rand(UInt32)
        #s = hex(n)
        s = string(n, base = 16)
        m = UInt32(parse(Int64,s, base = 16))
        @assert m == n
    end
    return n
end

execTime = 1000*(@elapsed parseintperf(iter))
@printf "include<test-name>:%f msec\n" execTime

function timeit()
    t = Float64[]
    for i = 1 : 5
        e = 1000*(@elapsed parseintperf(iter))
        if i > 0
            # warm up on first iteration
            push!(t, e)
        end
    end
    return minimum(t)
end

z = timeit()

@printf "timeit: %f msec\n" z

@printf "Ratio of include/timeit: %f\n\n" execTime/z

[johnfgibson]

The first call to parseintperf includes compilation. If you just repeat the two lines starting with execTime = ... ; @printf ... you get

julia> include("test.jl")
include<test-name>:14.303056 msec
include<test-name>:1.082944 msec
timeit: 1.054681 msec
Ratio of include/timeit: 1.026798

[rohith14]

Thanks for the quick response John. Wouldn't running with include multiple times have the same effect (in the same Julia session) ? Does it mean running with include re-compiles the code?

[johnfgibson]

You're welcome. The function gets recompiled every time time you reload the code with include("test.jl"). Doing anything else would be excessively complicated, like building a make system into the REPL.

[StefanKarpinski]

@rohith14, it might be best to ask these kinds of questions on Discourse; we can create a benchmarks tag that people can subscribe to.