Julia threads and HiGHS `threads` interaction

[bachdavi]

Hello again :)

I have a question about the interaction between Julia threads and the HiGHS threads option. It appears that HiGHS initializes itself for each Julia thread where I'm optimizing a model. Is that intended behavior?

Some background: Via gdb we have recently observed a large number of logical threads associated with HiGHS (HighsTaskExecutor::HighsTaskExecutor). We are solving a linear program that via the documentation does not use any parallelism. By default, HiGHS will initialize its scheduler with half the machine threads, which we want to avoid. HiGHS supports the threads option, which we can set via MOI.RawOptimizerAttribute("threads") or MOI.NumberOfThreads(). The documentation mentioned that this is a global setting and can only be changed after calling resetGlobalScheduler.

I can set the number of threads to 1 and can observe the expected increase of threads in the process. Here is a simple example:

using JuMP, HiGHS, MathOptInterface; begin 
           model = Model(HiGHS.Optimizer)
           MathOptInterface.set(model, MathOptInterface.NumberOfThreads(), 1)
           optimize!(model)
end

If I change 1 to 2, HiGHS mentions the need to reset the global scheduler.

But if I run the above snippet, with the changed threads count, in another Julia thread, via Threads.@spawn HiGHS sets the number of threads, apparently initializing a new scheduler. I can also observe an increase in the thread count of the Julia process.

Thank you very much for helping us out!

[odow]

See https://github.com/jump-dev/HiGHS.jl/issues/130#issuecomment-1289874281

[odow]

@bachdavi is https://github.com/jump-dev/HiGHS.jl/pull/182 helpful?

[chriscoey]

are there any changes we ought to suggest to the HiGHS devs directly?

[odow]

They're aware of the issue. I don't know if there is the desire or engineering time to change it.

[bachdavi]

@odow Sorry for taking such a long time to answer. I was a little swamped with work! Thank you for clarifying the Readme, I think that is great ❤️ I'm wondering what to make of the behavior of the HiGHS scheduler related to Julia threads. It appears that on each Julia thread where I'm running the HiGHS optimizer we are initializing the scheduler again. Looking at the ps output it adds half the machine's threads to the thread count of the Julia process for each unique Julia thread I'm using the solver in.

[odow]

cc @jajhall is there a way to limit HiGHS to one thread before we initialize it?

(@bachdavi Julian is traveling for the next week or two, so I don't know when we should expect a reply.)

[jajhall]

Yes, set the option 'threads' to 1. There should be little noticeable performance degradation as parallelism is only used in symmetry detection in the MIP solver.

If this is done before a call to 'Highs::run()' then the scheduler is initialised to use one thread.

The default value for 'threads' of 0 results in half the possible machine threads being initialised for use by HiGHS

[odow]

Hmm that's what @bachdavi had tried to do. I'll take a look at this next week.

[bachdavi]

@odow and @jajhall To clarify, I can set the number of threads to 1 and it works properly. I do have to do that on each Julia thread though.

I'm running the following in an __init__:

function __init__()
    _g_set_highs_threads!()
end

# Set the number of threads used by HiGHS _globally_ to `1`.
function _g_set_highs_threads!()
    Threads.@threads :static for i in 1:Threads.threadpoolsize()
        _set_highs_threads!()
    end
end

# Set the number of threads used by HiGHS to `1`. This applies to the
# current _Julia_ thread only!
function _set_highs_threads!()
    # Reset, otherwise we cannot change the number of threads.
    HiGHS.Highs_resetGlobalScheduler(1)

    model = Model(HiGHS.Optimizer; add_bridges = false)

    set_silent(model)

    # HiGHS will by default initialize its task executor with half the
    # available threads on a machine, but won't use them to solve LPs:
    # https://ergo-code.github.io/HiGHS/dev/parallel/. Set the number
    # of threads to 1 to avoid it. This is a _Julia_ thread-local
    # setting and can only be changed after calling
    # `HiGHS.Highs_resetGlobalScheduler`.
    MathOptInterface.set(model, MathOptInterface.RawOptimizerAttribute("threads"), 1)

    # We have to call `optimize!` to apply the setting above.
    optimize!(model)
end

I'm wondering if that is intended or expected :)

[odow]

@bachdavi what about something like this:

function __init__()
    _set_all_highs_threads_to_serial()
    return
end

function _set_all_highs_threads_to_serial()
    HiGHS.Highs_resetGlobalScheduler(1)
    Threads.@threads :static for _ in 1:Threads.threadpoolsize()
        _set_thread_local_highs_threads_to_serial()
    end
    return
end

"""
    _set_thread_local_highs_threads_to_serial()

Set the number of threads used by HiGHS to `1`. This applies to the current
_Julia_ thread only!

HiGHS will by default initialize its task executor with half the available
threads on a machine, but won't use them to solve LPs:
https://ergo-code.github.io/HiGHS/dev/parallel/. 

Set the number of threads to 1 to avoid it. This is a _Julia_ thread-local
setting and can only be changed after calling `Highs_resetGlobalScheduler`.
"""
function _set_thread_local_highs_threads_to_serial()
    highs = HiGHS.Highs_create()
    HiGHS.Highs_setIntOptionValue(highs, "threads", 1)
    HiGHS.Highs_destroy(highs)
    return
end

[bachdavi]

@odow I tried your suggestion, but I think the threads option is only applied when presolve is executed: https://github.com/ERGO-Code/HiGHS/blob/77583e20470b2ed5dad05a8eb04511034710ddd2/src/lp_data/Highs.cpp#L708-L737

In any case, I think #182 is a good addition to the README.

I'll close this issue, thank you :)