Initialize local CPU server if remote server not available

kpedro88 commented 4 years ago

Initializing a local CPU server when the requested remote server is not available can serve as a fallback option. Open questions:

At what stage should checking availability and initializing the local server take place? (python config, C++ module initialization, ....)
How to control # threads used by the local server? see https://github.com/triton-inference-server/server/issues/2018

If a local CPU server is used, SonicTriton clients should be forced to use Sync mode, to prevent contention between CMSSW and the local server (otherwise they would both try to use the same threads).

Assigned to: @kpedro88

kpedro88 commented 3 years ago

See https://github.com/cms-sw/cmssw/compare/master...kpedro88:TritonService for a functioning draft version of TritonService with automatic fallback support. Notes:

currently based on CMSSW_11_2_0_pre9
still potentially a work in progress, so the branch may be updated at any moment - don't get attached
all other ongoing SONIC-related PRs and branches will have to be rebased on this once it is merged (some API changes), but let's not worry about that yet

kpedro88 commented 3 years ago

draft PR now submitted: https://github.com/cms-sw/cmssw/pull/32576

kpedro88 commented 3 years ago

minor followup: https://github.com/cms-sw/cmssw/pull/32861

fastmachinelearning / SonicCMS

Initialize local CPU server if remote server not available #12