Spawn worker in custom environment

[lhnwrk]

As per #1461, MLServer currently spawns worker with the same Python version as the main process, so user-provided environment is forced to match the Python version of the server. However, multiprocessing allows setting the worker's executable path, this PR extends support for spawning workers in custom environments.

[sakoush]

@lhnwrk many thanks for your contribution to add the ability to run mlserver workers on different python versions.

Internally mlserver communicates with these parallel workers using multiprocessing.Queue (check here) and therefore our implementation requires that the main process and the workers to use the same python version (or at least compatible versions) in terms of communication and serialisation. We have not tested actually what versions of python can interplay. It is unlikely though that workers can be on python 2.x for example.

Having said that, the change you propose is slightly orthogonal to the above point as there is an argument to be made that we should set multiprocessing.set_executable from the custom env anyway.

We suggest the following for us to be able to accept this change:

• Extend the unit tests so that workers can be created using different python versions to back up your change.
• Provide a list of cases as docs where this change can / cannot be applicable.

We can provide more pointers if you are happy to address the above.

cc: @jesse-c, @lc525

[lhnwrk]

AFAIK we only need the worker process to have the same pickle protocol as the main process for multiprocessing.Queue to pass serialized objects between them. Unfortunately we don't get to control what protocol multiprocessing uses, its ForkingPickler simply falls back to the default pickle protocol. Version 4 has been the default since python 3.8, so any versions since should be compatible with one another as pickle guarantees backwards compatibility.

I've updated the docs to point out the caveats for users, and extended the test suites to include custom environments with different python versions. For now they include the main process's python, the minimum tested python (3.9), and the maximum tested python (3.10) so we cover all cases of running a worker process with the same, lower, or higher python version.

Let me know if there's anything else!

[sakoush]

@lhnwrk could you rebase on top of master to make sure that tests are passing on this PR?

[lhnwrk]

@sakoush Rebased on master!