[FIX] Must validate ENV settings or wrong gpu selected by nvidia-smi

[Qubitium]

nvidia-smi uses PCI_BUS_ID order but python program may be launched using default which is not PCI_BUS_ID order for gpu. If the env values do not match, wrong gpu is returned for gpu_id. Validate the env and raise error if issue exists.

TESTS

• [x] gpu_id in range
• [x] gpu_id out of range
• [x] gpu_id with multi-gpu + no PCI_BUS_ID order
• [x] gpu_id with multi-gpu + PCI_BUS_ID order set

[Qubitium]

@microsoft-github-policy-service agree

@microsoft-github-policy-service agree

[Qubitium]

@LeiWang1999 Ready for review. The CUDA order ENV must be validated (match nvidia-smi) in multi-gpu env or we get the wrong gpu back.