Minor improvements on making torch optional.

[jinjingforever]

• Throw error when add model from pytorch through config.
• Don't load the pytorch adapter when torch is not available.