Closed CaptainVincent closed 11 months ago
For the first issue of providing support for the Vec<&str>
argument to thePluginManager::nn_preload
API, implement the FromStr
trait for NNPreload
so as to use the following code:
use std::str::FromStr;
PluginManager::nn_preload(vec![NNPreload::from_str("default:GGML:CPU:llama-2-7b-chat.Q5_K_M.gguf")?]);
nn_preload
API as the C-API's.
In the current implementation, the order of nn_preload models should occur after loading wasi_nn plugin and before triggering the registration of wasi_nn plugin (At the very least, under the auto-detect plugin mechanism, these are two distinct time points).
This call sequence is not make sense to the end user of the API, but they must be aware of this order; otherwise, it will fail. Perhaps we can consider a similar approach to
WASMEDGE_PLUGIN_PATH
ex.WASMEDGE_WASINN_PRELOAD
, allowing the SDK to manage the timing of calling preload.Additionally, at the SDK layer crate, I think we need a helper function to parse a string pattern into enums (backend, target). This function would be useful for setting up models without user set model by code, as users often need to provide configuration through string as inputs.