Cannot use the LibraryClient with the watcher

[wangchen615]

The current design of libraryClient in the release version includes watcher as follows:

// Client for Watcher APIs as a library
type libraryClient struct {
    fetcherClient watcher.MetricsProviderClient
    watcher       *watcher.Watcher
}

Watcher libraries should only be used for load-watcher. libraryClient should not include starting the watcher when the libraryClient is also used by scheduler plugins.

// Creates a new watcher client when using watcher as a library
func NewLibraryClient(opts watcher.MetricsProviderOpts) (Client, error) {
    var err error
    client := libraryClient{}
    switch opts.Name {
    case watcher.PromClientName:
        client.fetcherClient, err = metricsprovider.NewPromClient(opts)
    case watcher.SignalFxClientName:
        client.fetcherClient, err = metricsprovider.NewSignalFxClient(opts)
    default:
        client.fetcherClient, err = metricsprovider.NewMetricsServerClient()
    }
    if err != nil {
        return client, err
    }

    client.watcher = watcher.NewWatcher(client.fetcherClient)
    client.watcher.StartWatching()
    return client, nil
}

The LibraryClient failed to getLatestMetrics when using it in plugins without select {}.

[zorro786]

Issue is with fetches below from windowWatcher() which run in separate go routine:

    for _, duration := range durations {
        go windowWatcher(duration)
    }

This may cause the metrics to be unavailable for a very short moment initially when using load watcher as a library. To fix this, the first fetch should be made synchronously, and subsequent fetches should use goroutines. Thanks for reporting @wangchen615