use of external cloud provider: deprecated cloud-provider flag in templates

[obeyler]

What happened:

Kubelet job warns that cloud providers are getting externalized through the following log:

I0527 13:20:55.174850 17526 server.go:417] Version: v1.14.1 W0527 13:20:55.175016 17526 plugins.go:118] WARNING: openstack built-in cloud provider is now deprecated. Please use 'external' cloud provider for openstack: https://github.com/kubernetes/cloud-provider-openstack

However when specifying the cloud-provider.type to external then the flag cloud-provider inside kube-api-server and kube-controller-manager is incorrectly set in current templates

https://github.com/cloudfoundry-incubator/kubo-release/blob/0b66efc0aa54deb522d0e04f16d2d2ccbbc74ed8/jobs/kube-apiserver/templates/config/bpm.yml.erb#L48-L51

https://github.com/cloudfoundry-incubator/kubo-release/blob/0b66efc0aa54deb522d0e04f16d2d2ccbbc74ed8/jobs/kube-controller-manager/templates/config/bpm.yml.erb#L26-L29

according to documentation at https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/#running-cloud-controller-manager reproduced below.

kube-apiserver and kube-controller-manager MUST NOT specify the --cloud-provider flag. This ensures that it does not run any cloud specific loops that would be run by cloud controller manager. In the future, this flag will be deprecated and removed.

I wonder whether this invalid flag may explain why my configured external openstack cloud provider fails to load

• Kubernetes version (kubectl version): 1.14.1
• Cloud provider (e.g. aws, gcp, vsphere): external => openstack

[obeyler]

Please note: that to make openstack cloud provider as external, master nodes must also not be hidden. (daemonset is collocated on master node by spec it has node selection). when external is set, every nodes are taint until daemonset pod avialability for cloud manager external. To make it run, I add kubelet on maste, with arg: register-with-taints: node-role.kubernetes.io/master=true:NoSchedule