search

cloudviz / agentless-system-crawler

A tool to crawl systems like crawlers for the web
Apache License 2.0
117 stars 44 forks source link

Crawler deployment on k8s GPU tainted nodes #325

Closed nadgowdas closed 7 years ago

nadgowdas commented 7 years ago

Description

In kubernetes/armada environment, gpu nodes are going to be tainted to restrict random pods being schedule on them. We need to add toleration annotation in the deployment manifest (pod specs) to enable crawler deployment on these gpu tainted nodes.

Currently armada cluster is using k8s v1.5.x, so we need to enable this annotation in the backward format. 

annotations:
        scheduler.alpha.kubernetes.io/tolerations: '[{"key":"dedicated","value":"gpu-task","effect":"NoSchedule"}]'

once the server version is upgraded to v1.6 we need to change it to new pod.spec format:

      tolerations:
      - key: "dedicated"
        operator: "Equal"
        value: "gpu-task"
        effect: "NoSchedule"
nadgowdas commented 7 years ago

PR #327 merged. Closing this issue.