Broken Image v1.0.0

[ryan-dyer]

The v1.0.0 image appears to have been rebuilt 2 days ago and no longer works with existing k8s manifests. Can you please revert the image to its original state and increment the version # if that is the intent.

[ryan-dyer]

Also even attempting to use the new image doesnt appear to work. My metadata is defined as:

service:
  default:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "created_by=kubernetes,expires=never,Environment=develop,Application=kube,Role=admin,Department=engops,System=develop-us-east-1-admin"
persistentVolumeClaim:
  default:
    annotations:
      ebs-tagger.kubernetes.io/ebs-additional-resource-tags: "created_by=kubernetes,expires=never,Environment=develop,Application=kube,Role=admin,Department=engops,System=develop-us-east-1-admin"

However I do not see these tags on the EBS volumes or ELBs.

In the log file for the k8s-metadata-injector I see:

W0723 12:40:25.284041       1 client_config.go:549] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0723 12:40:25.284940       1 controller.go:99] Starting ebs-tagger controller
I0723 12:40:25.284956       1 controller.go:105] Waiting for informer caches to sync
I0723 12:40:25.485225       1 controller.go:113] Starting workers
I0723 12:40:25.485353       1 controller.go:145] Error processing {pvc-3dea8e32-acb0-11e9-a9e9-124a01ac1d02 CREATE} (will retry): Invalid annotation "":
I0723 12:40:25.485379       1 controller.go:145] Error processing {pvc-405b99e1-acb0-11e9-a9e9-124a01ac1d02 CREATE} (will retry): Invalid annotation "":
I0723 12:40:25.485388       1 controller.go:145] Error processing {pvc-37ed600d-acb0-11e9-a9e9-124a01ac1d02 CREATE} (will retry): Invalid annotation "":
I0723 12:40:25.485542       1 controller.go:145] Error processing {pvc-86c70ecd-acb0-11e9-a9e9-124a01ac1d02 CREATE} (will retry): Invalid annotation "":
I0723 12:40:25.485559       1 controller.go:145] Error processing {pvc-a5965462-ac9c-11e9-a9e9-124a01ac1d02 CREATE} (will retry): Invalid annotation "":
I0723 12:40:25.485570       1 controller.go:145] Error processing {pvc-a17adf60-ac9c-11e9-a9e9-124a01ac1d02 CREATE} (will retry): Invalid annotation "":
I0723 12:40:25.485579       1 controller.go:145] Error processing {pvc-a59138b6-ac9c-11e9-a9e9-124a01ac1d02 CREATE} (will retry): Invalid annotation "":
I0723 12:40:25.485473       1 controller.go:145] Error processing {pvc-3f231043-acb0-11e9-a9e9-124a01ac1d02 CREATE} (will retry): Invalid annotation "":

I have looked at the pvc and pv for these and do not see any empty annotations.

[almariah]

@ryan-dyer Could you try to pull it again. Like redeployment of k8s-metadata-injector. Further I would recommend you to use v2.0.0. Be aware of some changes in v2.0.0 like grouping the config by namespaces, using default config "*" and -ebs-tagging=true

[ryan-dyer]

v1 still fails. the command line arguments are different. Yes I want to try v2, as it has the default changes which we desire, just have not had time yet.

[ryan-dyer]

Also have tried v2 now and receive same error as posted above.

W0723 15:06:35.966079       1 client_config.go:549] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0723 15:06:35.966937       1 controller.go:99] Starting ebs-tagger controller
I0723 15:06:35.966953       1 controller.go:105] Waiting for informer caches to sync
I0723 15:06:36.167216       1 controller.go:113] Starting workers
I0723 15:06:36.167370       1 controller.go:145] Error processing {pvc-37ed600d-acb0-11e9-a9e9-124a01ac1d02 CREATE} (will retry): Invalid annotation "":
I0723 15:06:36.167391       1 controller.go:145] Error processing {pvc-3f231043-acb0-11e9-a9e9-124a01ac1d02 CREATE} (will retry): Invalid annotation "":
I0723 15:06:36.167433       1 controller.go:145] Error processing {pvc-86c70ecd-acb0-11e9-a9e9-124a01ac1d02 CREATE} (will retry): Invalid annotation "":
I0723 15:06:36.167446       1 controller.go:145] Error processing {pvc-a5965462-ac9c-11e9-a9e9-124a01ac1d02 CREATE} (will retry): Invalid annotation "":
I0723 15:06:36.167493       1 controller.go:145] Error processing {pvc-a59138b6-ac9c-11e9-a9e9-124a01ac1d02 CREATE} (will retry): Invalid annotation "":
I0723 15:06:36.167506       1 controller.go:145] Error processing {pvc-3dea8e32-acb0-11e9-a9e9-124a01ac1d02 CREATE} (will retry): Invalid annotation "":
I0723 15:06:36.167515       1 controller.go:145] Error processing {pvc-405b99e1-acb0-11e9-a9e9-124a01ac1d02 CREATE} (will retry): Invalid annotation "":
I0723 15:06:36.167451       1 controller.go:145] Error processing {pvc-a17adf60-ac9c-11e9-a9e9-124a01ac1d02 CREATE} (will retry): Invalid annotation "":
I0723 15:06:36.167926       1 webhook.go:103] Starting the k8s-metadata-injector admission webhook server
I0723 15:06:36.172288       1 webhook-register.go:59] Updating existing MutatingWebhookConfiguration for the k8s-metadata-injector admission webhook

[almariah]

@ryan-dyer could you describe one of these volumes (pvc)?

[ryan-dyer]

Namespace:     es
StorageClass:  gp2
Status:        Bound
Volume:        pvc-37ed600d-acb0-11e9-a9e9-124a01ac1d02
Labels:        app=logstash
               release=logstash
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      2Gi
Access Modes:  RWO
Events:        <none>
Mounted By:    logstash-0

[almariah]

It seems that you did't configure the metadata config file for namespace es. That's why it complains about empty tag "", which is a bug. I will fix it. Further after fixing, it should't tag these volumes because they don't have the tag created by you or the metadata injector (config file).

[ryan-dyer]

For v2 I only configured the '*' namespace.

 k exec -it k8s-metadata-injector-7d86d9c7cb-c4dw2 sh
/ # ps -ef
PID   USER     TIME  COMMAND
    1 root      0:22 /usr/bin/k8s-metadata-injector -logtostderr=true -v=2 -ebs-tagging=true
   21 root      0:00 sh
   28 root      0:00 ps -ef
/ # find / -name metadataconfig.yaml
/etc/webhook/config/metadataconfig.yaml
/etc/webhook/config/..2019_07_23_15_06_35.795399135/metadataconfig.yaml
/ # cd /etc/webhook/config/
/etc/webhook/config # cat metadataconfig.yaml
namespaces:
  "*":
    service:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "created_by=kubernetes,expires=never,Environment=develop,Application=kube,Role=admin,Department=engops,System=develop-us-east-1-admin"
    persistentVolumeClaim:
      annotations:
        ebs-tagger.kubernetes.io/ebs-additional-resource-tags: "created_by=kubernetes,expires=never,Environment=develop,Application=kube,Role=admin,Department=engops,System=develop-us-east-1-admin"

[almariah]

Fixed! I would recommend using v2.0.0 after reading the docs again. There are few changes.

[ryan-dyer]

I'm no longer getting the error messages, but I am still not getting tags on my EBS volumes. I have read the documentation and updated my metadataconfig.yaml file accordingly as well as added the -ebs-tagging option. This is seen in the code block in my previous comment. What else am I missing?

[almariah]

You should see some logs after k8s-meatadata starting telling you about what is tagged or not! did you get any of them?

controller.go:232] Tags created for EBS .............

[almariah]

Further if metadata injector is installed after PVC are already deployed, the metadata will not be injected. It is working when you create or update your resource (provided that k8-metadata-injector is already deployed). So try to update any PVC, by adding any label or annotation manually. Using for example:

kubectl edit pvc ...... or even redeploy them if possible and then you should see tags are added.

[ryan-dyer]

Ok got v2 working now by updating the labels. Any thoughts on a bootstrap-like job which scans all objects and updates them accordingly during the apps bootup?