Removing namespace annotation does not remove IP from filtermap.

Description

When removing a namespace annotation, the corresponding IP is not removed from the filtermap, leading to the continuous generation of metrics.

Steps to Reproduce

Create pods in the namespace.
Make sure Retina is installed with enabled pod level and annotations set to true in the config.
Annotate the namespace.
Generate traffic between pods in ns using something like Kappinger.
Confirm that the filtermap is updated and metrics are being generated.
Remove the namespace annotation.
Check the filtermap. The metrics are still being generated even after removal of the annotation.

Expected Behavior

Upon removal of the namespace annotation, the associated IP should be removed from the filtermap, and metrics generation should cease.

Actual Behavior

Metrics continue to be generated after removing the namespace annotation. Reconciliation has been observed in the namespace controller, with no apparent errors.

Additional Information

Here is some logs of an automated test. Manual test on single ns or pod should produce same results.

Annotated a ns

Annotating namespace {"namespace": "test-drops-annotation-metrics-1696500004"}

Confirmed it was annotated

Annotated namespaces before removal {"annotatedns": [{"metadata":{"name":"test-drops-annotation-metrics-1696500004","uid":"643a065c-3915-4b2d-9636-a2e8f624ff6c","resourceVersion":"5093791","creationTimestamp":"2023-10-06T21:23:08Z","labels":{"e2e":"true","kubernetes.io/metadata.name":"test-drops-annotation-metrics-1696500004"},"annotations":{"retina.io/v1alpha1":"observe"},"managedFields":[{"manager":"dropreason.test","operation":"Update","apiVersion":"v1","time":"2023-10-06T21:23:27Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:retina.io/v1alpha1":{}},"f:labels":{".":{},"f:e2e":{},"f:kubernetes.io/metadata.name":{}}}}}]},"spec":{"finalizers":["kubernetes"]},"status":{"phase":"Active"}}]}

Removed annotation and confirmed it was removed.

Annotated namespaces after removal {"annotatedns": null}

Metrics still being generated. drop packet {"labels": {"Metric":"networkobservability_adv_drop_count","Labels":["direction","egress","reason","IPTABLE_RULE_DROP","ip","","namespace","test-drops-annotation-metrics-","podname","client","workloadKind","","workloadName",""]}, "value": 33}

Pod logs

After removing the namespace, the log from the name reconcile is present showing that the namespace has been removed.

https://github.com/microsoft/retina/blob/ac26b1d865eff2451eac964b68a7838c31d3b7e8/pkg/controllers/daemon/namespace/namespace_controller.go#L60

2023-10-06T21:23:47.630Z info NamespaceReconciler namespace/namespace_controller.go:60 Namespace does not have annotation {"namespace": "test-drops-annotation-metrics-1696500004", "annotations": null}

metrics still being generated.

2023-10-06T21:23:47.860Z        debug   MetricModule.dropreason-metricsmodule   metrics/drops.go:160    drop count metric is added in EGRESS in local ctx       {"labels": ["IPTABLE_RULE_DROP", "egress", "10.224.0.32", "test-drops-annotation-metrics-1696500004", "client", "unknown", "unknown"]}
2023-10-06T21:23:47.860Z        debug   MetricModule.dropreason-metricsmodule   metrics/drops.go:160    drop count metric is added in EGRESS in local ctx       {"labels": ["IPTABLE_RULE_DROP", "egress", "10.224.0.32", "test-drops-annotation-metrics-1696500004", "client", "unknown", "unknown"]}
2023-10-06T21:23:47.862Z        debug   MetricModule.dropreason-metricsmodule   metrics/drops.go:160    drop count metric is added in EGRESS in local ctx       {"labels": ["IPTABLE_RULE_DROP", "egress", "10.224.0.32", "test-drops-annotation-metrics-1696500004", "client", "unknown", "unknown"]}

Cache finding pod IP and enriching it

 Cache   cache/cache.go:155      pod found for IP        {"ip": "10.224.0.32", "pod Name": "test-drops-annotation-metrics-1696500004/client"}
2023-10-06T21:23:53.403Z        debug   enricher        enricher/enricher.go:132        enriched flow   {"flow": "time:{seconds:965422847940441}  verdict:DROPPED  IP:{source:\"10.224.0.32\"  destination:\"10.224.0.62\"  ipVersion:IPv4}  l4:{TCP:{source_port:61582  destination_port:20480}}  source:{namespace:\"test-drops-annotation-metrics-1696500004\"  labels:\"pod=client\"  pod_name:\"client\"}  traffic_direction:INGRESS  trace_observation_point:TO_HOST  extensions:{[type.googleapis.com/utils.RetinaMetadata]:{bytes:60}}"}
2023-10-06T21:23:53.403Z        debug   MetricModule    metrics/forward.go:160  forward count metric in EGRESS in local ctx     {"labels": ["egress", "10.224.0.32", "test-drops-annotation-metrics-1696500004", "client", "unknown", "unknown"]}

cc: @jimassa @anubhabMajumdar

microsoft / retina