enabling secure-forward on release-1.5 causes 'project not found' error in kibana

[stephenrlouie]

I'm using release-1.5 via openshift-ansible/playbook/byo/openshit-cluster/openshift-logging.yaml. I have modified the secure forward file to insecurely forward its logs to a fluentd aggregator I've set up.

I run the ansible playbook to install EFK with my modified forward file and have seen logs reach my aggregator. The issue is the host that has the forwarding on, no longer has projects show up in kibana. I see project..empty-project.* with no logs making it to Kibana. It also displays some errors saying

project.<my_project_name>.<HASH> is not a configured pattern. User default pattern.....

My understanding was updating the secure_forward file would be independent and not effect what is specified in the output-es-config. Any help would be great, I'll continue to poke around.

[jcantrill]

@richm or @nhosoi can you identify what we should be looking for in the fluentd pod so we know its working as expected

[richm]

oc exec $fluentd_pod -- cat /etc/fluent/fluent.conf oc exec $fluentd_pod -- ls -alrtF /etc/fluent/configs.d/openshift oc exec $fluentd_pod -- ls -alrtF /etc/fluent/configs.d/user oc exec $fluentd_pod -- cat /etc/fluent/configs.d/user/secure_forward.conf

[nhosoi]

How about logs from the fluentd and elesticsearch pod? oc logs $fluentd_pod oc logs $elasticsearch_pod

[jcantrill]

closing without prejudice