FATAL Error: Validation schema for [opendistro_security] was already registered

[opoplawski]

We suffered a disk full condition on one of our elastic nodes. After getting that dealt with, kibana is no longer starting:

kibana[744320]: {"type":"log","@timestamp":"2020-10-28T15:17:24Z","tags":["warning","plugins-discovery"],"pid":744320,"message":"Expect plugin \"id\" in camelCase, but found: opendistro-notebooks-kibana"}
kibana[744320]: {"type":"log","@timestamp":"2020-10-28T15:17:29Z","tags":["fatal","root"],"pid":744320,"message":"Error: Validation schema for [opendistro_security] was already registered.\n    at ConfigService.setSchema (/usr/share/kibana/src/core/server/config/config_service.js:60:13)\n    at MergeMapSubscriber.plugin$.pipe [as project] (/usr/share/kibana/src/core/server/plugins/plugins_service.js:134:46)\n    at MergeMapSubscriber._tryNext (/usr/share/kibana/node_modules/rxjs/internal/operators/mergeMap.js:69:27)\n    at MergeMapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/mergeMap.js:59:18)\n    at MergeMapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at FilterSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/filter.js:52:30)\n    at FilterSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at ReplaySubject._subscribe (/usr/share/kibana/node_modules/rxjs/internal/ReplaySubject.js:76:28)\n    at ReplaySubject.Observable._trySubscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:44:25)\n    at ReplaySubject.Subject._trySubscribe (/usr/share/kibana/node_modules/rxjs/internal/Subject.js:102:51)\n    at ReplaySubject.Observable.subscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:30:22)\n    at FilterSubscriber.shareReplayOperation (/usr/share/kibana/node_modules/rxjs/internal/operators/shareReplay.js:45:32)\n    at Observable.subscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:25:31)\n    at FilterOperator.call (/usr/share/kibana/node_modules/rxjs/internal/operators/filter.js:29:23)\n    at Observable.subscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:25:31)\n    at MergeMapOperator.call (/usr/share/kibana/node_modules/rxjs/internal/operators/mergeMap.js:39:23)"}
kibana[744320]:  FATAL  Error: Validation schema for [opendistro_security] was already registered.

Any suggestions for how to resolve this? Thanks

[zengyan-amazon]

looks like more than 1 plugin declare that they own the opendistro_security.* config, can you verify installed plugins? Kibana log should show what plugins are discovered during initialization process

[opoplawski]

I set 'logging.verbose: true' in /etc/kibana/kibana.yml. Then looking at the discovered plugin messages I see:

Nov 04 07:47:16 localhost kibana[178147]: {"type":"log","@timestamp":"2020-11-04T15:47:16Z","tags":["debug","plugins-discovery"],"pid":178147,"message":"Successfully discovered plugin \"opendistroSecurity\" at \"/usr/share/kibana/plugins/.plugin.installing\""}
Nov 04 07:47:16 localhost kibana[178147]: {"type":"log","@timestamp":"2020-11-04T15:47:16Z","tags":["debug","plugins-discovery"],"pid":178147,"message":"Successfully discovered plugin \"opendistroSecurity\" at \"/usr/share/kibana/plugins/opendistro_security\""}

That appears to be the only plugin discovered in .plugin.installing. Should I just delete that directory?

[opoplawski]

Looks like this is leftover from an attempt to install an update to opendistro_security as mentioned here: https://github.com/opendistro-for-elasticsearch/security-kibana-plugin/issues/503#issuecomment-707998584

Presumably that should have cleaned up the /usr/share/kibana/plugins/.plugin.installing directory?