Closed gbbirkisson closed 4 years ago
Initialazers are pure classes that get executed in OSGi activate methods of components that need them "most".
The drawback of this approach is when other components require services provided and initialization takes long Felix (OSGi framework) times out.
One way to fix is to increase timeout ds.lock.timeout.milliseconds
, but it is not very robust solution
We may need to extract initialization in own components and specify immediate
on them, and use CompletableFuture
ApplicationResource -> ApplicationService -> ApplicationRepoService -> NodeService -> RepositoryService -> SystemRepoInitializer in Activate
FilterPipelineImpl -> Filter (BasicAuthFilter) -> BasicAuthFilter -> SecurityService -> SecurityInitializer in Activate
When spinning up a cluster, with multiple nodes simultaneously for the first time, the elected master has to initialize the repos. If for some reason that process is slower then expected (probably slow disk in this case), some nodes might timeout waiting, try to start essential services and error out leaving them impaired or dead.
Elected master (master-2)
Dead node 1 (master-1)
Dead node 2 (backend-0)