search

faa-swim / fns-client

The System Wide Information Service (SWIM) Federal NOTAM System (FNS) Java Messaging Service (JMS) Reference Implementation (FnsClient) provides an example implementation on how to establish and maintain a local instance of the FNS NOTAM Database through the use of the FNS Initial Load (FIL) and SWIM FNS JMS services. FIL provides all active NOTAMS, via SFTP, that is required to initialize a NOTAM database and the SWIM JMS service provides, via JMS, NOTAM updates to keep the NOTAM database current. FIL also provides for re-initialization of a NOTAM database in the case of JMS service interruption.
Apache License 2.0
12 stars 6 forks source link

NotamDb initalization failed due to missed message identified during initalization process #3

Closed rhishirajnema closed 3 years ago

rhishirajnema commented 3 years ago

We keep getting this error while running FnsClient in our servers. Any way to resolve it? Here is complete stack trace -

java.lang.Exception NotamDb initalization failed due to missed message identified during initalization process. LogbackBugsnagAppender.java:71 bugsnag.soschat.LogbackBugsnagAppender.extractThrowable LogbackBugsnagAppender.java:42 bugsnag.soschat.LogbackBugsnagAppender.append LogbackBugsnagAppender.java:18 bugsnag.soschat.LogbackBugsnagAppender.append AppenderBase.java:82 ch.qos.logback.core.AppenderBase.doAppend AppenderAttachableImpl.java:51 ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders Logger.java:270 ch.qos.logback.classic.Logger.appendLoopOnAppenders Logger.java:257 ch.qos.logback.classic.Logger.callAppenders Logger.java:421 ch.qos.logback.classic.Logger.buildLoggingEventAndAppend Logger.java:383 ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus Logger.java:522 ch.qos.logback.classic.Logger.error FnsClient.java:198 com.soschat.notam.source.fns.FnsClient.initalizeNotamDbFromFil FnsClient.java:366 com.soschat.notam.source.fns.FnsClient.start NotamAppListener.java:27 com.soschat.notam.source.configuration.NotamAppListener.onApplicationEvent NotamAppListener.java:14 com.soschat.notam.source.configuration.NotamAppListener.onApplicationEvent SimpleApplicationEventMulticaster.java:172 org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener SimpleApplicationEventMulticaster.java:165 org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener SimpleApplicationEventMulticaster.java:139 org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent AbstractApplicationContext.java:402 org.springframework.context.support.AbstractApplicationContext.publishEvent AbstractApplicationContext.java:408 org.springframework.context.support.AbstractApplicationContext.publishEvent AbstractApplicationContext.java:359 org.springframework.context.support.AbstractApplicationContext.publishEvent EventPublishingRunListener.java:105 org.springframework.boot.context.event.EventPublishingRunListener.running SpringApplicationRunListeners.java:78 org.springframework.boot.SpringApplicationRunListeners.running SpringApplication.java:332 org.springframework.boot.SpringApplication.run SpringApplicationBuilder.java:139 org.springframework.boot.builder.SpringApplicationBuilder.run DefaultBinderFactory.java:250 org.springframework.cloud.stream.binder.DefaultBinderFactory.getBinderInstance DefaultBinderFactory.java:174 org.springframework.cloud.stream.binder.DefaultBinderFactory.doGetBinder DefaultBinderFactory.java:120 org.springframework.cloud.stream.binder.DefaultBinderFactory.getBinder BindingService.java:313 org.springframework.cloud.stream.binding.BindingService.getBinder BindingService.java:214 org.springframework.cloud.stream.binding.BindingService.bindProducer BindableProxyFactory.java:262 org.springframework.cloud.stream.binding.BindableProxyFactory.createAndBindOutputs OutputBindingLifecycle.java:55 org.springframework.cloud.stream.binding.OutputBindingLifecycle.doStartWithBindable LinkedHashMap.java:608 java.util.LinkedHashMap$LinkedValues.forEach AbstractBindingLifecycle.java:47 org.springframework.cloud.stream.binding.AbstractBindingLifecycle.start OutputBindingLifecycle.java:34 org.springframework.cloud.stream.binding.OutputBindingLifecycle.start DefaultLifecycleProcessor.java:182 org.springframework.context.support.DefaultLifecycleProcessor.doStart DefaultLifecycleProcessor.java:53 org.springframework.context.support.DefaultLifecycleProcessor.access$200 DefaultLifecycleProcessor.java:360 org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start DefaultLifecycleProcessor.java:158 org.springframework.context.support.DefaultLifecycleProcessor.startBeans DefaultLifecycleProcessor.java:122 org.springframework.context.support.DefaultLifecycleProcessor.onRefresh AbstractApplicationContext.java:893 org.springframework.context.support.AbstractApplicationContext.finishRefresh ServletWebServerApplicationContext.java:163 org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.finishRefresh AbstractApplicationContext.java:552 org.springframework.context.support.AbstractApplicationContext.refresh ServletWebServerApplicationContext.java:142 org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh SpringApplication.java:775 org.springframework.boot.SpringApplication.refresh SpringApplication.java:397 org.springframework.boot.SpringApplication.refreshContext SpringApplication.java:316 org.springframework.boot.SpringApplication.run SpringApplication.java:1260 org.springframework.boot.SpringApplication.run SpringApplication.java:1248 org.springframework.boot.SpringApplication.run SpringNotamSourceApp.java:14 com.soschat.notam.source.SpringNotamSourceApp.main NativeMethodAccessorImpl.java:-2 jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 NativeMethodAccessorImpl.java:62 jdk.internal.reflect.NativeMethodAccessorImpl.invoke DelegatingMethodAccessorImpl.java:43 jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke Method.java:566 java.lang.reflect.Method.invoke MainMethodRunner.java:48 org.springframework.boot.loader.MainMethodRunner.run Launcher.java:87 org.springframework.boot.loader.Launcher.launch Launcher.java:50 org.springframework.boot.loader.Launcher.launch JarLauncher.java:51 org.springframework.boot.loader.JarLauncher.main

armFAActr commented 3 years ago

When was the last pull performed for the code? A bug was fixed that would prevent reinitialization processes to start if a message was missed during initialization. Please ensure you have the most recent code version and follow up if the issue still persists. Also, when posting about missed messages, please include the time the message was missed.

rhishirajnema commented 3 years ago

Let me take the latest and rebuild our service.

rhishirajnema commented 3 years ago

@armFAActr I updated to latest code and here is what I see in logs, it keeps invalidating notam db and reinitialize it because of missed messages -

2021-04-12 15:27:04,167 INFO - dev-76 notam-source - - : - FnsClient.initalizeNotamDbFromFil(187) | NotamDb initalized 2021-04-12 15:27:04,168 WARN - dev-76 notam-source - - : - FnsClient$2.onMissed(93) | Missed Message Identified, setting NotamDb to Invalid and ReInitalizing from FNS Initial Load | Missed Messages {3883032:2021-04-12T19:21:49.780864Z, 3883023:2021-04-12T19:21:49.780864Z, 3883036:2021-04-12T19:21:49.780864Z, 3883024:2021-04-12T19:21:49.780864Z, 3883015:2021-04-12T19:21:46.931346Z, 3883014:2021-04-12T19:21:46.931346Z, 3883029:2021-04-12T19:21:49.780864Z, 3883013:2021-04-12T19:21:46.931346Z, 3883028:2021-04-12T19:21:49.780864Z

armFAActr commented 3 years ago

When you look at your SCDS metrics page, are there any discarded messages shown during the time after connecting?

rhishirajnema commented 3 years ago

@armFAActr where I can see that, this is what I see in metrics page -

Screen Shot 2021-04-14 at 4 56 22 PM
rhishirajnema commented 3 years ago

also, it takes forever to get notam db initialized. Why there is so much delay? it keeps printing this - Waiting for New FNS Initial Load File, current file modified time: Wed Apr 14 16:51:02 EDT 2021

armFAActr commented 3 years ago

Please check and ensure you do not have any filters on your SCDS subscription. fns-client requires all notmas messages to function.

armFAActr commented 3 years ago

also, it takes forever to get notam db initialized. Why there is so much delay? it keeps printing this - Waiting for New FNS Initial Load File, current file modified time: Wed Apr 14 16:51:02 EDT 2021

the FIL file is updated every 3 minutes at the most and can be longer if the processes of generating the FIL file takes longer. During initialization the fns-client checks for a FIL file that is updated after the initialization processes starts to ensure no notams are missed between the last time FIL was updated and when the initialization processes started.

rhishirajnema commented 3 years ago

Please check and ensure you do not have any filters on your SCDS subscription. fns-client requires all notmas messages to function.

@armFAActr Doesn't look like that. This is what we have -

Screen Shot 2021-04-16 at 2 04 03 PM
armFAActr commented 3 years ago

I did notice in the graph you sent showed it going from one to two connection. Did you run the client twice? Make sure no other clients are connected and try to rerun the test tool sent previously and let me know if you still see missed messages. SCDS uses queues which don't equally distribute messages to multiple clients. If you need to run two clients then you will need to creat another subscription.

armFAActr commented 3 years ago

@rhishirajnema, did you see my previous response?

rhishirajnema commented 3 years ago

@armFAActr yes, thanks! We are going to try that, the problem is we want to run two instances for higher availability and we use common config file for both instances. We are trying to figure out how to accomplish it with two subscription. I will keep you posted.

armFAActr commented 3 years ago

@rhishirajnema please reach out to scds@faa.gov if you need assistance on how to create another subscriptions.