Open SudharakaP opened 4 years ago
More detailed exception;
I think the task queue is being exhausted here;
@pascalgrimaud @jdubois : Given the large number of stat entries in one particular instance; I wonder if this is some kinda DDOS attack on the stat endpoint? 🤔
we should have an api to enable/disable the statistics. It should help us a lot
@pascalgrimaud would changing to an SimpleAsyncTaskExecutor() be an option?
@Bean(name = "statisticsExecutor") public Executor getStatisticsExecutor() { return new SyncTaskExecutor(); } }
Let me check the code first. I believe this is also used to run the generator, so there might be an impact.
So someone already fixed the question I had, as we have 2 different task executors, one for the generator, and another one for the stats service. So this is all good to change! I know the stats service isn't very well coded, so yes there could be some kind of DDOS attack there. In some way, that limit of 1000 tasks removes that DDOS attack as it doesn't accept more requests - and we'll lose some, but those are just stats, so that's not very important. We should rather fix the way the stats are being managed to resolve the original error. Otherwise I'm fine with the way it works, we can lose some stats.
@jdubois how do you replicate this though to even fix it?
I don’t know why the task is full: it might not even be our fault, I see in the stats that many people fork and hack the project (typically for internal usage inside their company, which is awesome), and many of those people send us back statistics without understanding what they do (we have lots of stats for options that do not exist …). So it might be one of those users who is DDOS’ing the queue, without any malicious intention. You could replicate by reducing the queue size to 1, it will be filled up very fast.
It was noticed that the following
TaskRejectedException
occurs from time to time in our production environment. And today (Sep 05th 2020) the number of occurances spiked (around 4,140) for some reason.TaskRejectedException
``` org.springframework.core.task.TaskRejectedException: Executor [io.github.jhipster.async.ExceptionHandlingAsyncTaskExecutor@2f58f492] did not accept task: org.springframework.aop.interceptor.AsyncExecutionInterceptor$$Lambda$1350/0x0000000100e94440@7422504e at org.springframework.core.task.support.TaskExecutorAdapter.submit(TaskExecutorAdapter.java:131) at org.springframework.aop.interceptor.AsyncExecutionAspectSupport.doSubmit(AsyncExecutionAspectSupport.java:290) at org.springframework.aop.interceptor.AsyncExecutionInterceptor.invoke(AsyncExecutionInterceptor.java:129) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689) at io.github.jhipster.online.service.StatisticsService$$EnhancerBySpringCGLIB$$6b01bb35.addEntityStats(