SudharakaP
commented
4 years ago More detailed exception;
Open SudharakaP opened 4 years ago
SudharakaP
commented
4 years ago More detailed exception;
SudharakaP
commented
4 years ago I think the task queue is being exhausted here;
@pascalgrimaud @jdubois : Given the large number of stat entries in one particular instance; I wonder if this is some kinda DDOS attack on the stat endpoint? 🤔
pascalgrimaud
commented
4 years ago we should have an api to enable/disable the statistics. It should help us a lot
somayaj
commented
3 years ago @pascalgrimaud would changing to an SimpleAsyncTaskExecutor() be an option?
@Bean(name = "statisticsExecutor") public Executor getStatisticsExecutor() { return new SyncTaskExecutor(); } }
jdubois
commented
3 years ago Let me check the code first. I believe this is also used to run the generator, so there might be an impact.
jdubois
commented
3 years ago So someone already fixed the question I had, as we have 2 different task executors, one for the generator, and another one for the stats service. So this is all good to change! I know the stats service isn't very well coded, so yes there could be some kind of DDOS attack there. In some way, that limit of 1000 tasks removes that DDOS attack as it doesn't accept more requests - and we'll lose some, but those are just stats, so that's not very important. We should rather fix the way the stats are being managed to resolve the original error. Otherwise I'm fine with the way it works, we can lose some stats.
somayaj
commented
3 years ago @jdubois how do you replicate this though to even fix it?
jdubois
commented
3 years ago I don’t know why the task is full: it might not even be our fault, I see in the stats that many people fork and hack the project (typically for internal usage inside their company, which is awesome), and many of those people send us back statistics without understanding what they do (we have lots of stats for options that do not exist …). So it might be one of those users who is DDOS’ing the queue, without any malicious intention. You could replicate by reducing the queue size to 1, it will be filled up very fast.
It was noticed that the following
TaskRejectedExceptionoccurs from time to time in our production environment. And today (Sep 05th 2020) the number of occurances spiked (around 4,140) for some reason.TaskRejectedException
``` org.springframework.core.task.TaskRejectedException: Executor [io.github.jhipster.async.ExceptionHandlingAsyncTaskExecutor@2f58f492] did not accept task: org.springframework.aop.interceptor.AsyncExecutionInterceptor$$Lambda$1350/0x0000000100e94440@7422504e at org.springframework.core.task.support.TaskExecutorAdapter.submit(TaskExecutorAdapter.java:131) at org.springframework.aop.interceptor.AsyncExecutionAspectSupport.doSubmit(AsyncExecutionAspectSupport.java:290) at org.springframework.aop.interceptor.AsyncExecutionInterceptor.invoke(AsyncExecutionInterceptor.java:129) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689) at io.github.jhipster.online.service.StatisticsService$$EnhancerBySpringCGLIB$$6b01bb35.addEntityStats(