search

exadel-inc / CompreFace

Leading free and open-source face recognition system
https://exadel.com/accelerator-showcase/compreface/
Apache License 2.0
5.67k stars 772 forks source link

Error during synchronization between servers #1060

Open zipbooksit opened 1 year ago

zipbooksit commented 1 year ago

Hello Team, It has been found that the users are facing frequent issues while using the face recognition service. For your reference we have attached the screen shot (which contains error message). Please check it and request you to provide a solution for this issue.

Issue : "Error during synchronization between servers: connect timed out executing POST http://compreface-core:3000/find_faces"

OS: ubuntu 20.04 lts

*Logs**

docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a192bb9112ff exadel/compreface-fe:1.1.0 "/docker-entrypoint.…" 6 weeks ago Up 3 weeks 0.0.0.0:8000->80/tcp, :::8000->80/tcp compreface-ui 2e1ceeff1146 exadel/compreface-admin:1.1.0 "sh -c 'java $ADMIN_…" 6 weeks ago Up 3 weeks compreface-admin 11e46ffa5fb7 exadel/compreface-api:1.1.0 "sh -c 'java $API_JA…" 6 weeks ago Up 3 weeks compreface-api 68f047b2eb0d exadel/compreface-postgres-db:1.1.0 "docker-entrypoint.s…" 6 weeks ago Up 3 weeks 5432/tcp compreface-postgres-db b24361c136c7 exadel/compreface-core:1.1.0 "uwsgi --ini uwsgi.i…" 6 weeks ago Up 3 weeks 3000/tcp compreface-core

myDockerCompose.log Screenshot_2023-03-16-10-44-48-918_in appslure kelly

xfstart07 commented 1 year ago

also

Platform: MacBook Pro M1 compreFace version: 1.1.0

Error message:

compreface-api          | 2023-04-24 02:44:14.270 ERROR 7 --- [io-8080-exec-10] c.e.f.c.h.ResponseExceptionHandler       : Defined exception occurred
compreface-api          |
compreface-api          | com.exadel.frs.commonservice.sdk.faces.exception.FacesServiceException: Error during synchronization between servers: Read timed out executing GET http://compreface-core:3000/status
compreface-api          |   at com.exadel.frs.commonservice.sdk.faces.service.FacesRestApiClient.getStatus(FacesRestApiClient.java:101)
compreface-api          |   at com.exadel.frs.commonservice.sdk.faces.service.FacesRestApiClient$$FastClassBySpringCGLIB$$517e8caf.invoke(<generated>)
compreface-api          |   at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
compreface-api          |   at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
compreface-api          |   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
compreface-api          |   at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
compreface-api          |   at org.springframework.cache.interceptor.CacheInterceptor.lambda$invoke$0(CacheInterceptor.java:54)
compreface-api          |   at org.springframework.cache.interceptor.CacheAspectSupport.invokeOperation(CacheAspectSupport.java:366)
compreface-api          |   at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:421)
compreface-api          |   at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:345)
compreface-api          |   at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:64)
compreface-api          |   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
compreface-api          |   at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
compreface-api          |   at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)
compreface-api          |   at com.exadel.frs.commonservice.sdk.faces.service.FacesRestApiClient$$EnhancerBySpringCGLIB$$3651c74b.getStatus(<generated>)
compreface-api          |   at com.exadel.frs.core.trainservice.controller.ConsistenceController.getCheckDemo(ConsistenceController.java:25)
compreface-api          |   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
compreface-api          |   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
compreface-api          |   at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
compreface-api          |   at java.base/java.lang.reflect.Method.invoke(Unknown Source)
compreface-api          |   at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
compreface-api          |   at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
compreface-api          |   at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
compreface-api          |   at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
compreface-api          |   at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
compreface-api          |   at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
compreface-api          |   at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)
compreface-api          |   at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
compreface-api          |   at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
compreface-api          |   at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
compreface-api          |   at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)
compreface-api          |   at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
compreface-api          |   at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
compreface-api          |   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
compreface-api          |   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
compreface-api          |   at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
compreface-api          |   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
compreface-api          |   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
compreface-api          |   at com.exadel.frs.core.trainservice.filter.SecurityValidationFilter.doFilter(SecurityValidationFilter.java:134)
compreface-api          |   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
compreface-api          |   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
compreface-api          |   at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
compreface-api          |   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
compreface-api          |   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
compreface-api          |   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
compreface-api          |   at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
compreface-api          |   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
compreface-api          |   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
compreface-api          |   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
compreface-api          |   at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
compreface-api          |   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
compreface-api          |   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
compreface-api          |   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
compreface-api          |   at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
compreface-api          |   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
compreface-api          |   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
compreface-api          |   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
compreface-api          |   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
compreface-api          |   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
compreface-api          |   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
compreface-api          |   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
compreface-api          |   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
compreface-api          |   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
compreface-api          |   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
compreface-api          |   at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
compreface-api          |   at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
compreface-api          |   at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890)
compreface-api          |   at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743)
compreface-api          |   at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
compreface-api          |   at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
compreface-api          |   at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
compreface-api          |   at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
compreface-api          |   at java.base/java.lang.Thread.run(Unknown Source)
compreface-api          |
compreface-ui           | 172.20.0.1 - - [24/Apr/2023:02:44:14 +0000] "GET /admin/user/roles HTTP/1.1" 200 141 "http://localhost:8000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111.0.1661.54"
pospielov commented 1 year ago

Issue : "Error during synchronization between servers: connect timed out executing POST http://compreface-core:3000/find_faces"

@zipbooksit Do I understand correctly that service works, but sometimes it returns this error? At least this is what I see from logs. Possible reason is high load. Another possible reason - you send huge image, so it can't recognize it fast. Take a look at this peace of log:

157.35.6.55 - - [15/Mar/2023:14:25:22 +0000] "POST /api/v1/verification/verify HTTP/1.1" 200 199 "-" "okhttp/3.12.0" 152.58.138.138 - - [15/Mar/2023:14:25:23 +0000] "POST /api/v1/verification/verify HTTP/1.1" 200 198 "-" "okhttp/3.12.0" 152.58.34.60 - - [15/Mar/2023:14:25:26 +0000] "POST /api/v1/verification/verify HTTP/1.1" 499 0 "-" "KellyServices/36 (com.ios.kelly; build:1; iOS 16.3.1) Alamofire/36" 106.206.196.145 - - [15/Mar/2023:14:25:26 +0000] "POST /api/v1/verification/verify HTTP/1.1" 200 197 "-" "okhttp/3.12.0" 157.48.197.243 - - [15/Mar/2023:14:25:27 +0000] "POST /api/v1/verification/verify HTTP/1.1" 200 200 "-" "okhttp/3.12.0"

I see that there are different clients. And some how client with IP 152.58.34.60 always gets an error. However, there is another log:

223.239.58.71 - - [01/Apr/2023:13:07:28 +0000] "POST /api/v1/verification/verify HTTP/1.1" 200 198 "-" "okhttp/3.12.0" 106.66.200.25 - - [01/Apr/2023:13:07:28 +0000] "POST /api/v1/verification/verify HTTP/1.1" 200 197 "-" "okhttp/3.12.0" 223.238.98.80 - - [01/Apr/2023:13:07:53 +0000] "POST /api/v1/verification/verify HTTP/1.1" 200 198 "-" "okhttp/3.12.0" 157.35.0.124 - - [01/Apr/2023:13:08:14 +0000] "POST /api/v1/verification/verify HTTP/1.1" 499 0 "-" "okhttp/3.12.0" 27.60.248.62 - - [01/Apr/2023:13:08:15 +0000] "POST /api/v1/verification/verify HTTP/1.1" 504 160 "-" "okhttp/3.12.0" 157.42.2.16 - - [01/Apr/2023:13:08:22 +0000] "POST /api/v1/verification/verify HTTP/1.1" 504 160 "-" "okhttp/3.12.0" 42.105.1.62 - - [01/Apr/2023:13:08:29 +0000] "POST /api/v1/verification/verify HTTP/1.1" 504 160 "-" "okhttp/3.12.0"

That shows that at some point it first fails with timeout and then it fails with 504 error. I don't see the error in this log, please run docker logs compreface-core and docker logs compreface-api, so I see more logs.

pospielov commented 1 year ago

@xfstart07 CompreFace doesn't support M1 CPU, as it based on ARM architecture.

zipbooksit commented 1 year ago

Hello Team,

As per your suggestion we have reduced image upload size and also we have increased server infrastructure to manage load but still the issue is coming frequently while face comparison.

Request you to please share required image size and provide a solution for this.

For your reference we have attached log file. api_log_19072023.txt

Anatolii-R commented 1 year ago

Hello If this issue occurs sporadically rather than consistently, it might be due to overloading your server. Could you please let me know which operating system you're currently using and the frequency of your request submissions?

Anatolii-R commented 1 year ago

Could you also provide me with the parameters related to this specific part from your .env file? compreface_api_java_options= compreface_admin_java_options= max_file_size= max_request_size= max_detect_size= uwsgi_processes= uwsgi_threads= connection_timeout= read_timeout= ADMIN_VERSION= API_VERSION= FE_VERSION= CORE_VERSION= POSTGRES_VERSION=

zipbooksit commented 1 year ago

Hello Team, Thanks for your Support!!! As per your suggestion please find below details. Operating System =Ubuntu 20.04.5 LTS The frequency of your request submissions=around 30 calls per minute compreface_api_java_options=-Xmx4g compreface_admin_java_options=-Xmx1g max_file_size=5MB max_request_size=10MB max_detect_size=(we don't find) uwsgi_processes=2 uwsgi_threads=1 connection_timeout=10000 read_timeout=60000 ADMIN_VERSION=latest API_VERSION=latest FE_VERSION=latest CORE_VERSION=latest POSTGRES_VERSION=13

Please check and guide us to get the proper data from the server if it is incorrect. Request you to provide a solution with this configuration. For your reference we have attached Docker file. Dockerfile.txt

Anatolii-R commented 1 year ago

Hello Zipbooksit. I attempted to reproduce this error to no avail. Could you please send all the logs related to this issue? Typically, you can find them at /var/lib/docker/containers//-json.log. (If log is huge and command
docker logs [OPTIONS] CONTAINER show only his part) For example it can looks like this - Screenshot from 2023-08-01 20-12-45 where f3a5d1af0bf8c1aac3be66bb40c727b30e7a3f8e47df8bfc6c9fbf05f989d385-json.log is my docker log

zipbooksit commented 1 year ago

Hello Team, As per your Requirement please find the attached log files. Currently, on our server, log files are in big size, so we are providing you with some of the content. Please check and let us know if any further details are required and request you to provide a solution for this. ComprefaceJSON_Core_log.txt ComprefaceJSON_API_log.log