search

exadel-inc / CompreFace

Leading free and open-source face recognition system
https://exadel.com/accelerator-showcase/compreface/
Apache License 2.0
5.52k stars 754 forks source link

Error when trying to use SubCenter-ArcFace-r100-GPU #1075

Open anthonymkz opened 1 year ago

anthonymkz commented 1 year ago

Describe the bug

I've tried multiple methods to run SubCenter-ArcFace-r100-GPU build in Unraid with no success. When trying to run recognition on a face I get the error:

error_outline Error during synchronization between servers: [500 INTERNAL SERVER ERROR] during [POST] to [http://compreface-core:3000/find_faces] [FacesFeignClient#findFaces(MultipartFile,Integer,Double,String)]: [{"message":"MXNetError: Traceback (most recent call last):\n File \"../include/mshadow/././././cuda/tensor_gpu-inl.cuh\", line 128\nName: Check failed: err == cudaSuccess (209 vs. 0) : MapPlanKernel ErrStr:no kernel image is available for execution on the device"} ]

To Reproduce

Steps to reproduce the behavior:

  1. Go to Unraid Community Applications to download a pre-built Docker Container or Try to Run ArcFace-GPU in Docker Compose on Unraid

Expected behavior

Facial Recognition using my Quadro K6000

Logs

Run those commands and attach result to the ticket:

docker ps docker ps

docker-compose logs compreface-ui | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration compreface-ui | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ compreface-ui | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh compreface-ui | 10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist compreface-ui | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh compreface-ui | 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/nginx.conf.template to /etc/nginx/conf.d/nginx.conf compreface-ui | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh compreface-ui | /docker-entrypoint.sh: Configuration complete; ready for start up compreface-ui | 192.168.0.117 - CommonClientId [27/May/2023:00:58:18 +0000] "POST /admin/oauth/token?grant_type=refresh_token&scope=all HTTP/1.1" 200 0 "http://192.168.0.154:8000/test-model?app=0a747680-1bf7-4a90-a00b-9113fa8219ee&model=390cabeb-e89e-40f8-ac50-37260356583b&type=RECOGNITION" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36" compreface-ui | 192.168.0.117 - CommonClientId [27/May/2023:01:03:26 +0000] "POST /admin/oauth/token?grant_type=refresh_token&scope=all HTTP/1.1" 200 0 "http://192.168.0.154:8000/test-model?app=0a747680-1bf7-4a90-a00b-9113fa8219ee&model=390cabeb-e89e-40f8-ac50-37260356583b&type=RECOGNITION" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36" compreface-admin | Listening for transport dtsocket at address: 5005 compreface-admin | compreface-admin | . ____ compreface-admin | /\ / __' () _ \ \ \ \ compreface-api | Listening for transport dtsocket at address: 5005 compreface-core | [uWSGI] getting INI configuration from uwsgi.ini compreface-api | compreface-core | Starting uWSGI 2.0.19 (64bit) on [Sat May 27 01:01:05 2023] compreface-api | . ____ compreface-api | /\ / __' () _ \ \ \ \ compreface-api | ( ( )_ | ' | '| | ' \/ ` | \ \ \ \ compreface-postgres-db | 2023-05-27 00:57:27.026 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 compreface-postgres-db | 2023-05-27 00:57:27.026 UTC [1] LOG: listening on IPv6 address "::", port 5432 compreface-postgres-db | 2023-05-27 00:57:27.104 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" compreface-api | \/ __)| |)| | | | | || (_| | ) ) ) ) compreface-postgres-db | 2023-05-27 00:57:27.232 UTC [24] LOG: database system was shut down at 2023-05-27 00:43:41 UTC compreface-core | compiled with version: 9.4.0 on 07 October 2022 10:41:31 compreface-postgres-db | 2023-05-27 00:57:27.262 UTC [1] LOG: database system is ready to accept connections compreface-core | os: Linux-5.19.17-Unraid #2 SMP PREEMPT_DYNAMIC Wed Nov 2 11:54:15 PDT 2022 compreface-core | nodename: 2ad9ad512210 compreface-core | machine: x8664 compreface-admin | ( ( )\ | ' | '| | ' \/ ` | \ \ \ \ compreface-admin | \/ )| |)| | | | | || (_| | ) ) ) ) compreface-core | clock source: unix compreface-core | detected number of CPU cores: 16 compreface-core | current working directory: /app/ml compreface-core | detected binary path: /usr/local/bin/uwsgi compreface-admin | ' |__| .|| ||| |_, | / / / / compreface-core | !!! no internal routing support, rebuild with pcre support !!! compreface-admin | =========||==============|/=//// compreface-core | setgid() to 33 compreface-postgres-db | 2023-05-27 00:57:35.179 UTC [32] WARNING: there is no transaction in progress compreface-admin | :: Spring Boot :: (v2.5.13) compreface-core | setuid() to 33 compreface-core | your processes number limit is 256149 compreface-core | your memory page size is 4096 bytes compreface-core | detected max file descriptor number: 40960 compreface-core | lock engine: pthread robust mutexes compreface-core | thunder lock: disabled (you can enable it with --thunder-lock) compreface-core | uwsgi socket 0 bound to TCP address 0.0.0.0:3000 fd 3 compreface-core | Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] compreface-core | Python main interpreter initialized at 0x55bd41c6d660 compreface-core | python threads support enabled compreface-core | your server socket listen backlog is limited to 100 connections compreface-core | your mercy for graceful operations on workers is 60 seconds compreface-api | ' || .|| ||| |_, | / / / / compreface-core | mapped 218712 bytes (213 KB) for 2 cores compreface-api | =========||==============|__/=//// compreface-api | :: Spring Boot :: (v2.5.13) compreface-api | compreface-api | 2023-05-27 00:57:29.107 INFO 7 --- [kground-preinit] o.h.validator.internal.util.Version : HV000001: Hibernate Validator 6.2.3.Final compreface-core | Operational MODE: preforking compreface-core | {"severity": "INFO", "message": "{'BUILD_VERSION': 'dev', 'CALCULATION_PLUGIN': 'insightface.Calculator@arcface-r100-msfdrop75', 'EXTRA_PLUGINS': ['insightface.LandmarksDetector', 'insightface.GenderDetector', 'insightface.AgeDetector', 'insightface.facemask.MaskDetector', 'insightface.PoseEstimator'], 'FACE_DETECTION_PLUGIN': 'insightface.FaceDetector@retinaface_r50_v1', 'GPU_IDX': 0, 'IMG_LENGTH_LIMIT': 640, 'INTEL_OPTIMIZATION': False, 'IS_DEV_ENV': False, 'LOGGING_LEVEL_NAME': 'DEBUG', 'ML_PORT': 3000, 'RUN_MODE': True}", "request": null, "logger": "src.app", "module": "app", "traceback": null, "build_version": "dev"} compreface-core | {"severity": "DEBUG", "message": "Creating new app for WSGI", "request": null, "logger": "src.app", "module": "app", "traceback": null, "buildversion": "dev"} compreface-core | WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0x55bd41c6d660 pid: 1 (default app) compreface-core | uWSGI is running in multiple interpreter mode compreface-core | spawned uWSGI master process (pid: 1) compreface-core | spawned uWSGI worker 1 (pid: 38, cores: 1) compreface-core | spawned uWSGI worker 2 (pid: 39, cores: 1) compreface-api | 2023-05-27 00:57:29.110 INFO 7 --- [ main] com.exadel.frs.TrainServiceApplication : Starting TrainServiceApplication v0.0.1-SNAPSHOT using Java 11.0.8 on 97d8874a3b6d with PID 7 (/home/app.jar started by root in /) compreface-admin | compreface-api | 2023-05-27 00:57:29.111 INFO 7 --- [ main] com.exadel.frs.TrainServiceApplication : The following 1 profile is active: "dev" compreface-api | 2023-05-27 00:57:29.190 WARN 7 --- [ main] o.s.b.c.config.ConfigDataEnvironment : Property 'spring.profiles' imported from location 'class path resource [application.yml]' is invalid and should be replaced with 'spring.config.activate.on-profile' [origin: class path resource [application.yml] from app.jar - 94:13] compreface-admin | 2023-05-27 00:57:29.552 INFO 7 --- [kground-preinit] o.h.validator.internal.util.Version : HV000001: Hibernate Validator 6.2.3.Final compreface-admin | 2023-05-27 00:57:29.571 INFO 7 --- [ main] com.exadel.frs.FrsApplication : Starting FrsApplication v0.0.1-SNAPSHOT using Java 11.0.8 on 11e979dfa108 with PID 7 (/home/app.jar started by root in /) compreface-admin | 2023-05-27 00:57:29.572 DEBUG 7 --- [ main] com.exadel.frs.FrsApplication : Running with Spring Boot v2.5.13, Spring v5.3.19 compreface-admin | 2023-05-27 00:57:29.572 INFO 7 --- [ main] com.exadel.frs.FrsApplication : The following 1 profile is active: "dev" compreface-admin | 2023-05-27 00:57:31.135 INFO 7 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. compreface-admin | 2023-05-27 00:57:31.242 INFO 7 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 100 ms. Found 11 JPA repository interfaces. compreface-admin | 2023-05-27 00:57:31.640 INFO 7 --- [ main] o.s.cloud.context.scope.GenericScope : BeanFactory id=87eecb33-b947-31b8-bbb7-8d063df92185 compreface-admin | 2023-05-27 00:57:32.620 INFO 7 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http) compreface-admin | 2023-05-27 00:57:32.637 INFO 7 --- [ main] o.a.coyote.http11.Http11NioProtocol : Initializing ProtocolHandler ["http-nio-8080"] compreface-admin | 2023-05-27 00:57:32.638 INFO 7 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat] compreface-admin | 2023-05-27 00:57:32.640 INFO 7 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.62] compreface-admin | 2023-05-27 00:57:32.752 INFO 7 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext compreface-admin | 2023-05-27 00:57:32.752 INFO 7 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 3140 ms compreface-admin | 2023-05-27 00:57:33.354 INFO 7 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting... compreface-admin | 2023-05-27 00:57:33.448 INFO 7 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed. compreface-admin | 2023-05-27 00:57:33.567 INFO 7 --- [ main] liquibase.database : Set default schema name to public compreface-admin | 2023-05-27 00:57:33.628 INFO 7 --- [ main] liquibase.lockservice : Successfully acquired change log lock compreface-admin | 2023-05-27 00:57:33.979 INFO 7 --- [ main] liquibase.changelog : Reading from public.databasechangelog compreface-admin | 2023-05-27 00:57:34.090 INFO 7 --- [ main] liquibase.lockservice : Successfully released change log lock compreface-admin | 2023-05-27 00:57:35.242 INFO 7 --- [ main] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default] compreface-admin | 2023-05-27 00:57:35.305 INFO 7 --- [ main] org.hibernate.Version : HHH000412: Hibernate ORM core version 5.4.33 compreface-admin | 2023-05-27 00:57:35.425 INFO 7 --- [ main] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.2.Final} compreface-admin | 2023-05-27 00:57:35.541 INFO 7 --- [ main] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.PostgreSQL10Dialect compreface-admin | 2023-05-27 00:57:35.617 INFO 7 --- [ main] Hibernate Types : This framework is proudly powered by: compreface-admin | compreface-api | 2023-05-27 00:57:30.025 INFO 7 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode. compreface-api | 2023-05-27 00:57:30.108 INFO 7 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 76 ms. Found 8 JPA repository interfaces. compreface-api | 2023-05-27 00:57:30.876 INFO 7 --- [ main] o.s.cloud.context.scope.GenericScope : BeanFactory id=9776d44d-52e1-38dc-b671-bd86977f22bf compreface-api | 2023-05-27 00:57:31.299 INFO 7 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'cacheConfig' of type [com.exadel.frs.core.trainservice.config.CacheConfig$$EnhancerBySpringCGLIB$$e01009b] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) compreface-api | 2023-05-27 00:57:31.794 INFO 7 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http) compreface-api | 2023-05-27 00:57:31.808 INFO 7 --- [ main] o.a.coyote.http11.Http11NioProtocol : Initializing ProtocolHandler ["http-nio-8080"] compreface-api | 2023-05-27 00:57:31.809 INFO 7 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat] compreface-api | 2023-05-27 00:57:31.809 INFO 7 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.62] compreface-api | 2023-05-27 00:57:31.914 INFO 7 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext compreface-admin | >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compreface-admin | compreface-admin | | | | | () | | compreface-admin | | || | _ | | __ compreface-admin | | | | | | ' \ / \ '/ | / | / \ ' \ / / \ compreface-admin | | | | | || | |) | / | \ \ _ \ || / | | | (_| / compreface-admin | || ||\, | ._/ \|_| |/|/\\|| ||__| compreface-admin | __/ | | compreface-admin | |__/|| compreface-admin | compreface-admin | At Hypersistence, we only build amazing tools, like Hibernate Types, Flexy Pool, or Hypersistence Optimizer. compreface-admin | compreface-api | 2023-05-27 00:57:31.914 INFO 7 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2722 ms compreface-api | 2023-05-27 00:57:32.507 INFO 7 --- [ main] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default] compreface-api | 2023-05-27 00:57:32.672 INFO 7 --- [ main] org.hibernate.Version : HHH000412: Hibernate ORM core version 5.4.33 compreface-api | 2023-05-27 00:57:32.676 INFO 7 --- [ main] org.hibernate.cfg.Environment : HHH000205: Loaded properties from resource hibernate.properties: {hibernate.bytecode.use_reflection_optimizer=false, hibernate.types.print.banner=false} compreface-api | 2023-05-27 00:57:32.826 INFO 7 --- [ main] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.2.Final} compreface-api | 2023-05-27 00:57:33.321 INFO 7 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting... compreface-api | 2023-05-27 00:57:33.413 INFO 7 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed. compreface-api | 2023-05-27 00:57:33.431 INFO 7 --- [ main] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.PostgreSQL10Dialect compreface-api | 2023-05-27 00:57:34.190 INFO 7 --- [ main] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform] compreface-api | 2023-05-27 00:57:34.197 INFO 7 --- [ main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default' compreface-api | 2023-05-27 00:57:36.106 INFO 7 --- [ main] o.s.b.a.e.web.EndpointLinksResolver : Exposing 1 endpoint(s) beneath base path '/actuator' compreface-api | 2023-05-27 00:57:36.307 INFO 7 --- [ main] pertySourcedRequestMappingHandlerMapping : Mapped URL path [/v2/api-docs] onto method [springfox.documentation.swagger2.web.Swagger2Controller#getDocumentation(String, HttpServletRequest)] compreface-api | 2023-05-27 00:57:36.361 INFO 7 --- [ main] org.quartz.impl.StdSchedulerFactory : Using default implementation for ThreadExecutor compreface-admin | What if there were a tool that could automatically detect JPA and Hibernate performance issues? compreface-api | 2023-05-27 00:57:36.372 INFO 7 --- [ main] org.quartz.core.SchedulerSignalerImpl : Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl compreface-admin | compreface-admin | Hypersistence Optimizer is that tool! For more details, go to: compreface-admin | compreface-admin | https://vladmihalcea.com/hypersistence-optimizer/ compreface-api | 2023-05-27 00:57:36.372 INFO 7 --- [ main] org.quartz.core.QuartzScheduler : Quartz Scheduler v.2.3.2 created. compreface-api | 2023-05-27 00:57:36.373 INFO 7 --- [ main] org.quartz.simpl.RAMJobStore : RAMJobStore initialized. compreface-api | 2023-05-27 00:57:36.374 INFO 7 --- [ main] org.quartz.core.QuartzScheduler : Scheduler meta-data: Quartz Scheduler (v2.3.2) 'quartzScheduler' with instanceId 'NONCLUSTERED' compreface-api | Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally. compreface-api | NOT STARTED. compreface-api | Currently in standby mode. compreface-api | Number of jobs executed: 0 compreface-api | Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads. compreface-api | Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered. compreface-api | compreface-api | 2023-05-27 00:57:36.374 INFO 7 --- [ main] org.quartz.impl.StdSchedulerFactory : Quartz scheduler 'quartzScheduler' initialized from an externally provided properties instance. compreface-api | 2023-05-27 00:57:36.374 INFO 7 --- [ main] org.quartz.impl.StdSchedulerFactory : Quartz scheduler version: 2.3.2 compreface-api | 2023-05-27 00:57:36.374 INFO 7 --- [ main] org.quartz.core.QuartzScheduler : JobFactory set to: org.springframework.scheduling.quartz.SpringBeanJobFactory@76220ef1 compreface-api | 2023-05-27 00:57:36.379 INFO 7 --- [ main] org.quartz.core.QuartzScheduler : Scheduler quartzScheduler$_NON_CLUSTERED started. compreface-api | 2023-05-27 00:57:36.852 INFO 7 --- [ main] o.a.coyote.http11.Http11NioProtocol : Starting ProtocolHandler ["http-nio-8080"] compreface-api | 2023-05-27 00:57:36.869 INFO 7 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path '' compreface-api | 2023-05-27 00:57:36.871 INFO 7 --- [ main] d.s.w.p.DocumentationPluginsBootstrapper : Context refreshed compreface-api | 2023-05-27 00:57:36.900 INFO 7 --- [ main] d.s.w.p.DocumentationPluginsBootstrapper : Found 2 custom documentation plugin(s) compreface-api | 2023-05-27 00:57:37.016 INFO 7 --- [ main] s.d.s.w.s.ApiListingReferenceScanner : Scanning for api listing references compreface-api | 2023-05-27 00:57:37.315 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: recognizeBase64UsingPOST_1 compreface-api | 2023-05-27 00:57:37.318 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: downloadImgUsingGET_1 compreface-api | 2023-05-27 00:57:37.395 INFO 7 --- [ main] s.d.s.w.s.ApiListingReferenceScanner : Scanning for api listing references compreface-api | 2023-05-27 00:57:37.425 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: getConfigUsingGET_1 compreface-api | 2023-05-27 00:57:37.428 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: getCheckDemoUsingGET_1 compreface-api | 2023-05-27 00:57:37.473 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: detectUsingPOST_1 compreface-api | 2023-05-27 00:57:37.477 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: detectBase64UsingPOST_1 compreface-api | 2023-05-27 00:57:37.481 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: addEmbeddingUsingPOST_1 compreface-api | 2023-05-27 00:57:37.484 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: addEmbeddingBase64UsingPOST_1 compreface-api | 2023-05-27 00:57:37.485 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: deleteEmbeddingByIdUsingDELETE_1 compreface-api | 2023-05-27 00:57:37.488 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: deleteEmbeddingsByIdUsingPOST_1 compreface-api | 2023-05-27 00:57:37.490 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: downloadImgUsingGET_2 compreface-api | 2023-05-27 00:57:37.495 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: listEmbeddingsUsingGET_1 compreface-api | 2023-05-27 00:57:37.505 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: recognizeBase64UsingPOST_2 compreface-api | 2023-05-27 00:57:37.507 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: recognizeFileUsingPOST_1 compreface-api | 2023-05-27 00:57:37.510 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: removeAllSubjectEmbeddingsUsingDELETE_1 compreface-api | 2023-05-27 00:57:37.512 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: migrateUsingPOST_1 compreface-api | 2023-05-27 00:57:37.516 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: handleUsingGET_1 compreface-api | 2023-05-27 00:57:37.525 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: recognizeUsingPOST_1 compreface-api | 2023-05-27 00:57:37.528 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: recognizeBase64UsingPOST_3 compreface-api | 2023-05-27 00:57:37.534 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: downloadImgUsingGET_3 compreface-api | 2023-05-27 00:57:37.539 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: createSubjectUsingPOST_1 compreface-api | 2023-05-27 00:57:37.542 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: deleteSubjectUsingDELETE_1 compreface-api | 2023-05-27 00:57:37.543 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: deleteSubjectsUsingDELETE_1 compreface-api | 2023-05-27 00:57:37.545 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: listSubjectsUsingGET_1 compreface-api | 2023-05-27 00:57:37.546 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: renameSubjectUsingPUT_1 compreface-api | 2023-05-27 00:57:37.557 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: verifyUsingPOST_1 compreface-api | 2023-05-27 00:57:37.562 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: verifyBase64UsingPOST_1 compreface-api | 2023-05-27 00:57:37.587 INFO 7 --- [ main] com.exadel.frs.TrainServiceApplication : Started TrainServiceApplication in 9.342 seconds (JVM running for 10.169) compreface-api | 2023-05-27 01:00:00.007 INFO 7 --- [ scheduling-1] c.e.f.c.t.service.ModelStatisticService : No statistic to update or record. compreface-admin | <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< compreface-admin | compreface-admin | 2023-05-27 00:57:36.675 INFO 7 --- [ main] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform] compreface-admin | 2023-05-27 00:57:36.685 INFO 7 --- [ main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default' compreface-admin | 2023-05-27 00:57:39.060 INFO 7 --- [ main] o.s.b.a.e.web.EndpointLinksResolver : Exposing 1 endpoint(s) beneath base path '/actuator' compreface-admin | 2023-05-27 00:57:39.147 INFO 7 --- [ main] pertySourcedRequestMappingHandlerMapping : Mapped URL path [/v2/api-docs] onto method [springfox.documentation.swagger2.web.Swagger2Controller#getDocumentation(String, HttpServletRequest)] compreface-admin | 2023-05-27 00:57:39.813 WARN 7 --- [ main] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/v2/api-docs']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. compreface-admin | 2023-05-27 00:57:39.814 INFO 7 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will secure Ant [pattern='/v2/api-docs'] with [] compreface-admin | 2023-05-27 00:57:39.814 WARN 7 --- [ main] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/configuration/ui']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. compreface-admin | 2023-05-27 00:57:39.814 INFO 7 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will secure Ant [pattern='/configuration/ui'] with [] compreface-admin | 2023-05-27 00:57:39.814 WARN 7 --- [ main] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/swagger-resources/']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. compreface-admin | 2023-05-27 00:57:39.814 INFO 7 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will secure Ant [pattern='/swagger-resources/'] with [] compreface-admin | 2023-05-27 00:57:39.814 WARN 7 --- [ main] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/configuration/']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. compreface-admin | 2023-05-27 00:57:39.814 INFO 7 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will secure Ant [pattern='/configuration/'] with [] compreface-admin | 2023-05-27 00:57:39.814 WARN 7 --- [ main] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/swagger-ui']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. compreface-admin | 2023-05-27 00:57:39.814 INFO 7 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will secure Ant [pattern='/swagger-ui'] with [] compreface-admin | 2023-05-27 00:57:39.814 WARN 7 --- [ main] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/webjars/']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. compreface-admin | 2023-05-27 00:57:39.815 INFO 7 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will secure Ant [pattern='/webjars/'] with [] compreface-admin | 2023-05-27 00:57:39.815 WARN 7 --- [ main] o.s.s.c.a.web.builders.WebSecurity : You are asking Spring Security to ignore Ant [pattern='/lms/']. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead. compreface-admin | 2023-05-27 00:57:39.815 INFO 7 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will secure Ant [pattern='/lms/'] with [] compreface-admin | 2023-05-27 00:57:39.858 INFO 7 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will not secure Or [Ant [pattern='/oauth/token'], Ant [pattern='/oauth/token_key'], Ant [pattern='/oauth/check_token']] compreface-admin | 2023-05-27 00:57:39.871 INFO 7 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will not secure org.springframework.security.oauth2.config.annotation.web.configuration.ResourceServerConfiguration$NotOAuthRequestMatcher@50628080 compreface-admin | 2023-05-27 00:57:39.874 INFO 7 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will not secure any request compreface-admin | 2023-05-27 00:57:39.980 INFO 7 --- [ main] org.quartz.impl.StdSchedulerFactory : Using default implementation for ThreadExecutor compreface-admin | 2023-05-27 00:57:39.993 INFO 7 --- [ main] org.quartz.core.SchedulerSignalerImpl : Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl compreface-admin | 2023-05-27 00:57:39.993 INFO 7 --- [ main] org.quartz.core.QuartzScheduler : Quartz Scheduler v.2.3.2 created. compreface-admin | 2023-05-27 00:57:39.996 INFO 7 --- [ main] o.s.s.quartz.LocalDataSourceJobStore : Using db table-based data access locking (synchronization). compreface-admin | 2023-05-27 00:57:39.999 INFO 7 --- [ main] o.s.s.quartz.LocalDataSourceJobStore : JobStoreCMT initialized. compreface-admin | 2023-05-27 00:57:40.000 INFO 7 --- [ main] org.quartz.core.QuartzScheduler : Scheduler meta-data: Quartz Scheduler (v2.3.2) 'statistics scheduler' with instanceId 'NONCLUSTERED' compreface-admin | Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally. compreface-admin | NOT STARTED. compreface-admin | Currently in standby mode. compreface-admin | Number of jobs executed: 0 compreface-admin | Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 1 threads. compreface-admin | Using job-store 'org.springframework.scheduling.quartz.LocalDataSourceJobStore' - which supports persistence. and is not clustered. compreface-admin | compreface-admin | 2023-05-27 00:57:40.000 INFO 7 --- [ main] org.quartz.impl.StdSchedulerFactory : Quartz scheduler 'statistics scheduler' initialized from an externally provided properties instance. compreface-admin | 2023-05-27 00:57:40.000 INFO 7 --- [ main] org.quartz.impl.StdSchedulerFactory : Quartz scheduler version: 2.3.2 compreface-admin | 2023-05-27 00:57:40.000 INFO 7 --- [ main] org.quartz.core.QuartzScheduler : JobFactory set to: org.springframework.scheduling.quartz.SpringBeanJobFactory@41948c13 compreface-admin | 2023-05-27 00:57:40.041 INFO 7 --- [ main] o.s.s.quartz.LocalDataSourceJobStore : Freed 0 triggers from 'acquired' / 'blocked' state. compreface-admin | 2023-05-27 00:57:40.042 INFO 7 --- [ main] o.s.s.quartz.LocalDataSourceJobStore : Recovering 0 jobs that were in-progress at the time of the last shut-down. compreface-admin | 2023-05-27 00:57:40.042 INFO 7 --- [ main] o.s.s.quartz.LocalDataSourceJobStore : Recovery complete. compreface-admin | 2023-05-27 00:57:40.043 INFO 7 --- [ main] o.s.s.quartz.LocalDataSourceJobStore : Removed 0 'complete' triggers. compreface-admin | 2023-05-27 00:57:40.043 INFO 7 --- [ main] o.s.s.quartz.LocalDataSourceJobStore : Removed 0 stale fired job entries. compreface-admin | 2023-05-27 00:57:40.053 INFO 7 --- [ main] org.quartz.core.QuartzScheduler : Scheduler statistics scheduler$_NON_CLUSTERED started. compreface-admin | 2023-05-27 00:57:40.518 INFO 7 --- [ main] o.a.coyote.http11.Http11NioProtocol : Starting ProtocolHandler ["http-nio-8080"] compreface-admin | 2023-05-27 00:57:40.545 INFO 7 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path '' compreface-admin | 2023-05-27 00:57:40.546 INFO 7 --- [ main] d.s.w.p.DocumentationPluginsBootstrapper : Context refreshed compreface-admin | 2023-05-27 00:57:40.574 INFO 7 --- [ main] d.s.w.p.DocumentationPluginsBootstrapper : Found 1 custom documentation plugin(s) compreface-admin | 2023-05-27 00:57:40.623 INFO 7 --- [ main] s.d.s.w.s.ApiListingReferenceScanner : Scanning for api listing references compreface-admin | 2023-05-27 00:57:40.848 INFO 7 --- [ main] .d.s.w.r.o.CachingOperationNameGenerator : Generating unique operation named: regenerateApiKeyUsingPUT_1 compreface-admin | 2023-05-27 00:57:40.899 INFO 7 --- [ main] com.exadel.frs.FrsApplication : Started FrsApplication in 12.187 seconds (JVM running for 12.985) compreface-admin | 2023-05-27 00:58:18.382 INFO 7 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring DispatcherServlet 'dispatcherServlet' compreface-admin | 2023-05-27 00:58:18.382 INFO 7 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Initializing Servlet 'dispatcherServlet' compreface-admin | 2023-05-27 00:58:18.383 INFO 7 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Completed initialization in 0 ms compreface-admin | 2023-05-27 00:58:18.600 INFO 7 --- [nio-8080-exec-1] o.s.s.o.p.token.store.JdbcTokenStore : Failed to find access token compreface-admin | 2023-05-27 01:03:26.403 INFO 7 --- [nio-8080-exec-2] o.s.s.o.p.token.store.JdbcTokenStore : Failed to find access token

Additional context

I've had no problem running the CPU only version and have had a CPU only version running on another server for months now. Someone please help. I've been driving myself nuts with this all day. I believe the error has something to do with the internal routing based on my admittedly amateur review of the error and the logs but I have no idea how to fix it.

pospielov commented 1 year ago

There are some problems with CUDA. But I can't understand what exact problem from logs. I have two guesses:

  1. Did you install docker nvidia runtime?
  2. Your GPU is quite old. But It should work with CompreFace 1.1. Still, try to run CompreFace 1.0.
anthonymkz commented 1 year ago

I bought this GPU based on the fact that it works with compreface 1.01 as per the posted compatible GPU's (not that I am trying to place blame - just stating it is supposed to be compatible). That being said I do have to use older drivers (v470.141.03) as this was the last driver version to support this GPU.

The 1.0.0 version does not work, I think there is a problem with the image for some reason. When I try to use 1.0.1 I get the attached error which ultimately is the same error I get using 1.1.0

From: Pospielov Serhii @.> Sent: Thursday, June 1, 2023 4:10 PM To: exadel-inc/CompreFace @.> Cc: anthonymkz @.>; Author @.> Subject: Re: [exadel-inc/CompreFace] Error when trying to use SubCenter-ArcFace-r100-GPU (Issue #1075)

There are some problems with CUDA. But I can't understand what exact problem from logs. I have two guesses:

  1. Did you install docker nvidia runtime?
  2. Your GPU is quite old. But It should work with CompreFace 1.1. Still, try to run CompreFace 1.0.

— Reply to this email directly, view it on GitHubhttps://github.com/exadel-inc/CompreFace/issues/1075#issuecomment-1572708108, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A5FXRIBFBM2ZWJU7KIY2YS3XJDZLXANCNFSM6AAAAAAYQ22GXQ. You are receiving this because you authored the thread.Message ID: @.***>

anthonymkz commented 1 year ago

So after a lot of digging and I mean a LOT of digging, it seems that this issue is being caused by a mismatch in the cudnn libraries being used by the core image and mxnet. I'm not sure if that's accurate but that's the best I can tell and unfortunately fixing this is way above my paygrade. Hoping this helps figure it out, though because it looks like others are having similar issues.