search

neuromatch / NEXT

NEXT is a machine learning system that runs in the cloud and makes it easy to develop, evaluate, and apply active learning in the real-world. Ask better questions. Get better results. Faster. Automated.
http://nextml.org
Apache License 2.0
0 stars 0 forks source link

CrowdKernel Time Limit Error #8

Open iamzoltan opened 1 month ago

iamzoltan commented 1 month ago 
t/getQuery
minionworker-1       | [2024-05-15 21:26:45,860: WARNING/ForkPoolWorker-1] ########## Finished namespace:5c9d71061565ca92b097ff494c969b_UncertaintySampling,  job_uid=28fcb8a2bb9ae4716a46e4b6a58e63d6,  time_enqueued=0.001793,  execution_time=0.06032109260559082 ##########
minionworker-1       | [2024-05-15 21:26:45,874: WARNING/ForkPoolWorker-1] #### Finished getQuery,  time_enqueued=0.001527,  execution_time=0.053614139556884766 ####
nextbackenddocker-1  | [2024-05-15 21:26:46 +0000] [7] [DEBUG] POST /api/experiment/processAnswer
minionworker-1       | [2024-05-15 21:26:46,688: WARNING/ForkPoolWorker-1] #### Finished processAnswer,  time_enqueued=0.004867,  execution_time=0.03965330123901367 ####
minionworker-1       | [2024-05-15 21:26:46,688: WARNING/ForkPoolWorker-1] >>>>>>>> Starting namespace:5c9d71061565ca92b097ff494c969b_RandomSampling,  job_uid=2bb003efb24bb9353c5d7b9372af0715,  time_enqueued=0.00165 <<<<<<<<<
nextbackenddocker-1  | [2024-05-15 21:26:46 +0000] [7] [DEBUG] POST /api/experiment/getQuery
minionworker-1       | [2024-05-15 21:26:46,790: WARNING/ForkPoolWorker-1] #### Finished getQuery,  time_enqueued=0.002666,  execution_time=0.05387735366821289 ####
minionworker-1       | [2024-05-15 21:26:46,809: WARNING/ForkPoolWorker-1] ########## Finished namespace:5c9d71061565ca92b097ff494c969b_RandomSampling,  job_uid=2bb003efb24bb9353c5d7b9372af0715,  time_enqueued=0.00165,  execution_time=0.08316993713378906 ##########
nextbackenddocker-1  | [2024-05-15 21:26:47 +0000] [7] [DEBUG] POST /api/experiment/processAnswer
minionworker-1       | [2024-05-15 21:26:47,452: WARNING/ForkPoolWorker-1] #### Finished processAnswer,  time_enqueued=0.00487,  execution_time=0.006047248840332031 ####
nextbackenddocker-1  | [2024-05-15 21:26:47 +0000] [7] [DEBUG] POST /api/experiment/getQuery
minionworker-1       | [2024-05-15 21:26:47,528: WARNING/MainProcess] Soft time limit (5s) exceeded for next.broker.celery_app.tasks.apply_sync_by_namespace[54b761dd-7806-4974-b0b2-1befc4cefbce]
minionworker-1       | [2024-05-15 21:26:47,529: WARNING/ForkPoolWorker-2] tasks Exception: SoftTimeLimitExceeded() Traceback (most recent call last):
minionworker-1       |   File "/next_backend/next/broker/celery_app/tasks.py", line 201, in apply_sync_by_namespace
minionworker-1       |     response, dt = next.utils.timeit(getattr(next_alg, task_name))(butler, args)
minionworker-1       |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/next/utils.py", line 219, in timed
minionworker-1       |     result = f(*args, **kw)
minionworker-1       |              ^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/myAlg.py", line 124, in incremental_embedding_update
minionworker-1       |     tau = utilsCrowdKernel.getCrowdKernelTauDistribution(X,S,mu)
minionworker-1       |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 208, in getCrowdKernelTauDistribution
minionworker-1       |     tau[a,i] = tau[a,i] + log( getCrowdKernelTripletProbability(X[q[0]], X[q[1]], X[i], mu=mu))
minionworker-1       |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 168, in getCrowdKernelTripletProbability
minionworker-1       |     ba = norm(b-a)
minionworker-1       |          ^^^^^^^^^
minionworker-1       |   File "/usr/local/lib/python3.11/site-packages/numpy/linalg/linalg.py", line 2379, in _norm_dispatcher
minionworker-1       |     def _norm_dispatcher(x, ord=None, axis=None, keepdims=None):
minionworker-1       |
minionworker-1       |   File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 228, in soft_timeout_sighandler
minionworker-1       |     raise SoftTimeLimitExceeded()
minionworker-1       | billiard.exceptions.SoftTimeLimitExceeded: SoftTimeLimitExceeded()
minionworker-1       | [2024-05-15 21:26:47,529: WARNING/ForkPoolWorker-2]   File "/next_backend/next/broker/celery_app/tasks.py", line 201, in apply_sync_by_namespace
minionworker-1       |     response, dt = next.utils.timeit(getattr(next_alg, task_name))(butler, args)
minionworker-1       |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 21:26:47,529: WARNING/ForkPoolWorker-2]   File "/next_backend/next/utils.py", line 219, in timed
minionworker-1       |     result = f(*args, **kw)
minionworker-1       |              ^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 21:26:47,529: WARNING/ForkPoolWorker-2]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/myAlg.py", line 124, in incremental_embedding_update
minionworker-1       |     tau = utilsCrowdKernel.getCrowdKernelTauDistribution(X,S,mu)
minionworker-1       |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 21:26:47,529: WARNING/ForkPoolWorker-2]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 208, in getCrowdKernelTauDistribution
minionworker-1       |     tau[a,i] = tau[a,i] + log( getCrowdKernelTripletProbability(X[q[0]], X[q[1]], X[i], mu=mu))
minionworker-1       |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 21:26:47,529: WARNING/ForkPoolWorker-2]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 168, in getCrowdKernelTripletProbability
minionworker-1       |     ba = norm(b-a)
minionworker-1       |          ^^^^^^^^^
minionworker-1       | [2024-05-15 21:26:47,529: WARNING/ForkPoolWorker-2]   File "/usr/local/lib/python3.11/site-packages/numpy/linalg/linalg.py", line 2379, in _norm_dispatcher
minionworker-1       |     def _norm_dispatcher(x, ord=None, axis=None, keepdims=None):
minionworker-1       |
minionworker-1       | [2024-05-15 21:26:47,529: WARNING/ForkPoolWorker-2]   File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 228, in soft_timeout_sighandler
minionworker-1       |     raise SoftTimeLimitExceeded()
minionworker-1       | [2024-05-15 21:26:47,530: WARNING/ForkPoolWorker-2] >>>>>>>> Starting namespace:5c9d71061565ca92b097ff494c969b_Test,  job_uid=8703e697993eb09dddde937b621b02ff,  time_enqueued=109.411547 <<<<<<<<<
minionworker-1       | [2024-05-15 21:26:47,551: WARNING/ForkPoolW
iamzoltan commented 1 month ago 
ceeded()
minionworker-1       | [2024-05-15 22:10:14,924: WARNING/ForkPoolWorker-1]   File "/next_backend/next/broker/celery_app/tasks.py", line 201, in apply_sync_by_namespace
minionworker-1       |     response, dt = next.utils.timeit(getattr(next_alg, task_name))(butler, args)
minionworker-1       |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:14,924: WARNING/ForkPoolWorker-1]   File "/next_backend/next/utils.py", line 219, in timed
minionworker-1       |     result = f(*args, **kw)
minionworker-1       |              ^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:14,924: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/myAlg.py", line 124, in incremental_embedding_update
minionworker-1       |     tau = utilsCrowdKernel.getCrowdKernelTauDistribution(X,S,mu)
minionworker-1       |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:14,924: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 208, in getCrowdKernelTauDistribution
minionworker-1       |     tau[a,i] = tau[a,i] + log( getCrowdKernelTripletProbability(X[q[0]], X[q[1]], X[i], mu=mu))
minionworker-1       |                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:14,924: WARNING/ForkPoolWorker-1]   File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 228, in soft_timeout_sighandler
minionworker-1       |     raise SoftTimeLimitExceeded()
minionworker-1       | [2024-05-15 22:10:14,926: WARNING/ForkPoolWorker-1] >>>>>>>> Starting namespace:5c9d71061565ca92b097ff494c969b_Test,  job_uid=c487e70e3d9b0d3404bd6ed37a34c69e,  time_enqueued=1497.313422 <<<<<<<<<
minionworker-1       | [2024-05-15 22:10:19,859: WARNING/ForkPoolWorker-1] ########## Finished namespace:5c9d71061565ca92b097ff494c969b_Test,  job_uid=c487e70e3d9b0d3404bd6ed37a34c69e,  time_enqueued=1497.313422,  execution_time=4.914435148239136 ##########
minionworker-1       | [2024-05-15 22:10:19,860: WARNING/ForkPoolWorker-1] >>>>>>>> Starting namespace:5c9d71061565ca92b097ff494c969b_Test,  job_uid=f8937f28325a1badfc80cc85f6ecf96f,  time_enqueued=1501.996853 <<<<<<<<<
minionworker-1       | [2024-05-15 22:10:24,861: WARNING/MainProcess] Soft time limit (5s) exceeded for next.broker.celery_app.tasks.apply_sync_by_namespace[afce0f3e-3b28-49c2-b2a9-ea14f9df1750]
minionworker-1       | [2024-05-15 22:10:24,861: WARNING/ForkPoolWorker-1] tasks Exception: SoftTimeLimitExceeded() Traceback (most recent call last):
minionworker-1       |   File "/next_backend/next/broker/celery_app/tasks.py", line 201, in apply_sync_by_namespace
minionworker-1       |     response, dt = next.utils.timeit(getattr(next_alg, task_name))(butler, args)
minionworker-1       |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/next/utils.py", line 219, in timed
minionworker-1       |     result = f(*args, **kw)
minionworker-1       |              ^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/myAlg.py", line 124, in incremental_embedding_update
minionworker-1       |     tau = utilsCrowdKernel.getCrowdKernelTauDistribution(X,S,mu)
minionworker-1       |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 208, in getCrowdKernelTauDistribution
minionworker-1       |     tau[a,i] = tau[a,i] + log( getCrowdKernelTripletProbability(X[q[0]], X[q[1]], X[i], mu=mu))
minionworker-1       |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 167, in getCrowdKernelTripletProbability
minionworker-1       |     ca = norm(c-a)
minionworker-1       |          ^^^^^^^^^
minionworker-1       |   File "/usr/local/lib/python3.11/site-packages/numpy/linalg/linalg.py", line 2552, in norm
minionworker-1       |     sqnorm = x.dot(x)
minionworker-1       |              ^^^^^^^^
minionworker-1       |   File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 228, in soft_timeout_sighandler
minionworker-1       |     raise SoftTimeLimitExceeded()
minionworker-1       | billiard.exceptions.SoftTimeLimitExceeded: SoftTimeLimitExceeded()
minionworker-1       | [2024-05-15 22:10:24,862: WARNING/ForkPoolWorker-1]   File "/next_backend/next/broker/celery_app/tasks.py", line 201, in apply_sync_by_namespace
minionworker-1       |     response, dt = next.utils.timeit(getattr(next_alg, task_name))(butler, args)
minionworker-1       |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:24,862: WARNING/ForkPoolWorker-1]   File "/next_backend/next/utils.py", line 219, in timed
minionworker-1       |     result = f(*args, **kw)
minionworker-1       |              ^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:24,862: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/myAlg.py", line 124, in incremental_embedding_update
minionworker-1       |     tau = utilsCrowdKernel.getCrowdKernelTauDistribution(X,S,mu)
minionworker-1       |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:24,862: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 208, in getCrowdKernelTauDistribution
minionworker-1       |     tau[a,i] = tau[a,i] + log( getCrowdKernelTripletProbability(X[q[0]], X[q[1]], X[i], mu=mu))
minionworker-1       |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:24,862: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 167, in getCrowdKernelTripletProbability
minionworker-1       |     ca = norm(c-a)
minionworker-1       |          ^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:24,862: WARNING/ForkPoolWorker-1]   File "/usr/local/lib/python3.11/site-packages/numpy/linalg/linalg.py", line 2552, in norm
minionworker-1       |     sqnorm = x.dot(x)
minionworker-1       |              ^^^^^^^^
minionworker-1       | [2024-05-15 22:10:24,862: WARNING/ForkPoolWorker-1]   File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 228, in soft_timeout_sighandler
minionworker-1       |     raise SoftTimeLimitExceeded()
minionworker-1       | [2024-05-15 22:10:24,864: WARNING/ForkPoolWorker-1] >>>>>>>> Starting namespace:5c9d71061565ca92b097ff494c969b_Test,  job_uid=b047e418c61c5076bf8295d0fa57384b,  time_enqueued=1506.169664 <<<<<<<<<
minionworker-1       | [2024-05-15 22:10:29,866: WARNING/MainProcess] Soft time limit (5s) exceeded for next.broker.celery_app.tasks.apply_sync_by_namespace[e3532eff-4269-48e0-ac98-b9294cc2daaf]
minionworker-1       | [2024-05-15 22:10:29,867: WARNING/ForkPoolWorker-1] tasks Exception: SoftTimeLimitExceeded() Traceback (most recent call last):
minionworker-1       |   File "/next_backend/next/broker/celery_app/tasks.py", line 201, in apply_sync_by_namespace
minionworker-1       |     response, dt = next.utils.timeit(getattr(next_alg, task_name))(butler, args)
minionworker-1       |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/next/utils.py", line 219, in timed
minionworker-1       |     result = f(*args, **kw)
minionworker-1       |              ^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/myAlg.py", line 124, in incremental_embedding_update
minionworker-1       |     tau = utilsCrowdKernel.getCrowdKernelTauDistribution(X,S,mu)
minionworker-1       |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 208, in getCrowdKernelTauDistribution
minionworker-1       |     tau[a,i] = tau[a,i] + log( getCrowdKernelTripletProbability(X[q[0]], X[q[1]], X[i], mu=mu))
minionworker-1       |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 167, in getCrowdKernelTripletProbability
minionworker-1       |     ca = norm(c-a)
minionworker-1       |          ^^^^^^^^^
minionworker-1       |   File "/usr/local/lib/python3.11/site-packages/numpy/linalg/linalg.py", line 2546, in norm
minionworker-1       |     x = x.ravel(order='K')
minionworker-1       |         ^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 228, in soft_timeout_sighandler
minionworker-1       |     raise SoftTimeLimitExceeded()
minionworker-1       | billiard.exceptions.SoftTimeLimitExceeded: SoftTimeLimitExceeded()
minionworker-1       | [2024-05-15 22:10:29,867: WARNING/ForkPoolWorker-1]   File "/next_backend/next/broker/celery_app/tasks.py", line 201, in apply_sync_by_namespace
minionworker-1       |     response, dt = next.utils.timeit(getattr(next_alg, task_name))(butler, args)
minionworker-1       |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:29,867: WARNING/ForkPoolWorker-1]   File "/next_backend/next/utils.py", line 219, in timed
minionworker-1       |     result = f(*args, **kw)
minionworker-1       |              ^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:29,867: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/myAlg.py", line 124, in incremental_embedding_update
minionworker-1       |     tau = utilsCrowdKernel.getCrowdKernelTauDistribution(X,S,mu)
minionworker-1       |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:29,867: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 208, in getCrowdKernelTauDistribution
minionworker-1       |     tau[a,i] = tau[a,i] + log( getCrowdKernelTripletProbability(X[q[0]], X[q[1]], X[i], mu=mu))
minionworker-1       |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:29,867: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 167, in getCrowdKernelTripletProbability
minionworker-1       |     ca = norm(c-a)
minionworker-1       |          ^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:29,867: WARNING/ForkPoolWorker-1]   File "/usr/local/lib/python3.11/site-packages/numpy/linalg/linalg.py", line 2546, in norm
minionworker-1       |     x = x.ravel(order='K')
minionworker-1       |         ^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:29,867: WARNING/ForkPoolWorker-1]   File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 228, in soft_timeout_sighandler
minionworker-1       |     raise SoftTimeLimitExceeded()
minionworker-1       | [2024-05-15 22:10:29,869: WARNING/ForkPoolWorker-1] >>>>>>>> Starting namespace:5c9d71061565ca92b097ff494c969b_Test,  job_uid=946a918166627ce902e142eda4ac18c8,  time_enqueued=1509.985539 <<<<<<<<<
minionworker-1       | [2024-05-15 22:10:34,870: WARNING/MainProcess] Soft time limit (5s) exceeded for next.broker.celery_app.tasks.apply_sync_by_namespace[9cc99a61-3f6c-472d-9d58-5cff778d3370]
minionworker-1       | [2024-05-15 22:10:34,870: WARNING/ForkPoolWorker-1] tasks Exception: SoftTimeLimitExceeded() Traceback (most recent call last):
minionworker-1       |   File "/next_backend/next/broker/celery_app/tasks.py", line 201, in apply_sync_by_namespace
minionworker-1       |     response, dt = next.utils.timeit(getattr(next_alg, task_name))(butler, args)
minionworker-1       |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/next/utils.py", line 219, in timed
minionworker-1       |     result = f(*args, **kw)
minionworker-1       |              ^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/myAlg.py", line 124, in incremental_embedding_update
minionworker-1       |     tau = utilsCrowdKernel.getCrowdKernelTauDistribution(X,S,mu)
minionworker-1       |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 208, in getCrowdKernelTauDistribution
minionworker-1       |     tau[a,i] = tau[a,i] + log( getCrowdKernelTripletProbability(X[q[0]], X[q[1]], X[i], mu=mu))
minionworker-1       |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 168, in getCrowdKernelTripletProbability
minionworker-1       |     ba = norm(b-a)
minionworker-1       |          ^^^^^^^^^
minionworker-1       |   File "/usr/local/lib/python3.11/site-packages/numpy/linalg/linalg.py", line 2379, in _norm_dispatcher
minionworker-1       |     def _norm_dispatcher(x, ord=None, axis=None, keepdims=None):
minionworker-1       |
minionworker-1       |   File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 228, in soft_timeout_sighandler
minionworker-1       |     raise SoftTimeLimitExceeded()
minionworker-1       | billiard.exceptions.SoftTimeLimitExceeded: SoftTimeLimitExceeded()
minionworker-1       | [2024-05-15 22:10:34,870: WARNING/ForkPoolWorker-1]   File "/next_backend/next/broker/celery_app/tasks.py", line 201, in apply_sync_by_namespace
minionworker-1       |     response, dt = next.utils.timeit(getattr(next_alg, task_name))(butler, args)
minionworker-1       |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:34,870: WARNING/ForkPoolWorker-1]   File "/next_backend/next/utils.py", line 219, in timed
minionworker-1       |     result = f(*args, **kw)
minionworker-1       |              ^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:34,871: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/myAlg.py", line 124, in incremental_embedding_update
minionworker-1       |     tau = utilsCrowdKernel.getCrowdKernelTauDistribution(X,S,mu)
minionworker-1       |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:34,871: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 208, in getCrowdKernelTauDistribution
minionworker-1       |     tau[a,i] = tau[a,i] + log( getCrowdKernelTripletProbability(X[q[0]], X[q[1]], X[i], mu=mu))
minionworker-1       |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:34,871: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 168, in getCrowdKernelTripletProbability
minionworker-1       |     ba = norm(b-a)
minionworker-1       |          ^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:34,871: WARNING/ForkPoolWorker-1]   File "/usr/local/lib/python3.11/site-packages/numpy/linalg/linalg.py", line 2379, in _norm_dispatcher
minionworker-1       |     def _norm_dispatcher(x, ord=None, axis=None, keepdims=None):
minionworker-1       |
minionworker-1       | [2024-05-15 22:10:34,871: WARNING/ForkPoolWorker-1]   File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 228, in soft_timeout_sighandler
minionworker-1       |     raise SoftTimeLimitExceeded()
minionworker-1       | [2024-05-15 22:10:34,872: WARNING/ForkPoolWorker-1] >>>>>>>> Starting namespace:5c9d71061565ca92b097ff494c969b_Test,  job_uid=e01b2df9e2aeb66f1e8173fe4008b3bb,  time_enqueued=1513.741367 <<<<<<<<<
minionworker-1       | [2024-05-15 22:10:39,872: WARNING/MainProcess] Soft time limit (5s) exceeded for next.broker.celery_app.tasks.apply_sync_by_namespace[229ba661-5919-4cbd-919a-b7a53199fa6a]
minionworker-1       | [2024-05-15 22:10:39,873: WARNING/ForkPoolWorker-1] tasks Exception: SoftTimeLimitExceeded() Traceback (most recent call last):
minionworker-1       |   File "/next_backend/next/broker/celery_app/tasks.py", line 201, in apply_sync_by_namespace
minionworker-1       |     response, dt = next.utils.timeit(getattr(next_alg, task_name))(butler, args)
minionworker-1       |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/next/utils.py", line 219, in timed
minionworker-1       |     result = f(*args, **kw)
minionworker-1       |              ^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/myAlg.py", line 124, in incremental_embedding_update
minionworker-1       |     tau = utilsCrowdKernel.getCrowdKernelTauDistribution(X,S,mu)
minionworker-1       |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 208, in getCrowdKernelTauDistribution
minionworker-1       |     tau[a,i] = tau[a,i] + log( getCrowdKernelTripletProbability(X[q[0]], X[q[1]], X[i], mu=mu))
minionworker-1       |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 168, in getCrowdKernelTripletProbability
minionworker-1       |     ba = norm(b-a)
minionworker-1       |          ^^^^^^^^^
minionworker-1       |   File "/usr/local/lib/python3.11/site-packages/numpy/linalg/linalg.py", line 2379, in _norm_dispatcher
minionworker-1       |     def _norm_dispatcher(x, ord=None, axis=None, keepdims=None):
minionworker-1       |
minionworker-1       |   File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 228, in soft_timeout_sighandler
minionworker-1       |     raise SoftTimeLimitExceeded()
minionworker-1       | billiard.exceptions.SoftTimeLimitExceeded: SoftTimeLimitExceeded()
minionworker-1       | [2024-05-15 22:10:39,873: WARNING/ForkPoolWorker-1]   File "/next_backend/next/broker/celery_app/tasks.py", line 201, in apply_sync_by_namespace
minionworker-1       |     response, dt = next.utils.timeit(getattr(next_alg, task_name))(butler, args)
minionworker-1       |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:39,873: WARNING/ForkPoolWorker-1]   File "/next_backend/next/utils.py", line 219, in timed
minionworker-1       |     result = f(*args, **kw)
minionworker-1       |              ^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:39,873: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/myAlg.py", line 124, in incremental_embedding_update
minionworker-1       |     tau = utilsCrowdKernel.getCrowdKernelTauDistribution(X,S,mu)
minionworker-1       |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:39,873: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 208, in getCrowdKernelTauDistribution
minionworker-1       |     tau[a,i] = tau[a,i] + log( getCrowdKernelTripletProbability(X[q[0]], X[q[1]], X[i], mu=mu))
minionworker-1       |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:39,873: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 168, in getCrowdKernelTripletProbability
minionworker-1       |     ba = norm(b-a)
minionworker-1       |          ^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:39,873: WARNING/ForkPoolWorker-1]   File "/usr/local/lib/python3.11/site-packages/numpy/linalg/linalg.py", line 2379, in _norm_dispatcher
minionworker-1       |     def _norm_dispatcher(x, ord=None, axis=None, keepdims=None):
minionworker-1       |
minionworker-1       | [2024-05-15 22:10:39,873: WARNING/ForkPoolWorker-1]   File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 228, in soft_timeout_sighandler
minionworker-1       |     raise SoftTimeLimitExceeded()
minionworker-1       | [2024-05-15 22:10:39,874: WARNING/ForkPoolWorker-1] >>>>>>>> Starting namespace:5c9d71061565ca92b097ff494c969b_Test,  job_uid=bb59a54cd5234049af11ba6d9cae44ea,  time_enqueued=1517.899519 <<<<<<<<<