search

rfcx / arbimon-jobs

Apache License 2.0
0 stars 0 forks source link

problem with training #8

Closed lazychino closed 9 years ago

lazychino commented 9 years ago

user reported error with training models after looking into it found

#LOG:2015-05-28 18:22:02:   Traceback (most recent call last):
  File "/var/lib/arbimon2/jobs/scripts/PatternMatching/train.py", line 433, in <module>
    models[i].validate()
  File "/var/lib/arbimon2/jobs/lib/a2audio/model.py", line 112, in validate
    truePosPredicted =  [predictions[i] for i in presentIndeces]
IndexError: index 20 is out of bounds for axis 0 with size 20
rafael-alvarez commented 9 years ago

Commit a3314fe52ec7b794917a3088e52c11ed04f85bab addresses that error

lazychino commented 9 years ago

hay 3 jobs stuck 2 de mitch y 1 de un usuario (el que reporto el problema original) los de mitch dicen processing el otro dice initializing, pero el job queue esta apagado :confused:

@rafael-alvarez @g-i-o- en lo que pueda ayudar me avisan (revisando codigo, testing or something)

lazychino commented 9 years ago

esta es la grafica de amazon del job-queue no se si el server se apago random, weird screenshot from 2015-05-30 14 41 58

lazychino commented 9 years ago

log last modified 2:16pm

#LOG:2015-05-30 18:15:46:   script started with job id:902
#LOG:2015-05-30 18:15:46:   script started with job id:900
#LOG:2015-05-30 18:15:47:   Unkown model type requested
Traceback (most recent call last):
  File "/var/lib/arbimon2/jobs/scripts/PatternMatching/train.py", line 527, in <module>
    shutil.rmtree(tempFolders+"/training_"+str(jobId))
  File "/usr/lib/python2.7/shutil.py", line 239, in rmtree
    onerror(os.listdir, path, sys.exc_info())
  File "/usr/lib/python2.7/shutil.py", line 237, in rmtree
    names = os.listdir(path)
OSError: [Errno 2] No such file or directory: '/tmp/training_900'
#LOG:2015-05-30 18:15:47:   Pattern Matching (modified Alvarez thesis)
/var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying.
  warn(RuntimeWarning("Cannot provide views on a non-contiguous input "
/var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying.
  warn(RuntimeWarning("Cannot provide views on a non-contiguous input "
/var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying.
  warn(RuntimeWarning("Cannot provide views on a non-contiguous input "
/var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying.
  warn(RuntimeWarning("Cannot provide views on a non-contiguous input "
/var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying.
  warn(RuntimeWarning("Cannot provide views on a non-contiguous input "
/var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying.
  warn(RuntimeWarning("Cannot provide views on a non-contiguous input "
/var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying.
  warn(RuntimeWarning("Cannot provide views on a non-contiguous input "
/var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying.
  warn(RuntimeWarning("Cannot provide views on a non-contiguous input "
#LOG:2015-05-30 18:16:42:   script started with job id:903
#LOG:2015-05-30 18:16:42:   Pattern Matching (modified Alvarez thesis)
/var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying.
  warn(RuntimeWarning("Cannot provide views on a non-contiguous input "
/var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying.
  warn(RuntimeWarning("Cannot provide views on a non-contiguous input "
/var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying.
  warn(RuntimeWarning("Cannot provide views on a non-contiguous input "
/var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying.
  warn(RuntimeWarning("Cannot provide views on a non-contiguous input "
/var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying.
  warn(RuntimeWarning("Cannot provide views on a non-contiguous input "
/var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying.
  warn(RuntimeWarning("Cannot provide views on a non-contiguous input "
/var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying.
  warn(RuntimeWarning("Cannot provide views on a non-contiguous input "
/var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying.
  warn(RuntimeWarning("Cannot provide views on a non-contiguous input "
rafael-alvarez commented 9 years ago

mitch pidio que pusiera el repositorio viejo de los jobs

en ese repo solo se corren model_type = 1

hay que anadir en la base de datos un flag para ponerle disabled a los model_types 2 y 3 ,y hacer que el GUI no ensenhe estos models types (los disabled)

On Sat, May 30, 2015 at 3:02 PM, Pedro A. Melendez <notifications@github.com

wrote:

log de ese momento

LOG:2015-05-30 18:15:46: script started with job id:902

LOG:2015-05-30 18:15:46: script started with job id:900

LOG:2015-05-30 18:15:47: Unkown model type requested

Traceback (most recent call last): File "/var/lib/arbimon2/jobs/scripts/PatternMatching/train.py", line 527, in shutil.rmtree(tempFolders+"/training_"+str(jobId)) File "/usr/lib/python2.7/shutil.py", line 239, in rmtree onerror(os.listdir, path, sys.exc_info()) File "/usr/lib/python2.7/shutil.py", line 237, in rmtree names = os.listdir(path) OSError: [Errno 2] No such file or directory: '/tmp/training_900'

LOG:2015-05-30 18:15:47: Pattern Matching (modified Alvarez thesis)

/var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying. warn(RuntimeWarning("Cannot provide views on a non-contiguous input " /var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying. warn(RuntimeWarning("Cannot provide views on a non-contiguous input " /var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying. warn(RuntimeWarning("Cannot provide views on a non-contiguous input " /var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying. warn(RuntimeWarning("Cannot provide views on a non-contiguous input " /var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying. warn(RuntimeWarning("Cannot provide views on a non-contiguous input " /var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying. warn(RuntimeWarning("Cannot provide views on a non-contiguous input " /var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying. warn(RuntimeWarning("Cannot provide views on a non-contiguous input " /var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying. warn(RuntimeWarning("Cannot provide views on a non-contiguous input "

LOG:2015-05-30 18:16:42: script started with job id:903

LOG:2015-05-30 18:16:42: Pattern Matching (modified Alvarez thesis)

/var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying. warn(RuntimeWarning("Cannot provide views on a non-contiguous input " /var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying. warn(RuntimeWarning("Cannot provide views on a non-contiguous input " /var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying. warn(RuntimeWarning("Cannot provide views on a non-contiguous input " /var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying. warn(RuntimeWarning("Cannot provide views on a non-contiguous input " /var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying. warn(RuntimeWarning("Cannot provide views on a non-contiguous input " /var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying. warn(RuntimeWarning("Cannot provide views on a non-contiguous input " /var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying. warn(RuntimeWarning("Cannot provide views on a non-contiguous input " /var/lib/arbimon2/jobs/.env/local/lib/python2.7/site-packages/skimage/util/shape.py:237: RuntimeWarning: Cannot provide views on a non-contiguous input array without copying. warn(RuntimeWarning("Cannot provide views on a non-contiguous input "

— Reply to this email directly or view it on GitHub https://github.com/Sieve-Analytics/arbimon2-jobs/issues/8#issuecomment-107076088 .

Raf

lazychino commented 9 years ago

el flag lo ponemos en la tabla model_types?? un enable como en job type

rafael-alvarez commented 9 years ago

yes en esa tabla

rafael-alvarez commented 9 years ago

jobs are running, but there is a bug in the training progress update

server

lazychino commented 9 years ago

dale voy a bregar con el app local cuando tenga los cambios ready te aviso

lazychino commented 9 years ago

frontend esta ready lo voy a subir commit fc4dacc041

lazychino commented 9 years ago

este error creo que se resolvio ahora los issues son otros asi que voy a close por ahora