search

rwth-i6 / returnn

The RWTH extensible training framework for universal recurrent neural networks
http://returnn.readthedocs.io/
Other
349 stars 130 forks source link

PicklingError on Windows #19

Closed RashmiPandeyatGit closed 6 years ago

RashmiPandeyatGit commented 7 years ago

By the latest download, i am trying to execute IAM example from demos using config_real but i am getting below error which is related to pickling/multiprocessing :

(Pdb) WARNING (theano.sandbox.cuda): The cuda backend is deprecated and will be removed in the next release (v0.10).  Please switch to the gpuarray backend. You can get more information about how to switch at this URL:
 https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29

Using gpu device 0: GeForce 940MX (CNMeM is enabled with initial size: 70.0% of memory, cuDNN 5110)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\ProgramData\Anaconda2\lib\multiprocessing\forking.py", line 381, in main
    self = load(from_parent)
  File "C:\ProgramData\Anaconda2\lib\pickle.py", line 1384, in load
    return Unpickler(file).load()
  File "C:\ProgramData\Anaconda2\lib\pickle.py", line 864, in load
    dispatch[key](self)
  File "C:\ProgramData\Anaconda2\lib\pickle.py", line 886, in load_eof
    raise EOFError
EOFError

System configuration: Using gpu device 0: GeForce 940MX 2GB (CNMeM is enabled with initial size: 70.0% of memory, cuDNN 5110)

Windows 10 64bit Intel Core i7 7th generation GeForce 940MX 2GB 8GB RAM

albertz commented 7 years ago

Usually that is the result of another error/exception which happened earlier. Can you post your full output? Also maybe increase log_verbosity in the config, maybe to 5.

RashmiPandeyatGit commented 7 years ago

Please find the complete error below... I have used pdb to debug the code step by step. Also in windows command prompt I am unable to understand the error/exception. 

(Pdb)
> d:\returnn-master\device.py(378)_startProc()
-> env_update=env_update)
(Pdb)
PicklingError: Pickling..._error",)
> d:\returnn-master\device.py(378)_startProc()
-> env_update=env_update)
(Pdb)
--Return--
> d:\returnn-master\device.py(378)_startProc()->None
-> env_update=env_update)
(Pdb)
PicklingError: Pickling..._error",)
> d:\returnn-master\device.py(332)startProc()
-> self._startProc(*args, **kwargs)
(Pdb)
> d:\returnn-master\device.py(333)startProc()
-> except BaseException:
(Pdb)
> d:\returnn-master\device.py(334)startProc()
-> try:
(Pdb)
> d:\returnn-master\device.py(335)startProc()
-> sys.excepthook(*sys.exc_info())
(Pdb)
EXCEPTION
Traceback (most recent call last):
File "D:\returnn-master\Device.py", line 332, in startProc
 line: self._startProc(args, kwargs)
 locals:
self =  <Device.Device object at 0x000000001BB1CDA0>
self._startProc =  <bound method Device._startProc of <Device.Device object at 0x000000001BB1CDA0>>
args =  ('gpuZ',)
kwargs =  {}
File "D:\returnn-master\Device.py", line 378, in _startProc
 line: self.proc = AsyncTask(
func=self.process,
name="Device %s proc" % self.name,
mustExec=True,
env_update=env_update)
 locals:
self =  <Device.Device object at 0x000000001BB1CDA0>
self.proc =  !AttributeError: 'Device' object has no attribute 'proc'
AsyncTask =  <class TaskSystem.AsyncTask at 0x0000000004241EE8>
func = 
self.process =  <bound method Device.process of <Device.Device object at 0x000000001BB1CDA0>>
name = 
self.name =  'gpuX'
mustExec = 
env_update =  {'THEANO_FLAGS': 'compiledir_format=compiledir_%(platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s--dev-gpuZ,device=gpu,force_device=True'}
File "D:\returnn-master\TaskSystem.py", line 1277, in __init__
 line: self.proc.start()
 locals:
self =  <TaskSystem.AsyncTask instance at 0x000000001BB67808>
self.proc =  <Process(Device gpuX proc worker process, initial daemon)>
self.proc.start =  <bound method Process.start of <Process(Device gpuX proc worker process, initial daemon)>>
File "C:\ProgramData\Anaconda2\lib\multiprocessing\process.py", line 136, in start
 line: self._popen = Popen(self)
 locals:
self =  <Process(Device gpuX proc worker process, initial daemon)>
self._popen =  None
Popen =  <TaskSystem.Win32_mp_Popen_wrapper instance at 0x000000001BB679C8>
File "D:\returnn-master\TaskSystem.py", line 1207, in __call__
 line: return self.Popen(process_obj, self.env)
 locals:
self =  <TaskSystem.Win32_mp_Popen_wrapper instance at 0x000000001BB679C8>
self.Popen =  <class 'TaskSystem.Popen'>
process_obj =  <Process(Device gpuX proc worker process, initial daemon)>
self.env =  {'TMP': 'C:\Users\ADMINI~1\AppData\Local\Temp', 'COMPUTERNAME': 'MLX-KRC-06', 'VS140COMNTOOLS': 'C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\Tools\', 'VS110COMNTOOLS': 'C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Tools\', 'USERDOMAIN': 'MLX-KRC-06', ..., len = 48
File "D:\returnn-master\TaskSystem.py", line 1200, in __init__
 line: dump(process_obj, to_child, HIGHEST_PROTOCOL)
 locals:
dump =  <function dump at 0x0000000004265F98>
process_obj =  <Process(Device gpuX proc worker process, initial daemon)>
to_child =  <closed file '', mode 'wb' at 0x000000001BA5CA50>
HIGHEST_PROTOCOL =  2
File "C:\ProgramData\Anaconda2\lib\multiprocessing\forking.py", line 202, in dump
 line: ForkingPickler(file, protocol).dump(obj)
 locals:
ForkingPickler =  <class multiprocessing.forking.ForkingPickler at 0x0000000004241E28>
file =  <closed file '', mode 'wb' at 0x000000001BA5CA50>
protocol =  2
dump =  <function dump at 0x0000000004265F98>
obj =  <Process(Device gpuX proc worker process, initial daemon)>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 224, in dump
 line: self.save(obj)
 locals:
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
self.save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
obj =  <Process(Device gpuX proc worker process, initial daemon)>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 331, in save
 line: self.save_reduce(obj=obj, rv)
 locals:
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
self.save_reduce =  <bound method ForkingPickler.save_reduce of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
obj =  <Process(Device gpuX proc worker process, initial daemon)>
rv =  (<function newobj at 0x0000000003516DD8>, (<class 'multiprocessing.process.Process'>,), {'_daemonic': True, '_target': <function funcCall at 0x0000000004264358>, '_args': ((<class TaskSystem.AsyncTask at 0x0000000004241EE8>, '_asyncCall'), (<TaskSystem.AsyncTask instance at 0x000000001BB67808...
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 425, in save_reduce
 line: save(state)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
state =  {'_daemonic': True, '_target': <function funcCall at 0x0000000004264358>, '_args': ((<class TaskSystem.AsyncTask at 0x0000000004241EE8>, '_asyncCall'), (<TaskSystem.AsyncTask instance at 0x000000001BB67808>,)), '_tempdir': None, '_name': 'Device gpuX proc worker process', '_authkey': "\x1f[M\xb2..., len = 11
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_dict at 0x000000000423B4A8>
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
obj =  {'_daemonic': True, '_target': <function funcCall at 0x0000000004264358>, '_args': ((<class TaskSystem.AsyncTask at 0x0000000004241EE8>, '_asyncCall'), (<TaskSystem.AsyncTask instance at 0x000000001BB67808>,)), '_tempdir': None, '_name': 'Device gpuX proc worker process', '_authkey': "\x1f[M\xb2..., len = 11
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 655, in save_dict
 line: self._batch_setitems(obj.iteritems())
 locals:
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
self._batch_setitems =  <bound method ForkingPickler._batch_setitems of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
obj =  {'_daemonic': True, '_target': <function funcCall at 0x0000000004264358>, '_args': ((<class TaskSystem.AsyncTask at 0x0000000004241EE8>, '_asyncCall'), (<TaskSystem.AsyncTask instance at 0x000000001BB67808>,)), '_tempdir': None, '_name': 'Device gpuX proc worker process', 'authkey': "\x1f[M\xb2..., len = 11
obj.iteritems =  <built-in method iteritems of dict object at 0x000000001BB698C8>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 687, in _batch_setitems
 line: save(v)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
v =  ((<class TaskSystem.AsyncTask at 0x0000000004241EE8>, 'asyncCall'), (<TaskSystem.AsyncTask instance at 0x000000001BB67808>,))
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_tuple at 0x000000000423B2E8>
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
obj =  ((<class TaskSystem.AsyncTask at 0x0000000004241EE8>, 'asyncCall'), (<TaskSystem.AsyncTask instance at 0x000000001BB67808>,))
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 554, in save_tuple
 line: save(element)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
element =  (<TaskSystem.AsyncTask instance at 0x000000001BB67808>,)
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_tuple at 0x000000000423B2E8>
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
obj =  (<TaskSystem.AsyncTask instance at 0x000000001BB67808>,)
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 554, in save_tuple
 line: save(element)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
element =  <TaskSystem.AsyncTask instance at 0x000000001BB67808>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_inst at 0x000000000423B588>
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
obj =  <TaskSystem.AsyncTask instance at 0x000000001BB67808>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 731, in save_inst
 line: save(stuff)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
stuff =  {'env_update': {'THEANO_FLAGS': 'compiledir_format=compiledir%(platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s--dev-gpuZ,device=gpu,force_device=True'}, 'parent_pid': 11564, 'name': 'Device gpuX proc', 'Process': <class 'multiprocessing.process.Process'>, 'parent_conn': <ExecingP..., len = 11
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_dict at 0x000000000423B4A8>
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
obj =  {'env_update': {'THEANO_FLAGS': 'compiledir_format=compiledir%(platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s--dev-gpuZ,device=gpu,force_device=True'}, 'parent_pid': 11564, 'name': 'Device gpuX proc', 'Process': <class 'multiprocessing.process.Process'>, 'parent_conn': <ExecingP..., len = 11
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 655, in save_dict
 line: self._batch_setitems(obj.iteritems())
 locals:
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
self._batch_setitems =  <bound method ForkingPickler._batch_setitems of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
obj =  {'env_update': {'THEANO_FLAGS': 'compiledir_format=compiledir%(platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s--dev-gpuZ,device=gpu,force_device=True'}, 'parent_pid': 11564, 'name': 'Device gpuX proc', 'Process': <class 'multiprocessing.process.Process'>, 'parent_conn': <ExecingP..., len = 11
obj.iteritems =  <built-in method iteritems of dict object at 0x000000001BB696A8>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 687, in _batch_setitems
 line: save(v)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
v =  <bound method Device.process of <Device.Device object at 0x000000001BB1CDA0>>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function dispatcher at 0x0000000004265C88>
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
obj =  <bound method Device.process of <Device.Device object at 0x000000001BB1CDA0>>
File "C:\ProgramData\Anaconda2\lib\multiprocessing\forking.py", line 67, in dispatcher
 line: self.save_reduce(obj=obj, rv)
 locals:
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
self.save_reduce =  <bound method ForkingPickler.save_reduce of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
obj =  <bound method Device.process of <Device.Device object at 0x000000001BB1CDA0>>
rv =  (<built-in function getattr>, (<Device.Device object at 0x000000001BB1CDA0>, 'process'))
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 401, in save_reduce
 line: save(args)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
args =  (<Device.Device object at 0x000000001BB1CDA0>, 'process')
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_tuple at 0x000000000423B2E8>
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
obj =  (<Device.Device object at 0x000000001BB1CDA0>, 'process')
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 554, in save_tuple
 line: save(element)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
element =  <Device.Device object at 0x000000001BB1CDA0>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 331, in save
 line: self.save_reduce(obj=obj, rv)
 locals:
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
self.save_reduce =  <bound method ForkingPickler.save_reduce of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
obj =  <Device.Device object at 0x000000001BB1CDA0>
rv =  (<function newobj at 0x0000000003516DD8>, (<class 'Device.Device'>,), {'compute_total_time': 0, 'num_updates': 0, 'name': 'gpuX', 'num_frames': NumbersDict(0), 'run_called_count': 0, 'update_specs': {'layers': [], 'block_size': 0, 'update_params': {}, 'update_rule': 'global'}, 'main_pid': 115...
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 425, in save_reduce
 line: save(state)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
state =  {'compute_total_time': 0, 'num_updates': 0, 'name': 'gpuX', 'num_frames': NumbersDict(0), 'run_called_count': 0, 'update_specs': {'layers': [], 'block_size': 0, 'update_params': {}, 'update_rule': 'global'}, 'main_pid': 11564, 'wait_for_result_call': False, 'train_outputs_format': None, 'epoch': ..., len = 18
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_dict at 0x000000000423B4A8>
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
obj =  {'compute_total_time': 0, 'num_updates': 0, 'name': 'gpuX', 'num_frames': NumbersDict(0), 'run_called_count': 0, 'update_specs': {'layers': [], 'block_size': 0, 'update_params': {}, 'update_rule': 'global'}, 'main_pid': 11564, 'wait_for_result_call': False, 'train_outputs_format': None, 'epoch': ..., len = 18
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 655, in save_dict
 line: self._batch_setitems(obj.iteritems())
 locals:
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
self._batch_setitems =  <bound method ForkingPickler._batch_setitems of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
obj =  {'compute_total_time': 0, 'num_updates': 0, 'name': 'gpuX', 'num_frames': NumbersDict(0), 'run_called_count': 0, 'update_specs': {'layers': [], 'block_size': 0, 'update_params': {}, 'update_rule': 'global'}, 'main_pid': 11564, 'wait_for_result_call': False, 'train_outputs_format': None, 'epoch': ..., len = 18
obj.iteritems =  <built-in method iteritems of dict object at 0x000000001BB5DBF8>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 687, in _batch_setitems
 line: save(v)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
v =  NumbersDict(0)
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_inst at 0x000000000423B588>
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
obj =  NumbersDict(0)
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 731, in save_inst
 line: save(stuff)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
stuff =  {'max': <function __max_error at 0x000000001B0D2358>, 'dict': {}, 'value': 0}
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_dict at 0x000000000423B4A8>
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
obj =  {'max': <function __max_error at 0x000000001B0D2358>, 'dict': {}, 'value': 0}
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 655, in save_dict
 line: self._batch_setitems(obj.iteritems())
 locals:
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
self._batch_setitems =  <bound method ForkingPickler._batch_setitems of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
obj =  {'max': <function __max_error at 0x000000001B0D2358>, 'dict': {}, 'value': 0}
obj.iteritems =  <built-in method iteritems of dict object at 0x000000001BB5DD08>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 687, in _batch_setitems
 line: save(v)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
v =  <function __max_error at 0x000000001B0D2358>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_global at 0x000000000423B5F8>
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
obj =  <function __max_error at 0x000000001B0D2358>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 754, in save_global
 line: raise PicklingError(
"Can't pickle %r: it's not found as %s.%s" %
(obj, module, name))
 locals:
PicklingError =  <class 'pickle.PicklingError'>
obj =  <function __max_error at 0x000000001B0D2358>
module =  'Util'
name =  '__max_error', len = 11
PicklingError: Can't pickle <function __max_error at 0x000000001B0D2358>: it's not found as Util.__max_error
KeyboardInterrupt
EXCEPTION
Traceback (most recent call last):
File "......\rnn.py", line 526, in main
 line: init(commandLineOptions=argv[1:])
 locals:
init =  <function init at 0x000000001BB162E8>
commandLineOptions = 
argv =  ['..\..\..\rnn.py', 'config_real'], _[0]: {len = 15}
File "......\rnn.py", line 339, in init
 line: devices = initDevices()
 locals:
devices = 
initDevices =  <function initDevices at 0x000000001BB17F28>
File "......\rnn.py", line 152, in initDevices
 line: time.sleep(0.25)
 locals:
time =  <module 'time' (built-in)>
time.sleep =  <built-in function sleep>
KeyboardInterrupt
Quitting

az@azmacbookpro-2 ~> 
az@azmacbookpro-2 ~> rm txt
az@azmacbookpro-2 ~> touch txt
az@azmacbookpro-2 ~> open txt
az@azmacbookpro-2 ~> cat txt | sed -r 's/�\[[0-9]+(;[0-9]+)?[Hhlrm]//g'
EXCEPTION
Traceback (most recent call last):
File "D:\returnn-master\Device.py", line 332, in startProc
 line: self._startProc(args, *kwargs)
 locals:
self =  <Device.Device object at 0x0000000062BBFFD0>
self._startProc =  <bound method Device._startProc of <Device.Device object at 0x0000000062BBFFD0>>
args =  ('gpu0',)
kwargs =  {}
File "D:\returnn-master\Device.py", line 378, in _startProc
 line: self.proc = AsyncTask(
func=self.process,
name="Device %s proc" % self.name,
mustExec=True,
env_update=env_update)
 locals:
self =  <Device.Device object at 0x0000000062BBFFD0>
self.proc =  !AttributeError: 'Device' object has no attribute 'proc'
AsyncTask =  <class TaskSystem.AsyncTask at 0x00000000042CA408>
func = 
self.process =  <bound method Device.process of <Device.Device object at 0x0000000062BBFFD0>>
name = 
self.name =  'gpu0'
mustExec = 
env_update =  {'THEANO_FLAGS': 'compiledir_format=compiledir%(platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s--dev-gpu0,device=gpu0,force_device=True'}
File "D:\returnn-master\TaskSystem.py", line 1246, in __init
 line: self.proc.start()
 locals:
self =  <TaskSystem.AsyncTask instance at 0x0000000062D58248>
self.proc =  <Process(Device gpu0 proc worker process, initial daemon)>
self.proc.start =  <bound method Process.start of <Process(Device gpu0 proc worker process, initial daemon)>>
File "C:\ProgramData\Anaconda2\lib\multiprocessing\process.py", line 130, in start
 line: self._popen = Popen(self)
 locals:
self =  <Process(Device gpu0 proc worker process, initial daemon)>
self._popen =  None
Popen =  <TaskSystem.Win32_mp_Popen_wrapper instance at 0x0000000062D58B88>
File "D:\returnn-master\TaskSystem.py", line 1199, in __call
 line: return self.Popen(process_obj, self.env)
 locals:
self =  <TaskSystem.Win32_mp_Popen_wrapper instance at 0x0000000062D58B88>
self.Popen =  <class 'TaskSystem.Popen'>
process_obj =  <Process(Device gpu0 proc worker process, initial daemon)>
self.env =  {'TMP': 'C:\Users\ADMINI~1\AppData\Local\Temp', 'COMPUTERNAME': 'MLX-KRC-06', 'VS140COMNTOOLS': 'C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\Tools\', 'VS110COMNTOOLS': 'C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Tools\', 'USERDOMAIN': 'MLX-KRC-06', ..., len = 48
File "D:\returnn-master\TaskSystem.py", line 1193, in __init_
 line: dump(process_obj, to_child, HIGHEST_PROTOCOL)
 locals:
dump =  <function dump at 0x00000000042CC5F8>
process_obj =  <Process(Device gpu0 proc worker process, initial daemon)>
to_child =  <closed file '', mode 'wb' at 0x0000000062CE89C0>
HIGHEST_PROTOCOL =  2
File "C:\ProgramData\Anaconda2\lib\multiprocessing\forking.py", line 199, in dump
 line: ForkingPickler(file, protocol).dump(obj)
 locals:
ForkingPickler =  <class multiprocessing.forking.ForkingPickler at 0x00000000042CA348>
file =  <closed file '', mode 'wb' at 0x0000000062CE89C0>
protocol =  2
dump =  <function dump at 0x00000000042CC5F8>
obj =  <Process(Device gpu0 proc worker process, initial daemon)>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 224, in dump
 line: self.save(obj)
 locals:
self =  <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>
self.save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>>
obj =  <Process(Device gpu0 proc worker process, initial daemon)>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 331, in save
 line: self.save_reduce(obj=obj, rv)
 locals:
self =  <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>
self.save_reduce =  <bound method ForkingPickler.save_reduce of <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>>
obj =  <Process(Device gpu0 proc worker process, initial daemon)>
rv =  (<function newobj at 0x00000000035B6DD8>, (<class 'multiprocessing.process.Process'>,), {'_daemonic': True, '_target': <function funcCall at 0x00000000042C5978>, '_args': ((<class TaskSystem.AsyncTask at 0x00000000042CA408>, '_asyncCall'), (<TaskSystem.AsyncTask instance at 0x0000000062D58248...
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 425, in save_reduce
 line: save(state)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>>
state =  {'_daemonic': True, '_target': <function funcCall at 0x00000000042C5978>, '_args': ((<class TaskSystem.AsyncTask at 0x00000000042CA408>, '_asyncCall'), (<TaskSystem.AsyncTask instance at 0x0000000062D58248>,)), '_tempdir': None, '_name': 'Device gpu0 proc worker process', '_authkey': '\xe9\x10\xb..., len = 11
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_dict at 0x000000000429CAC8>
self =  <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>
obj =  {'_daemonic': True, '_target': <function funcCall at 0x00000000042C5978>, '_args': ((<class TaskSystem.AsyncTask at 0x00000000042CA408>, '_asyncCall'), (<TaskSystem.AsyncTask instance at 0x0000000062D58248>,)), '_tempdir': None, '_name': 'Device gpu0 proc worker process', '_authkey': '\xe9\x10\xb..., len = 11
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 655, in save_dict
 line: self._batch_setitems(obj.iteritems())
 locals:
self =  <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>
self._batch_setitems =  <bound method ForkingPickler._batch_setitems of <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>>
obj =  {'_daemonic': True, '_target': <function funcCall at 0x00000000042C5978>, '_args': ((<class TaskSystem.AsyncTask at 0x00000000042CA408>, '_asyncCall'), (<TaskSystem.AsyncTask instance at 0x0000000062D58248>,)), '_tempdir': None, '_name': 'Device gpu0 proc worker process', 'authkey': '\xe9\x10\xb..., len = 11
obj.iteritems =  <built-in method iteritems of dict object at 0x0000000062D52BF8>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 687, in _batch_setitems
 line: save(v)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>>
v =  ((<class TaskSystem.AsyncTask at 0x00000000042CA408>, 'asyncCall'), (<TaskSystem.AsyncTask instance at 0x0000000062D58248>,))
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_tuple at 0x000000000429C908>
self =  <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>
obj =  ((<class TaskSystem.AsyncTask at 0x00000000042CA408>, 'asyncCall'), (<TaskSystem.AsyncTask instance at 0x0000000062D58248>,))
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 554, in save_tuple
 line: save(element)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>>
element =  (<TaskSystem.AsyncTask instance at 0x0000000062D58248>,)
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_tuple at 0x000000000429C908>
self =  <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>
obj =  (<TaskSystem.AsyncTask instance at 0x0000000062D58248>,)
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 554, in save_tuple
 line: save(element)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>>
element =  <TaskSystem.AsyncTask instance at 0x0000000062D58248>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_inst at 0x000000000429CBA8>
self =  <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>
obj =  <TaskSystem.AsyncTask instance at 0x0000000062D58248>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 731, in save_inst
 line: save(stuff)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>>
stuff =  {'env_update': {'THEANO_FLAGS': 'compiledir_format=compiledir%(platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s--dev-gpu0,device=gpu0,force_device=True'}, 'parent_pid': 11460, 'name': 'Device gpu0 proc', 'Process': <class 'multiprocessing.process.Process'>, 'parent_conn': <Execing..., len = 11
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_dict at 0x000000000429CAC8>
self =  <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>
obj =  {'env_update': {'THEANO_FLAGS': 'compiledir_format=compiledir%(platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s--dev-gpu0,device=gpu0,force_device=True'}, 'parent_pid': 11460, 'name': 'Device gpu0 proc', 'Process': <class 'multiprocessing.process.Process'>, 'parent_conn': <Execing..., len = 11
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 655, in save_dict
 line: self._batch_setitems(obj.iteritems())
 locals:
self =  <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>
self._batch_setitems =  <bound method ForkingPickler._batch_setitems of <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>>
obj =  {'env_update': {'THEANO_FLAGS': 'compiledir_format=compiledir%(platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s--dev-gpu0,device=gpu0,force_device=True'}, 'parent_pid': 11460, 'name': 'Device gpu0 proc', 'Process': <class 'multiprocessing.process.Process'>, 'parent_conn': <Execing..., len = 11
obj.iteritems =  <built-in method iteritems of dict object at 0x0000000062D52AE8>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 687, in _batch_setitems
 line: save(v)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>>
v =  <bound method Device.process of <Device.Device object at 0x0000000062BBFFD0>>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function dispatcher at 0x00000000042CC2E8>
self =  <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>
obj =  <bound method Device.process of <Device.Device object at 0x0000000062BBFFD0>>
File "C:\ProgramData\Anaconda2\lib\multiprocessing\forking.py", line 67, in dispatcher
 line: self.save_reduce(obj=obj, rv)
 locals:
self =  <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>
self.save_reduce =  <bound method ForkingPickler.save_reduce of <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>>
obj =  <bound method Device.process of <Device.Device object at 0x0000000062BBFFD0>>
rv =  (<built-in function getattr>, (<Device.Device object at 0x0000000062BBFFD0>, 'process'))
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 401, in save_reduce
 line: save(args)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>>
args =  (<Device.Device object at 0x0000000062BBFFD0>, 'process')
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_tuple at 0x000000000429C908>
self =  <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>
obj =  (<Device.Device object at 0x0000000062BBFFD0>, 'process')
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 554, in save_tuple
 line: save(element)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>>
element =  <Device.Device object at 0x0000000062BBFFD0>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 331, in save
 line: self.save_reduce(obj=obj, *rv)
 locals:
self =  <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>
self.save_reduce =  <bound method ForkingPickler.save_reduce of <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>>
obj =  <Device.Device object at 0x0000000062BBFFD0>
rv =  (<function newobj at 0x00000000035B6DD8>, (<class 'Device.Device'>,), {'compute_total_time': 0, 'num_updates': 0, 'name': 'gpu0', 'num_frames': NumbersDict(0), 'run_called_count': 0, 'update_specs': {'layers': [], 'block_size': 0, 'update_params': {}, 'update_rule': 'global'}, 'main_pid': 114...
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 425, in save_reduce
 line: save(state)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>>
state =  {'compute_total_time': 0, 'num_updates': 0, 'name': 'gpu0', 'num_frames': NumbersDict(0), 'run_called_count': 0, 'update_specs': {'layers': [], 'block_size': 0, 'update_params': {}, 'update_rule': 'global'}, 'main_pid': 11460, 'wait_for_result_call': False, 'train_outputs_format': None, 'epoch': ..., len = 18
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_dict at 0x000000000429CAC8>
self =  <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>
obj =  {'compute_total_time': 0, 'num_updates': 0, 'name': 'gpu0', 'num_frames': NumbersDict(0), 'run_called_count': 0, 'update_specs': {'layers': [], 'block_size': 0, 'update_params': {}, 'update_rule': 'global'}, 'main_pid': 11460, 'wait_for_result_call': False, 'train_outputs_format': None, 'epoch': ..., len = 18
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 655, in save_dict
 line: self._batch_setitems(obj.iteritems())
 locals:
self =  <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>
self._batch_setitems =  <bound method ForkingPickler._batch_setitems of <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>>
obj =  {'compute_total_time': 0, 'num_updates': 0, 'name': 'gpu0', 'num_frames': NumbersDict(0), 'run_called_count': 0, 'update_specs': {'layers': [], 'block_size': 0, 'update_params': {}, 'update_rule': 'global'}, 'main_pid': 11460, 'wait_for_result_call': False, 'train_outputs_format': None, 'epoch': ..., len = 18
obj.iteritems =  <built-in method iteritems of dict object at 0x0000000062C0E6A8>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 687, in _batch_setitems
 line: save(v)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>>
v =  NumbersDict(0)
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_inst at 0x000000000429CBA8>
self =  <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>
obj =  NumbersDict(0)
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 731, in save_inst
 line: save(stuff)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>>
stuff =  {'max': <function __max_error at 0x000000001D8CEAC8>, 'dict': {}, 'value': 0}
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_dict at 0x000000000429CAC8>
self =  <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>
obj =  {'max': <function __max_error at 0x000000001D8CEAC8>, 'dict': {}, 'value': 0}
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 655, in save_dict
 line: self._batch_setitems(obj.iteritems())
 locals:
self =  <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>
self._batch_setitems =  <bound method ForkingPickler._batch_setitems of <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>>
obj =  {'max': <function __max_error at 0x000000001D8CEAC8>, 'dict': {}, 'value': 0}
obj.iteritems =  <built-in method iteritems of dict object at 0x0000000062C0E7B8>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 687, in _batch_setitems
 line: save(v)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>>
v =  <function __max_error at 0x000000001D8CEAC8>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_global at 0x000000000429CC18>
self =  <multiprocessing.forking.ForkingPickler instance at 0x0000000062D58C08>
obj =  <function __max_error at 0x000000001D8CEAC8>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 754, in save_global
 line: raise PicklingError(
"Can't pickle %r: it's not found as %s.%s" %
(obj, module, name))
 locals:
PicklingError =  <class 'pickle.PicklingError'>
obj =  <function __max_error at 0x000000001D8CEAC8>
module =  'Util'
name =  '__max_error', len = 11
PicklingError: Can't pickle <function __max_error at 0x000000001D8CEAC8>: it's not found as Util.__max_error
RashmiPandeyatGit commented 7 years ago

This is how my config_real looks like

{ "task": "train", "device": "gpu", "multiprocessing": false, "batching": "default", "cache_size": "2G", "max_seqs": 10, "batch_size": 600, "nadam": 1, "learning_rate": 0.0005, "learning_rates": "{1: 0.0005, 25: 0.0003, 35: 0.0001}", "num_epochs": 2, "update_batch_size": 2000000, "reinit": 0, "log_verbosity": 5, "train": "features/raw/train.1.h5,features/raw/train.2.h5", "dev": "features/raw/train_valid.h5", "start_epoch": 1, "num_inputs": 1, "num_outputs": {"data": [1,2], "classes": [79,1], "sizes": [2,1]}, "network": { "classes_source": {"class": "source", "data_key": "sizes", "from": [], "n_out": 2, "dtype":"int32"}, "1Dto2D": {"class": "1Dto2D", "from" : ["data", "classes_source"] }, "conv0": { "class" : "conv2", "n_features" : 15, "filter" : [3, 3], "pool_size" : [2, 2], "from" : ["1Dto2D"] }, "mdlstm0": { "class" : "mdlstm", "n_out" : 30, "dropout": 0.25, "from" : ["conv0"] }, "conv1" : { "class" : "conv2", "n_features" : 45, "dropout": 0.25, "filter" : [3, 3], "pool_size" : [2, 2], "from" : ["mdlstm0"] }, "mdlstm1": { "class" : "mdlstm", "n_out" : 60, "dropout": 0.25, "from" : ["conv1"] }, "conv2" : { "class" : "conv2", "n_features" : 75, "dropout": 0.25, "filter" : [3, 3], "pool_size" : [2, 2], "from" : ["mdlstm1"] }, "mdlstm2": { "class" : "mdlstm", "n_out" : 90, "dropout": 0.25, "from" : ["conv2"] }, "conv3" : { "class" : "conv2", "n_features" : 105, "dropout": 0.25, "filter" : [3, 3], "pool_size" : [1, 1], "from" : ["mdlstm2"] }, "mdlstm3": { "class" : "mdlstm", "n_out" : 120, "dropout": 0.25, "from" : ["conv3"] }, "conv4" : { "class" : "conv2", "n_features" : 105, "dropout": 0.25, "filter" : [3, 3], "pool_size" : [1, 1], "from" : ["mdlstm3"] }, "mdlstm4": { "class" : "mdlstm", "n_out" : 120, "dropout": 0.25, "from" : ["conv4"] }, "output": { "class" : "softmax", "loss": "ctc", "dropout": 0.25, "from" : ["mdlstm4"] } },

"ctc_prior_file": "priors/mdlstm_real", "model": "models/mdlstm_real", "log": "log/mdlstm_real" }

pvoigtlaender commented 7 years ago

Hi,

can you please try to set "multiprocessing": true, and run again?

RashmiPandeyatGit commented 7 years ago

it is already set=true.

pvoigtlaender commented 7 years ago

Are you sure? In the config_real you posted, it says "multiprocessing": false, We didn't use the multiprocessing false mode much internally and it is most probably no longer working, so you should really make sure, that multiprocessing is set to true.

Do you know, why the log from above contains so many (atleast under my system) unreadable characters?

RashmiPandeyatGit commented 7 years ago

we changed later on after reading few issues. Now it is set=true. I am not getting why this kind of unreadable characters are coming as error. In windows will this framework will work or not?. There should be requirement.txt maintain for windows as well as Ubuntu system.

RashmiPandeyatGit commented 7 years ago

Current Error (ANSI escape code removed via sed -r 's/�\[[0-9]+(;[0-9]+)?[Hhlrm]//g'):

D:\returnn-master\demos\mdlstm\IAM>python ..\..\..\rnn.py config_real
WARNING (theano.sandbox.cuda): The cuda backend is deprecated and will be removed in the next release (v0.10).  Please switch to the gpuarray backend. You can get more information about how to switch at this URL:
 https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29

Using gpu device 0: GeForce 940MX (CNMeM is enabled with initial size: 50.0% of memory, cuDNN 5110)
CRNN starting up, version unknown(git exception: WindowsError(2, 'The system cannot find the file specified')), pid 11564, cwd D:\returnn-master\demos\mdlstm\IAM
CRNN command line options: ['config_real']
Theano: 0.9.0.dev-c697eeab84... (<not-under-git> in C:\ProgramData\Anaconda2\lib\site-packages\theano)
CUDA already initialized in proc 11564
EXCEPTION
Traceback (most recent call last):
File "D:\returnn-master\Device.py", line 332, in startProc
 line: self._startProc(args, kwargs)
 locals:
self =  <Device.Device object at 0x000000001BB1CDA0>
self._startProc =  <bound method Device._startProc of <Device.Device object at 0x000000001BB1CDA0>>
args =  ('gpuZ',)
kwargs =  {}
File "D:\returnn-master\Device.py", line 378, in _startProc
 line: self.proc = AsyncTask(
func=self.process,
name="Device %s proc" % self.name,
mustExec=True,
env_update=env_update)
 locals:
self =  <Device.Device object at 0x000000001BB1CDA0>
self.proc =  !AttributeError: 'Device' object has no attribute 'proc'
AsyncTask =  <class TaskSystem.AsyncTask at 0x0000000004241EE8>
func = 
self.process =  <bound method Device.process of <Device.Device object at 0x000000001BB1CDA0>>
name = 
self.name =  'gpuX'
mustExec = 
env_update =  {'THEANO_FLAGS': 'compiledir_format=compiledir_%(platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s--dev-gpuZ,device=gpu,force_device=True'}
File "D:\returnn-master\TaskSystem.py", line 1277, in __init__
 line: self.proc.start()
 locals:
self =  <TaskSystem.AsyncTask instance at 0x000000001BB67808>
self.proc =  <Process(Device gpuX proc worker process, initial daemon)>
self.proc.start =  <bound method Process.start of <Process(Device gpuX proc worker process, initial daemon)>>
File "C:\ProgramData\Anaconda2\lib\multiprocessing\process.py", line 136, in start
 line: self._popen = Popen(self)
 locals:
self =  <Process(Device gpuX proc worker process, initial daemon)>
self._popen =  None
Popen =  <TaskSystem.Win32_mp_Popen_wrapper instance at 0x000000001BB679C8>
File "D:\returnn-master\TaskSystem.py", line 1207, in __call__
 line: return self.Popen(process_obj, self.env)
 locals:
self =  <TaskSystem.Win32_mp_Popen_wrapper instance at 0x000000001BB679C8>
self.Popen =  <class 'TaskSystem.Popen'>
process_obj =  <Process(Device gpuX proc worker process, initial daemon)>
self.env =  {'TMP': 'C:\Users\ADMINI~1\AppData\Local\Temp', 'COMPUTERNAME': 'MLX-KRC-06', 'VS140COMNTOOLS': 'C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\Tools\', 'VS110COMNTOOLS': 'C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Tools\', 'USERDOMAIN': 'MLX-KRC-06', ..., len = 48
File "D:\returnn-master\TaskSystem.py", line 1200, in __init__
 line: dump(process_obj, to_child, HIGHEST_PROTOCOL)
 locals:
dump =  <function dump at 0x0000000004265F98>
process_obj =  <Process(Device gpuX proc worker process, initial daemon)>
to_child =  <closed file '', mode 'wb' at 0x000000001BA5CA50>
HIGHEST_PROTOCOL =  2
File "C:\ProgramData\Anaconda2\lib\multiprocessing\forking.py", line 202, in dump
 line: ForkingPickler(file, protocol).dump(obj)
 locals:
ForkingPickler =  <class multiprocessing.forking.ForkingPickler at 0x0000000004241E28>
file =  <closed file '', mode 'wb' at 0x000000001BA5CA50>
protocol =  2
dump =  <function dump at 0x0000000004265F98>
obj =  <Process(Device gpuX proc worker process, initial daemon)>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 224, in dump
 line: self.save(obj)
 locals:
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
self.save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
obj =  <Process(Device gpuX proc worker process, initial daemon)>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 331, in save
 line: self.save_reduce(obj=obj, rv)
 locals:
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
self.save_reduce =  <bound method ForkingPickler.save_reduce of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
obj =  <Process(Device gpuX proc worker process, initial daemon)>
rv =  (<function newobj at 0x0000000003516DD8>, (<class 'multiprocessing.process.Process'>,), {'_daemonic': True, '_target': <function funcCall at 0x0000000004264358>, '_args': ((<class TaskSystem.AsyncTask at 0x0000000004241EE8>, '_asyncCall'), (<TaskSystem.AsyncTask instance at 0x000000001BB67808...
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 425, in save_reduce
 line: save(state)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
state =  {'_daemonic': True, '_target': <function funcCall at 0x0000000004264358>, '_args': ((<class TaskSystem.AsyncTask at 0x0000000004241EE8>, '_asyncCall'), (<TaskSystem.AsyncTask instance at 0x000000001BB67808>,)), '_tempdir': None, '_name': 'Device gpuX proc worker process', '_authkey': "\x1f[M\xb2..., len = 11
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_dict at 0x000000000423B4A8>
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
obj =  {'_daemonic': True, '_target': <function funcCall at 0x0000000004264358>, '_args': ((<class TaskSystem.AsyncTask at 0x0000000004241EE8>, '_asyncCall'), (<TaskSystem.AsyncTask instance at 0x000000001BB67808>,)), '_tempdir': None, '_name': 'Device gpuX proc worker process', '_authkey': "\x1f[M\xb2..., len = 11
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 655, in save_dict
 line: self._batch_setitems(obj.iteritems())
 locals:
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
self._batch_setitems =  <bound method ForkingPickler._batch_setitems of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
obj =  {'_daemonic': True, '_target': <function funcCall at 0x0000000004264358>, '_args': ((<class TaskSystem.AsyncTask at 0x0000000004241EE8>, '_asyncCall'), (<TaskSystem.AsyncTask instance at 0x000000001BB67808>,)), '_tempdir': None, '_name': 'Device gpuX proc worker process', 'authkey': "\x1f[M\xb2..., len = 11
obj.iteritems =  <built-in method iteritems of dict object at 0x000000001BB698C8>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 687, in _batch_setitems
 line: save(v)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
v =  ((<class TaskSystem.AsyncTask at 0x0000000004241EE8>, 'asyncCall'), (<TaskSystem.AsyncTask instance at 0x000000001BB67808>,))
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_tuple at 0x000000000423B2E8>
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
obj =  ((<class TaskSystem.AsyncTask at 0x0000000004241EE8>, 'asyncCall'), (<TaskSystem.AsyncTask instance at 0x000000001BB67808>,))
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 554, in save_tuple
 line: save(element)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
element =  (<TaskSystem.AsyncTask instance at 0x000000001BB67808>,)
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_tuple at 0x000000000423B2E8>
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
obj =  (<TaskSystem.AsyncTask instance at 0x000000001BB67808>,)
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 554, in save_tuple
 line: save(element)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
element =  <TaskSystem.AsyncTask instance at 0x000000001BB67808>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_inst at 0x000000000423B588>
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
obj =  <TaskSystem.AsyncTask instance at 0x000000001BB67808>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 731, in save_inst
 line: save(stuff)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
stuff =  {'env_update': {'THEANO_FLAGS': 'compiledir_format=compiledir%(platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s--dev-gpuZ,device=gpu,force_device=True'}, 'parent_pid': 11564, 'name': 'Device gpuX proc', 'Process': <class 'multiprocessing.process.Process'>, 'parent_conn': <ExecingP..., len = 11
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_dict at 0x000000000423B4A8>
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
obj =  {'env_update': {'THEANO_FLAGS': 'compiledir_format=compiledir%(platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s--dev-gpuZ,device=gpu,force_device=True'}, 'parent_pid': 11564, 'name': 'Device gpuX proc', 'Process': <class 'multiprocessing.process.Process'>, 'parent_conn': <ExecingP..., len = 11
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 655, in save_dict
 line: self._batch_setitems(obj.iteritems())
 locals:
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
self._batch_setitems =  <bound method ForkingPickler._batch_setitems of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
obj =  {'env_update': {'THEANO_FLAGS': 'compiledir_format=compiledir%(platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s--dev-gpuZ,device=gpu,force_device=True'}, 'parent_pid': 11564, 'name': 'Device gpuX proc', 'Process': <class 'multiprocessing.process.Process'>, 'parent_conn': <ExecingP..., len = 11
obj.iteritems =  <built-in method iteritems of dict object at 0x000000001BB696A8>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 687, in _batch_setitems
 line: save(v)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
v =  <bound method Device.process of <Device.Device object at 0x000000001BB1CDA0>>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function dispatcher at 0x0000000004265C88>
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
obj =  <bound method Device.process of <Device.Device object at 0x000000001BB1CDA0>>
File "C:\ProgramData\Anaconda2\lib\multiprocessing\forking.py", line 67, in dispatcher
 line: self.save_reduce(obj=obj, rv)
 locals:
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
self.save_reduce =  <bound method ForkingPickler.save_reduce of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
obj =  <bound method Device.process of <Device.Device object at 0x000000001BB1CDA0>>
rv =  (<built-in function getattr>, (<Device.Device object at 0x000000001BB1CDA0>, 'process'))
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 401, in save_reduce
 line: save(args)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
args =  (<Device.Device object at 0x000000001BB1CDA0>, 'process')
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_tuple at 0x000000000423B2E8>
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
obj =  (<Device.Device object at 0x000000001BB1CDA0>, 'process')
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 554, in save_tuple
 line: save(element)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
element =  <Device.Device object at 0x000000001BB1CDA0>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 331, in save
 line: self.save_reduce(obj=obj, rv)
 locals:
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
self.save_reduce =  <bound method ForkingPickler.save_reduce of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
obj =  <Device.Device object at 0x000000001BB1CDA0>
rv =  (<function newobj at 0x0000000003516DD8>, (<class 'Device.Device'>,), {'compute_total_time': 0, 'num_updates': 0, 'name': 'gpuX', 'num_frames': NumbersDict(0), 'run_called_count': 0, 'update_specs': {'layers': [], 'block_size': 0, 'update_params': {}, 'update_rule': 'global'}, 'main_pid': 115...
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 425, in save_reduce
 line: save(state)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
state =  {'compute_total_time': 0, 'num_updates': 0, 'name': 'gpuX', 'num_frames': NumbersDict(0), 'run_called_count': 0, 'update_specs': {'layers': [], 'block_size': 0, 'update_params': {}, 'update_rule': 'global'}, 'main_pid': 11564, 'wait_for_result_call': False, 'train_outputs_format': None, 'epoch': ..., len = 18
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_dict at 0x000000000423B4A8>
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
obj =  {'compute_total_time': 0, 'num_updates': 0, 'name': 'gpuX', 'num_frames': NumbersDict(0), 'run_called_count': 0, 'update_specs': {'layers': [], 'block_size': 0, 'update_params': {}, 'update_rule': 'global'}, 'main_pid': 11564, 'wait_for_result_call': False, 'train_outputs_format': None, 'epoch': ..., len = 18
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 655, in save_dict
 line: self._batch_setitems(obj.iteritems())
 locals:
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
self._batch_setitems =  <bound method ForkingPickler._batch_setitems of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
obj =  {'compute_total_time': 0, 'num_updates': 0, 'name': 'gpuX', 'num_frames': NumbersDict(0), 'run_called_count': 0, 'update_specs': {'layers': [], 'block_size': 0, 'update_params': {}, 'update_rule': 'global'}, 'main_pid': 11564, 'wait_for_result_call': False, 'train_outputs_format': None, 'epoch': ..., len = 18
obj.iteritems =  <built-in method iteritems of dict object at 0x000000001BB5DBF8>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 687, in _batch_setitems
 line: save(v)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
v =  NumbersDict(0)
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_inst at 0x000000000423B588>
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
obj =  NumbersDict(0)
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 731, in save_inst
 line: save(stuff)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
stuff =  {'max': <function __max_error at 0x000000001B0D2358>, 'dict': {}, 'value': 0}
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_dict at 0x000000000423B4A8>
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
obj =  {'max': <function __max_error at 0x000000001B0D2358>, 'dict': {}, 'value': 0}
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 655, in save_dict
 line: self._batch_setitems(obj.iteritems())
 locals:
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
self._batch_setitems =  <bound method ForkingPickler._batch_setitems of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
obj =  {'max': <function __max_error at 0x000000001B0D2358>, 'dict': {}, 'value': 0}
obj.iteritems =  <built-in method iteritems of dict object at 0x000000001BB5DD08>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 687, in _batch_setitems
 line: save(v)
 locals:
save =  <bound method ForkingPickler.save of <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>>
v =  <function __max_error at 0x000000001B0D2358>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 286, in save
 line: f(self, obj) # Call unbound method with explicit self
 locals:
f =  <function save_global at 0x000000000423B5F8>
self =  <multiprocessing.forking.ForkingPickler instance at 0x000000001BB737C8>
obj =  <function __max_error at 0x000000001B0D2358>
File "C:\ProgramData\Anaconda2\lib\pickle.py", line 754, in save_global
 line: raise PicklingError(
"Can't pickle %r: it's not found as %s.%s" %
(obj, module, name))
 locals:
PicklingError =  <class 'pickle.PicklingError'>
obj =  <function __max_error at 0x000000001B0D2358>
module =  'Util'
name =  '__max_error', len = 11
PicklingError: Can't pickle <function __max_error at 0x000000001B0D2358>: it's not found as Util.__max_error
KeyboardInterrupt
EXCEPTION
Traceback (most recent call last):
File "......\rnn.py", line 526, in main
 line: init(commandLineOptions=argv[1:])
 locals:
init =  <function init at 0x000000001BB162E8>
commandLineOptions = 
argv =  ['..\..\..\rnn.py', 'config_real'], _[0]: {len = 15}
File "......\rnn.py", line 339, in init
 line: devices = initDevices()
 locals:
devices = 
initDevices =  <function initDevices at 0x000000001BB17F28>
File "......\rnn.py", line 152, in initDevices
 line: time.sleep(0.25)
 locals:
time =  <module 'time' (built-in)>
time.sleep =  <built-in function sleep>
KeyboardInterrupt
Quitting

D:\returnn-master\demos\mdlstm\IAM>WARNING (theano.sandbox.cuda): The cuda backend is deprecated and will be removed in the next release (v0.10). Please switch to the gpuarray backend. You can get more information about how to switch at this URL: https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29

Using gpu device 0: GeForce 940MX (CNMeM is enabled with initial size: 50.0% of memory, cuDNN 5110) Traceback (most recent call last): File "", line 1, in File "C:\ProgramData\Anaconda2\lib\multiprocessing\forking.py", line 384, in main self = load(from_parent) EOFError

After debugging we came to know the above error is related to Pickling and the error is generated from the below below class of TaskSystem.py (Bold lines in this below class code)

class Popen(mp_Popen):
  # noinspection PyMissingConstructor
  def __init__(self, process_obj, env):
    # No super init call by intention!

    from multiprocessing.forking import duplicate, get_command_line, _python_exe, close, get_preparation_data, HIGHEST_PROTOCOL, dump
    import msvcrt
    import _subprocess

    # create pipe for communication with child
    rfd, wfd = os.pipe()

    # get handle for read end of the pipe and make it inheritable
    rhandle = duplicate(msvcrt.get_osfhandle(rfd), inheritable=True)
    os.close(rfd)

    # start process
    cmd = get_command_line() + [rhandle]
    cmd = ' '.join('"%s"' % x for x in cmd)
    hp, ht, pid, tid = _subprocess.CreateProcess(
      _python_exe, cmd, None, None, 1, 0, env, None, None
    )
    ht.Close()
    close(rhandle)

    # set attributes of self
    self.pid = pid
    self.returncode = None
    self._handle = hp

    # send information to child
    prep_data = get_preparation_data(process_obj._name)

    to_child = os.fdopen(wfd, 'wb')

    mp_Popen._tls.process_handle = int(hp)

    try:          
      **dump(prep_data, to_child, HIGHEST_PROTOCOL)          
      dump(process_obj, to_child, HIGHEST_PROTOCOL)**          
    finally:
      del mp_Popen._tls.process_handle
      to_child.close()

.theanorc file:

[global] floatX = float32 device = gpu

[dnn] include_path=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\include library_path=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\lib\x64

[lib] cnmem=.50

[nvcc] flags=-LC:\ProgramData\Anaconda2\libs compiler_bindir=C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin

[blas] ldflags = -LC:\Anaconda3\Library\bin -lmkl_rt

config_real: { "task": "train", "device": "gpu", "multiprocessing": true, "batching": "default", "cache_size": "2G", "max_seqs": 10, "batch_size": 600, "nadam": 1, "learning_rate": 0.0005, "learning_rates": "{1: 0.0005, 25: 0.0003, 35: 0.0001}", "num_epochs": 2, "update_batch_size": 2000000, "reinit": 0, "log_verbosity": 5, "train": "features/raw/train.1.h5,features/raw/train.2.h5", "dev": "features/raw/train_valid.h5", "start_epoch": 1, "num_inputs": 1, "num_outputs": {"data": [1,2], "classes": [79,1], "sizes": [2,1]}, "network": { "classes_source": {"class": "source", "data_key": "sizes", "from": [], "n_out": 2, "dtype":"int32"}, "1Dto2D": {"class": "1Dto2D", "from" : ["data", "classes_source"] }, "conv0": { "class" : "conv2", "n_features" : 15, "filter" : [3, 3], "pool_size" : [2, 2], "from" : ["1Dto2D"] }, "mdlstm0": { "class" : "mdlstm", "n_out" : 30, "dropout": 0.25, "from" : ["conv0"] }, "conv1" : { "class" : "conv2", "n_features" : 45, "dropout": 0.25, "filter" : [3, 3], "pool_size" : [2, 2], "from" : ["mdlstm0"] }, "mdlstm1": { "class" : "mdlstm", "n_out" : 60, "dropout": 0.25, "from" : ["conv1"] }, "conv2" : { "class" : "conv2", "n_features" : 75, "dropout": 0.25, "filter" : [3, 3], "pool_size" : [2, 2], "from" : ["mdlstm1"] }, "mdlstm2": { "class" : "mdlstm", "n_out" : 90, "dropout": 0.25, "from" : ["conv2"] }, "conv3" : { "class" : "conv2", "n_features" : 105, "dropout": 0.25, "filter" : [3, 3], "pool_size" : [1, 1], "from" : ["mdlstm2"] }, "mdlstm3": { "class" : "mdlstm", "n_out" : 120, "dropout": 0.25, "from" : ["conv3"] }, "conv4" : { "class" : "conv2", "n_features" : 105, "dropout": 0.25, "filter" : [3, 3], "pool_size" : [1, 1], "from" : ["mdlstm3"] }, "mdlstm4": { "class" : "mdlstm", "n_out" : 120, "dropout": 0.25, "from" : ["conv4"] }, "output": { "class" : "softmax", "loss": "ctc", "dropout": 0.25, "from" : ["mdlstm4"] } },

"ctc_prior_file": "priors/mdlstm_real", "model": "models/mdlstm_real", "log": "log/mdlstm_real" }

pvoigtlaender commented 7 years ago

the issue seems that for some reason 82 classes instead of 80 are used in the output layer. I'm not yet sure, why this is happening. It might be related to the data files. Does the demo setup work for you?

In general, we usually do not try to run under windows, but we will try to help to make it work under windows as long as the effort is reasonable

RashmiPandeyatGit commented 7 years ago

I updated the issue. Please do the needful.

pvoigtlaender commented 7 years ago

I noticed that you again posted "multiprocessing": false, is that what you used for the latest run? It might be, that multiprocessing true will crash in windows with the Pickle errors, and multiprocessing: false can avoid these errors, but then multiprocessing false will introduce other problems, since this was not tested or used for a long time.

One unverified theory I have, about what is happening with the number of classes: From the dataset, the network gets 79 classes. For CTC it adds one class for the blank label. Maybe because of the multiprocessing options, it will add it multiple (seems to be 3) times instead of only a single time, so it ends up with 82 classes, which leads to a mismatch.

You are currently trying to run config_real with the real IAM data. Did you also try the config_demo with the 3 demo images?

bdotgradb commented 7 years ago

I’ve seen this error with their being 2 too many classes when I have used false for multiprocessing on Ubuntu. Setting it to true has always resolved the issue for me.

RashmiPandeyatGit commented 7 years ago

@pvoigtlaender It’s my mistake to write it as False but it’s true..I am grateful to you for giving such time to resolve issue. But when I tried demo also it was giving us similar pickling error..

RashmiPandeyatGit commented 7 years ago

@bdotgradb Actually i am executing returnn programs on a Windows 64bit system. So whenever I set multiprocessing to true I get pickle error. Please guide me on what is the best configuration/packages for executing RETURNN programs. Below is my system(SW & HW) configurations.

OS : Windows 10 64bit Graphics : Nvidia Geforce 940MX (2 GB) Driver : 382.05 Cuda : 8.0 Theano : 0.9 Anaconda2 (Python2.7)

As I am unable to solve the pickel error by using Windows 10. Will it be a better idea to move to ubuntu 16.04 with same above mentioned configuration?

albertz commented 7 years ago

Windows is not really supported. Can you run it on Linux?

Am 31.10.2017 13:41 schrieb "Rashmi Pandey" notifications@github.com:

@bdotgradb https://github.com/bdotgradb Actually i am executing returnn programs on a Windows 64bit system. So whenever we set multiprocessing to true I get pickle error. Please guide me on what is the best configuration/packages for executing RETURNN programs. Below is my system(SW & HW) configurations.

OS : Windows 10 64bit Graphics : Nvidia Geforce 940MX (2 GB) Driver : 382.05 Cuda : 8.0 Theano : 0.9 Anaconda2 (Python2.7)

As I am unable to solve the pickel error by using Windows 10. Will it be a better idea to move to ubuntu 16.04 with same above mentioned configuration?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/rwth-i6/returnn/issues/19#issuecomment-340750188, or mute the thread https://github.com/notifications/unsubscribe-auth/AADm_MVVYT3dvMkd5wh8_Ou9eRoA6frWks5sxxVygaJpZM4QLWhh .

RashmiPandeyatGit commented 7 years ago

@albertz , @bdotgradb and @pvoigtlaender

Thank you all for giving your valuable time and helping me. Now I will setup Ubuntu environment and try to execute the codes

albertz commented 7 years ago

Sorry for not be more helpful but noone of us uses Windows. Feel free to fix this and provide a pull request, and ask any questions about the current code. As you have seen, there was some initial try to make some of it working on Windows but it seems it's broken again. I guess it will be just easier to run it on Linux.

The unreadable characters are ANSI escape codes. I tried to remove them in your comments. They make the output colorful as long as the terminal supports ANSI escape codes, which is normally the case but probably not on the Windows terminal. But also on Windows there should be other terminals which support them.

RoysonFrancis commented 7 years ago

I have a query.

Q1. In config_real I see the network architecture designed as:

"1Dto2D": {"class": "1Dto2D", "from" : ["data", "classes_source"] }, "conv0": { "class" : "conv2", "n_features" : 15, "filter" : [3, 3], "pool_size" : [2, 2], "from" : ["1Dto2D"] }, "mdlstm0": { "class" : "mdlstm", "n_out" : 30, "dropout": 0.25, "from" : ["conv0"] }, "conv1" : { "class" : "conv2", "n_features" : 45, "dropout": 0.25, "filter" : [3, 3], "pool_size" : [2, 2], "from" : ["mdlstm0"] }, "mdlstm1": { "class" : "mdlstm", "n_out" : 60, "dropout": 0.25, "from" : ["conv1"] }, "conv2" : { "class" : "conv2", "n_features" : 75, "dropout": 0.25, "filter" : [3, 3], "pool_size" : [2, 2], "from" : ["mdlstm1"] }, "mdlstm2": { "class" : "mdlstm", "n_out" : 90, "dropout": 0.25, "from" : ["conv2"] }, "conv3" : { "class" : "conv2", "n_features" : 105, "dropout": 0.25, "filter" : [3, 3], "pool_size" : [1, 1], "from" : ["mdlstm2"] }, "mdlstm3": { "class" : "mdlstm", "n_out" : 120, "dropout": 0.25, "from" : ["conv3"] }, "conv4" : { "class" : "conv2", "n_features" : 105, "dropout": 0.25, "filter" : [3, 3], "pool_size" : [1, 1], "from" : ["mdlstm3"] }, "mdlstm4": { "class" : "mdlstm", "n_out" : 120, "dropout": 0.25, "from" : ["conv4"] }, "output": { "class" : "softmax", "loss": "ctc", "dropout": 0.25, "from" : ["mdlstm4"] }

Q1. Are the MDLSTM and Convolution layers stacked on top of each other ? Q2. Does the "from" key in the dictionary (e.g { "class" : "mdlstm", "n_out" : 30, "dropout": 0.25, "from" : ["conv0"] }) ...... say that the ouput of conv is passed as input to mdlstm?

RashmiPandeyatGit commented 7 years ago

@albertz @pvoigtlaender @bdotgradb Will it work for Ubuntu 17?

albertz commented 7 years ago

Yes, stacked on each other. See the documentation here for documentation. It should work on Ubuntu 17. We have tried on Ubuntu 16.

albertz commented 6 years ago

I'm closing this now. Windows-support is nothing which we aim at, so if it does not work, please fix it yourself and send us a pull-request.