search

hitachi-speech / EEND

End-to-End Neural Diarization
MIT License
368 stars 57 forks source link

Tcl_AsyncDelete error #5

Closed AntonOkhotnikov closed 4 years ago

AntonOkhotnikov commented 4 years ago

Hello, some time ago I caught the next error while training:

   0.55256 iters/sec. Estimated time to finish: 19 days, 0:00:23.591611.
     total [####..............................................]  9.99%
this epoch [#################################################.] 99.30%
    100700 iter, 9 epoch / 100 epochs
   0.59537 iters/sec. Estimated time to finish: 17 days, 15:10:34.544256.
Tcl_AsyncDelete: async handler deleted by the wrong thread
# Accounting: time=221865 threads=1

Do you have any ideas of why this could happen? Stackoverflow says that matplotlib should use 'Agg' backend, and it does.

yubouf commented 4 years ago

I don't know the reason. If it is related to matplotlib, stopping graph plots by commenting out below may be a workaround. https://github.com/hitachi-speech/EEND/blob/9a0f211ce7e377eaea242490c3d7ec0f6adab8af/eend/chainer_backend/train.py#L197-L205

You can reproduce such a plot from the log file, so removing the above will not be a problem.

AntonOkhotnikov commented 4 years ago

@yubouf Thank you so much, I will try it

AntonOkhotnikov commented 4 years ago

@yubouf The suggested solution helped to overcome this issue, so I'm closing the issue now. Thanks again