WindowsでのAttributeErrorについて

BURI55 commented 5 years ago

動作推奨環境について https://github.com/Hiroshiba/become-yukarin/issues/27 で、

・Windows環境ではマルチスレッド処理のpickleでエラーが起きます

と指摘があり、Linux推奨になったと思いますが、

windowsでのvoice_conversion_test.pyの実行エラーについて https://github.com/Hiroshiba/become-yukarin/issues/34

にて、

・train.py実行時、if name == 'main' が無いというエラーが発生する。こちらもmultiprocessingに関連するエラーのようです。どの部分をif name == 'main'で囲めばよいのか分からないため全体を囲ってから実行しました。

これは8月25日に直りましたが、

・train.py実行時、pickle使用時にAttributeErrorが発生する windows環境ではmultiprocessingの開始にはspawnを使用する必要があります。 https://docs.python.jp/3/library/multiprocessing.html おそらくこのプログラムはforkを前提として記載しているため、spawnでpickleを使用する際にエラーが発生します。この点を改善するために、pickleではなくdillを使用するようにしてpickle使用時のエラーに対応しました。

これはまだ直っていません。

その後も、 AttributeError: Can't pickle local object 'create_sr..' https://github.com/Hiroshiba/become-yukarin/issues/54 という関連質問があり、

今も同じエラーが発生しています。

こちらも改修してはいただけないでしょうか？

参考 Pythonのpickleでserializeできない問題の回避策（dill） http://nishidy.hatenablog.com/entry/2016/02/01/225750

Hiroshiba commented 5 years ago

私もwindowsで試してみたのですが、そのエラーは発生しませんでした。簡単そうであれば解決策を探したいと思いますので、エラーメッセージを書いてください。

BURI55 commented 5 years ago

user@DESKTOP-VH77PEE MINGW64 ~/deep_yukarin $ python become-yukarin/train.py dat/1st_models_by/yukari/config.json dat/1st_models_by/yukari/1st_yukari_model_by Exception in main training loop: Can't pickle local object 'create..' Traceback (most recent call last): File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\chainer\training\trainer.py", line 315, in run update() File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\chainer\training\updaters\standard_updater.py", line 165, in update self.update_core() File "C:\Users\user\deep_yukarin\become-yukarin\become_yukarin\updater\updater.py", line 82, in update_core batch = self.get_iterator('main').next() File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\chainer\iterators\multiprocess_iterator.py", line 149, in next self._prefetch_loop.launch_thread() File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\chainer\iterators\multiprocess_iterator.py", line 433, in launch_thread initargs=(self.dataset, self.mem_size, self.mem_bulk)) File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\multiprocessing\context.py", line 119, in Pool context=self.get_context()) File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\multiprocessing\pool.py", line 176, in init self._repopulate_pool() File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\multiprocessing\pool.py", line 241, in _repopulate_pool w.start() File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\multiprocessing\process.py", line 112, in start self._popen = self._Popen(self) File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\multiprocessing\popen_spawn_win32.py", line 89, in init reduction.dump(process_obj, to_child) File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\multiprocessing\reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) Will finalize trainer extensions and updater before reraising the exception. Traceback (most recent call last): File "become-yukarin/train.py", line 82, in trainer.run() File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\chainer\training\trainer.py", line 329, in run six.reraise(*sys.exc_info()) File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\six.py", line 693, in reraise raise value File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\chainer\training\trainer.py", line 315, in run update() File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\chainer\training\updaters\standard_updater.py", line 165, in update self.update_core() File "C:\Users\user\deep_yukarin\become-yukarin\become_yukarin\updater\updater.py", line 82, in update_core batch = self.get_iterator('main').next() File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\chainer\iterators\multiprocess_iterator.py", line 149, in next self._prefetch_loop.launch_thread() File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\chainer\iterators\multiprocess_iterator.py", line 433, in launch_thread initargs=(self.dataset, self.mem_size, self.mem_bulk)) File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\multiprocessing\context.py", line 119, in Pool context=self.get_context()) File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\multiprocessing\pool.py", line 176, in init self._repopulate_pool() File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\multiprocessing\pool.py", line 241, in _repopulate_pool w.start() File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\multiprocessing\process.py", line 112, in start self._popen = self._Popen(self) File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\multiprocessing\popen_spawn_win32.py", line 89, in init reduction.dump(process_obj, to_child) File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\multiprocessing\reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) AttributeError: Can't pickle local object 'create..' Error in sys.excepthook:

Original exception was: Error in sys.excepthook:

Original exception was:

user@DESKTOP-VH77PEE MINGW64 ~/deep_yukarin $ Traceback (most recent call last): File "", line 1, in File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 115, in _main self = reduction.pickle.load(from_parent) EOFError: Ran out of input

エラーメッセージは次の通りになります。

BURI55 commented 5 years ago

E:\become-yukarin>python train_sr.py recipe/config_sr.json dat/model/yukari_2nd/test Exception in main training loop: Can't pickle local object 'create_sr..' Traceback (most recent call last): File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\site-packages\chainer\training\trainer.py", line 315, in run update() File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\site-packages\chainer\training\updaters\standard_updater.py", line 165, in update self.update_core() File "E:\become-yukarin\become_yukarin\updater\sr_updater.py", line 75, in update_core batch = self.get_iterator('main').next() File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\site-packages\chainer\iterators\multiprocess_iterator.py", line 149, in next self._prefetch_loop.launch_thread() File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\site-packages\chainer\iterators\multiprocess_iterator.py", line 433, in launch_thread initargs=(self.dataset, self.mem_size, self.mem_bulk)) File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\multiprocessing\context.py", line 119, in Pool context=self.get_context()) File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\multiprocessing\pool.py", line 175, in init self._repopulate_pool() File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\multiprocessing\pool.py", line 236, in _repopulate_pool self._wrap_exception) File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\multiprocessing\pool.py", line 255, in _repopulate_pool_static w.start() File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\multiprocessing\process.py", line 105, in start self._popen = self._Popen(self) File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\multiprocessing\popen_spawn_win32.py", line 65, in init reduction.dump(process_obj, to_child) File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\multiprocessing\reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) Will finalize trainer extensions and updater before reraising the exception. Traceback (most recent call last): File "train_sr.py", line 83, in trainer.run() File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\site-packages\chainer\training\trainer.py", line 329, in run six.reraise(*sys.exc_info()) File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\site-packages\six.py", line 693, in reraise raise value File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\site-packages\chainer\training\trainer.py", line 315, in run update() File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\site-packages\chainer\training\updaters\standard_updater.py", line 165, in update self.update_core() File "E:\become-yukarin\become_yukarin\updater\sr_updater.py", line 75, in update_core batch = self.get_iterator('main').next() File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\site-packages\chainer\iterators\multiprocess_iterator.py", line 149, in next self._prefetch_loop.launch_thread() File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\site-packages\chainer\iterators\multiprocess_iterator.py", line 433, in launch_thread initargs=(self.dataset, self.mem_size, self.mem_bulk)) File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\multiprocessing\context.py", line 119, in Pool context=self.get_context()) File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\multiprocessing\pool.py", line 175, in init self._repopulate_pool() File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\multiprocessing\pool.py", line 236, in _repopulate_pool self._wrap_exception) File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\multiprocessing\pool.py", line 255, in _repopulate_pool_static w.start() File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\multiprocessing\process.py", line 105, in start self._popen = self._Popen(self) File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\multiprocessing\popen_spawn_win32.py", line 65, in init reduction.dump(process_obj, to_child) File "C:\Users\kiritan\AppData\Local\Programs\Python\Python36\lib\multiprocessing\reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) AttributeError: Can't pickle local object 'create_sr..' ついでにAI_きりたんさんのエラーログも張ります。

Hiroshiba commented 5 years ago

エラーログを見てみたのですが、原因はわかりませんでした。推奨環境のLinuxを使ってくださればと思います。

train.pyにあるMultiprocessIteratorをSerialIteratorにすれば回避できるかもしれません。その場合は学習が遅くなると思います。

issueは一旦閉じますが、また何かわかればコメントしてください。

BURI55 commented 5 years ago

train.pyにあるMultiprocessIteratorをSerialIteratorにしたところ、以下のエラーが出ます。 user@DESKTOP-VH77PEE MINGW64 ~/deep_yukarin $ python become-yukarin/train_sr.py \

dat/2nd_models_by/yukari/config_sr.json \ dat/2nd_models_by/yukari/2nd_yukari_model_by Traceback (most recent call last): File "become-yukarin/train_sr.py", line 38, in train_iter = MultiprocessIterator(dataset['train'], config.train.batchsize) NameError: name 'MultiprocessIterator' is not defined

BURI55 commented 5 years ago

全て直したところ動きましたが、10万イテレーションに60時間かかるようです。

BURI55 commented 5 years ago

35000イテレーションに12時間かかりました。

BURI55 commented 5 years ago

回避方法を教えていただきました。 C:\Usersユーザ名\Anaconda3\pkgs\python-3.6.5-h0c2934d_0\Lib\multiprocessing\http://reduction.py/ の15行目にある　import pickleを import dill as pickleに変更 dillはpipなどからインストールしてくださいだそうです。ヒホさんのソースをいじるのではなく、Pythonそのものをいじるやり方になります。現在試運転中ですが、マルチプロセッシングでエラー出ないでうまく動いているようです。

Hiroshiba / become-yukarin

WindowsでのAttributeErrorについて #65