TalkNet_Training_Offline error

GPU available: True, used: True TPU available: False, using: 0 TPU cores [NeMo W 2022-12-02 09:58:37 modelPT:138] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader. Train config : dataset: target: nemo.collections.asr.data.audio_to_text.AudioToCharWithDursF0Dataset manifest_filepath: H:/ControllableTalkNet/tTrump\trainfiles.json max_duration: null min_duration: 0.1 int_values: false load_audio: false normalize: false sample_rate: 22050 trim: false durs_file: H:/ControllableTalkNet/tTrump\durations.pt f0_file: H:/ControllableTalkNet/tTrump\f0s.pt blanking: true vocab: notation: phonemes punct: true spaces: true stresses: false add_blank_at: last dataloader_params: drop_last: false shuffle: true batch_size: 16 num_workers: 4

[NeMo W 2022-12-02 09:58:37 modelPT:145] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s). Validation config : dataset: target: nemo.collections.asr.data.audio_to_text.AudioToCharWithDursF0Dataset manifest_filepath: H:/ControllableTalkNet/tTrump\valfiles.json max_duration: null min_duration: 0.1 int_values: false load_audio: false normalize: false sample_rate: 22050 trim: false durs_file: H:/ControllableTalkNet/tTrump\durations.pt f0_file: H:/ControllableTalkNet/tTrump\f0s.pt blanking: true vocab: notation: phonemes punct: true spaces: true stresses: false add_blank_at: last dataloader_params: drop_last: false shuffle: false batch_size: 16 num_workers: 1

[NeMo I 2022-12-02 09:58:37 modelPT:439] Model TalkNetDursModel was successfully restored from H:\ControllableTalkNet\talknet_durs.nemo. [NeMo I 2022-12-02 09:58:37 collections:173] Dataset loaded with 134 files totalling 0.21 hours [NeMo I 2022-12-02 09:58:37 collections:174] 0 files were filtered totalling 0.00 hours [NeMo I 2022-12-02 09:58:37 collections:173] Dataset loaded with 134 files totalling 0.21 hours [NeMo I 2022-12-02 09:58:37 collections:174] 0 files were filtered totalling 0.00 hours [NeMo W 2022-12-02 09:58:37 modelPT:660] The lightning trainer received accelerator: dp. We recommend to use 'ddp' instead. [NeMo I 2022-12-02 09:58:37 modelPT:751] Optimizer config = Adam ( Parameter Group 0 amsgrad: False betas: (0.9, 0.999) eps: 1e-08 lr: 0.001 weight_decay: 1e-06 ) [NeMo I 2022-12-02 09:58:37 lr_scheduler:621] Scheduler "<nemo.core.optim.lr_scheduler.CosineAnnealing object at 0x0000021A2DF86EB0>" will be used during training (effective maximum steps = 180) - Parameters : (min_lr: 3.0e-06 warmup_ratio: 0.02 max_steps: 180 ) Warm-starting from H:\ControllableTalkNet\talknet_durs.nemo [NeMo I 2022-12-02 09:58:37 exp_manager:216] Experiments will be logged at H:\ControllableTalkNet\tTrump\TalkNetDurs\2022-12-02_09-57-24 [NeMo I 2022-12-02 09:58:37 exp_manager:563] TensorboardLogger has been set up LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] [NeMo W 2022-12-02 09:58:38 modelPT:660] The lightning trainer received accelerator: dp. We recommend to use 'ddp' instead. [NeMo I 2022-12-02 09:58:38 modelPT:751] Optimizer config = Adam ( Parameter Group 0 amsgrad: False betas: (0.9, 0.999) eps: 1e-08 lr: 0.001 weight_decay: 1e-06 ) [NeMo I 2022-12-02 09:58:38 lr_scheduler:621] Scheduler "<nemo.core.optim.lr_scheduler.CosineAnnealing object at 0x0000021A2E22DCD0>" will be used during training (effective maximum steps = 180) - Parameters : (min_lr: 3.0e-06 warmup_ratio: 0.02 max_steps: 180 )

| Name | Type | Params

0 | embed | Embedding | 7.6 K 1 | model | ConvASREncoder | 2.5 M 2 | proj | Conv1d | 513

2.5 M Trainable params 0 Non-trainable params 2.5 M Total params 9.841 Total estimated model params size (MB) Validation sanity check: 0% 0/2 [00:00<?, ?it/s]

PicklingError Traceback (most recent call last) Cell In[6], line 68 66 initialize(config_path="conf") 67 cfg = compose(config_name="talknet-durs") ---> 68 train(cfg)

Cell In[6], line 62, in train(cfg) 60 exp_manager(trainer, cfg.get('exp_manager', None)) 61 trainer.callbacks.extend([pl.callbacks.LearningRateMonitor(), LogEpochTimeCallback()]) # noqa ---> 62 trainer.fit(model)

File ~\anaconda3\envs\talknet\lib\site-packages\pytorch_lightning\trainer\trainer.py:460, in Trainer.fit(self, model, train_dataloader, val_dataloaders, datamodule) 455 # links data to the trainer 456 self.data_connector.attach_data( 457 model, train_dataloader=train_dataloader, val_dataloaders=val_dataloaders, datamodule=datamodule 458 ) --> 460 self._run(model) 462 assert self.state.stopped 463 self.training = False

File ~\anaconda3\envs\talknet\lib\site-packages\pytorch_lightning\trainer\trainer.py:758, in Trainer._run(self, model) 755 self.pre_dispatch() 757 # dispatch start_training or start_evaluating or start_predicting --> 758 self.dispatch() 760 # plugin will finalized fitting (e.g. ddp_spawn will load trained model) 761 self.post_dispatch()

File ~\anaconda3\envs\talknet\lib\site-packages\pytorch_lightning\trainer\trainer.py:799, in Trainer.dispatch(self) 797 self.accelerator.start_predicting(self) 798 else: --> 799 self.accelerator.start_training(self)

File ~\anaconda3\envs\talknet\lib\site-packages\pytorch_lightning\accelerators\accelerator.py:96, in Accelerator.start_training(self, trainer) 95 def start_training(self, trainer: 'pl.Trainer') -> None: ---> 96 self.training_type_plugin.start_training(trainer)

File ~\anaconda3\envs\talknet\lib\site-packages\pytorch_lightning\plugins\training_type\training_type_plugin.py:144, in TrainingTypePlugin.start_training(self, trainer) 142 def start_training(self, trainer: 'pl.Trainer') -> None: 143 # double dispatch to initiate the training loop --> 144 self._results = trainer.run_stage()

File ~\anaconda3\envs\talknet\lib\site-packages\pytorch_lightning\trainer\trainer.py:809, in Trainer.run_stage(self) 807 if self.predicting: 808 return self.run_predict() --> 809 return self.run_train()

File ~\anaconda3\envs\talknet\lib\site-packages\pytorch_lightning\trainer\trainer.py:844, in Trainer.run_train(self) 841 if not self.is_global_zero and self.progress_bar_callback is not None: 842 self.progress_bar_callback.disable() --> 844 self.run_sanity_check(self.lightning_module) 846 self.checkpoint_connector.has_trained = False 848 # enable train mode

File ~\anaconda3\envs\talknet\lib\site-packages\pytorch_lightning\trainer\trainer.py:1112, in Trainer.run_sanity_check(self, ref_model) 1109 self.on_sanity_check_start() 1111 # run eval step -> 1112 self.run_evaluation() 1114 self.on_sanity_check_end() 1116 self.state.stage = stage

File ~\anaconda3\envs\talknet\lib\site-packages\pytorch_lightning\trainer\trainer.py:954, in Trainer.run_evaluation(self, on_epoch) 951 dataloader = self.accelerator.process_dataloader(dataloader) 952 dl_max_batches = self.evaluation_loop.max_batches[dataloader_idx] --> 954 for batch_idx, batch in enumerate(dataloader): 955 if batch is None: 956 continue

File ~\anaconda3\envs\talknet\lib\site-packages\torch\utils\data\dataloader.py:355, in DataLoader.iter(self) 353 return self._iterator 354 else: --> 355 return self._get_iterator()

File ~\anaconda3\envs\talknet\lib\site-packages\torch\utils\data\dataloader.py:301, in DataLoader._get_iterator(self) 299 else: 300 self.check_worker_number_rationality() --> 301 return _MultiProcessingDataLoaderIter(self)

File ~\anaconda3\envs\talknet\lib\site-packages\torch\utils\data\dataloader.py:914, in _MultiProcessingDataLoaderIter.init(self, loader) 907 w.daemon = True 908 # NB: Process.start() actually take some time as it needs to 909 # start a process and pass the arguments over via a pipe. 910 # Therefore, we only add a worker to self._workers list after 911 # it started, so that we do not call .join() if program dies 912 # before it starts, and del tries to join but will get: 913 # AssertionError: can only join a started process. --> 914 w.start() 915 self._index_queues.append(index_queue) 916 self._workers.append(w)

File ~\anaconda3\envs\talknet\lib\multiprocessing\process.py:121, in BaseProcess.start(self) 118 assert not _current_process._config.get('daemon'), \ 119 'daemonic processes are not allowed to have children' 120 _cleanup() --> 121 self._popen = self._Popen(self) 122 self._sentinel = self._popen.sentinel 123 # Avoid a refcycle if the target function holds an indirect 124 # reference to the process object (see bpo-30775)

File ~\anaconda3\envs\talknet\lib\multiprocessing\context.py:224, in Process._Popen(process_obj) 222 @staticmethod 223 def _Popen(process_obj): --> 224 return _default_context.get_context().Process._Popen(process_obj)

File ~\anaconda3\envs\talknet\lib\multiprocessing\context.py:327, in SpawnProcess._Popen(process_obj) 324 @staticmethod 325 def _Popen(process_obj): 326 from .popen_spawn_win32 import Popen --> 327 return Popen(process_obj)

File ~\anaconda3\envs\talknet\lib\multiprocessing\popen_spawn_win32.py:93, in Popen.init(self, process_obj) 91 try: 92 reduction.dump(prep_data, to_child) ---> 93 reduction.dump(process_obj, to_child) 94 finally: 95 set_spawning_popen(None)

File ~\anaconda3\envs\talknet\lib\multiprocessing\reduction.py:60, in dump(obj, file, protocol) 58 def dump(obj, file, protocol=None): 59 '''Replacement for pickle.dump() using ForkingPickler.''' ---> 60 ForkingPickler(file, protocol).dump(obj)

PicklingError: Can't pickle <class 'nemo.collections.common.parts.preprocessing.collections.AudioTextEntity'>: attribute lookup AudioTextEntity on nemo.collections.common.parts.preprocessing.collections failed

SortAnon / ControllableTalkNet