uber / petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
Apache License 2.0
1.78k stars 285 forks source link

'WorkerThread' object has no attribute 'isAlive' #692

Closed jmpanfil closed 3 years ago

jmpanfil commented 3 years ago

After updating to Python 3.9.5 and using Petastorm '0.11.1', I am getting an error 'WorkerThread' object has no attribute 'isAlive'. My code mostly follows the databricks example. Based off some other posts such as this, it looks like

It seems that thread's method isAlive() was renamed to is_alive() in python verson 3.9

Is there a temporary workaround?

Thanks

Full error:

AttributeError                            Traceback (most recent call last)
D:\Data\BayesNN.py in train_and_evaluate_SVI(self, svi, criterion, bs, ne, lr)
    141 
--> 142                 train_loss = self.train_one_epoch_SVI(svi, train_dataloader_iter, steps_per_epoch, epoch, self.device)
    143 

d:\data\venv\lib\site-packages\petastorm\spark\spark_dataset_converter.py in __exit__(self, exc_type, exc_value, exc_traceback)
    403         self.reader.stop()
--> 404         self.reader.join()
    405 

d:\data\venv\lib\site-packages\petastorm\reader.py in join(self)
    658         """Joins all worker threads/processes. Will block until all worker workers have been fully terminated."""
--> 659         self._workers_pool.join()
    660 

d:\data\venv\lib\site-packages\petastorm\workers_pool\thread_pool.py in join(self)
    186         for w in self._workers:
--> 187             if w.isAlive():
    188                 w.join()

AttributeError: 'WorkerThread' object has no attribute 'isAlive'

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
<timed eval> in <module>

D:\Data\BayesNN.py in train_and_evaluate_SVI(self, svi, criterion, bs, ne, lr)
    140                 if (epoch + 1) % 10 == 0: print('-' * 10)
    141 
--> 142                 train_loss = self.train_one_epoch_SVI(svi, train_dataloader_iter, steps_per_epoch, epoch, self.device)
    143 
    144                 #if (epoch + 1) % 10 == 0: val_loss = evaluate(model, criterion, val_dataloader_iter, validation_steps, device) #val_loss =

d:\data\venv\lib\site-packages\petastorm\spark\spark_dataset_converter.py in __exit__(self, exc_type, exc_value, exc_traceback)
    402     def __exit__(self, exc_type, exc_value, exc_traceback):
    403         self.reader.stop()
--> 404         self.reader.join()
    405 
    406 

d:\data\venv\lib\site-packages\petastorm\reader.py in join(self)
    657     def join(self):
    658         """Joins all worker threads/processes. Will block until all worker workers have been fully terminated."""
--> 659         self._workers_pool.join()
    660 
    661     @property

d:\data\venv\lib\site-packages\petastorm\workers_pool\thread_pool.py in join(self)
    185         """Block until all workers are terminated."""
    186         for w in self._workers:
--> 187             if w.isAlive():
    188                 w.join()
    189 

AttributeError: 'WorkerThread' object has no attribute 'isAlive'
jmpanfil commented 3 years ago

I should note that downgrading to Python 3.8.10 works