search

uber / petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
Apache License 2.0
1.76k stars 281 forks source link

Replace process_iter by pid_exists #757

Closed MostafaFarahani closed 2 years ago

MostafaFarahani commented 2 years ago

process_iter is very slow for checking if a process is alive. pid_exists is more efficient.

See: https://psutil.readthedocs.io/en/latest/index.html?highlight=process_iter#psutil.pid_exists

CLAassistant commented 2 years ago

CLA assistant check
All committers have signed the CLA.

codecov[bot] commented 2 years ago

Codecov Report

Merging #757 (958b59c) into master (d32709d) will not change coverage. The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #757   +/-   ##
=======================================
  Coverage   86.27%   86.27%           
=======================================
  Files          85       85           
  Lines        5084     5084           
  Branches      787      785    -2     
=======================================
  Hits         4386     4386           
  Misses        559      559           
  Partials      139      139
Impacted Files Coverage Δ
petastorm/workers_pool/process_pool.py 92.74% <100.00%> (ø)
petastorm/workers_pool/tests/test_workers_pool.py 98.88% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update d32709d...958b59c. Read the comment docs.

MostafaFarahani commented 2 years ago

@selitvin - Thanks for the quick review!