Closed mesner closed 6 months ago
It seems this was caused by instantiating the calculator at the script / global / module scope.
This full example works for me. In my actual script, I rewrote the compute function to not use tqdm.
How do others run this package in a distributed manner?
import os
if "C:\\Program Files\\GNU Octave\\Octave-8.3.0\\mingw64\\bin" not in os.environ["PATH"]: os.environ["PATH"] += ";C:\\Program Files\\GNU Octave\\Octave-8.3.0\\mingw64\\bin"
import numpy as np
import pandas as pd
import random
import time
from pyspi.calculator import Calculator
from multiprocessing import Pool
def runCompute(dataset):
if not hasattr(runCompute, "calc"):
runCompute.calc = Calculator()
calc = runCompute.calc
calc.load_dataset(dataset)
calc.compute()
return calc.table.copy()
if __name__ == "__main__":
print("running pool")
random.seed(42)
M = 2
T = 50
dataset = np.random.randn(M,T)
with Pool(1) as p:
rows = p.map(runCompute, [dataset])
print("complete")
print(rows)
I think @jmoo2880 and @anniegbryant have experience running distributed pyspi. cf. https://time-series-features.gitbook.io/pyspi/usage/advanced-usage/distributing-calculations-on-a-cluster
Hi @mesner, sorry for the delayed reply. We recommend checking out the pyspi-distribute repo to submit jobs through a PBS job scheduling system on a high-performance computing cluster if you have access to this. We included a simple demo Jupyter notebook walking users through an example workflow. Hope this helps -- let us know if you encounter any issues with that or if it doesn't resolve your problem!
I'm trying to distribute thousands of pairs of datasets to the calculator, but it hangs indefinitely when run as a child process using python's multiprocessing (often at dcorr, dcorr_biased, or dtw when included). top indicates no cpu usage.