search

dtscalibration / python-dts-calibration

A Python package to load raw Distributed Temperature Sensing (DTS) files, perform a calibration, and plot the result.
https://python-dts-calibration.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
31 stars 18 forks source link

A mpi script for calibration #151

Open zrxdaly opened 2 years ago

zrxdaly commented 2 years ago

Is your feature request related to a problem? Please describe. I guess some may also need this to calibrate tons of files, I tested a script for mpi calibration.

  1. install mpi4py: https://mpi4py.readthedocs.io/en/latest/install.html
  2. code example mainly from here https://gist.github.com/joezuntz/7c590a8652a1da6dc4c9:

import dtscalibration from mpi4py import MPI ... comm = MPI.COMM_WORLD rank = MPI.COMM_WORLD.Get_rank() size = MPI.COMM_WORLD.Get_size()

def cali_fun(file):

calibration function

comm.Barrier()

for i, file in enumerate(grouped_files): if i%size!=rank: continue print("Task number %d being done by processor %d of %d" % (i, rank, size)) cali_fun(file)

BSchilperoort commented 2 years ago

Hi Dai, is this script just an example for how other people can do this?

Python also has the built-in multiprocessing module available, in case you just want to split up the jobs on your own machine.

zrxdaly commented 2 years ago

Hi Dai, is this script just an example for how other people can do this?

Yes, just thought it would be nice to share it here.

Python also has the built-in multiprocessing module available, in case you just want to split up the jobs on your own machine.

Oh nice, just tested it, multiprocessing module is also very handy to use for local machine. Since the data is quite a lot, I use mpi4py in Defltblue.