USEPA / WNTR

An EPANET compatible python package to simulate and analyze water distribution networks under disaster scenarios.
Other
310 stars 177 forks source link

Problems to run parallel processing with WNTR / EPANET #431

Open marinhosg opened 2 weeks ago

marinhosg commented 2 weeks ago

Dear colleagues,

I'm having some problems when try I to run my WNTR simulations with parallel processing. In sum, I'm trying to run a Water Net (".inp" EPANET file) and compute some parameters from this simulation, as bridge_density, central_point_dominance, todini_index, etc. However, I need to run this simulations several times, considering n hypothesis, and the total of simulations is around 6,000. So, I knew that using parallel processing would be an alternative to speeding up my simulations and save time. But, when I try to use multiprocessing or concurrent.futures tools, the simulations don't run or take more time than the thread process (normal analysis, one by one). Is that a way to solve this problem? I'll attach an example of what I'm working with. Sometimes, EPANET error 200 appears on screen. Thank you.

import wntr
import numpy as np
import pandas as pd
import networkx as nx
import copy
import matplotlib.pyplot as plt
import concurrent.futures
import multiprocessing
import time

[...]
# Import and create the network model from the original INP file
inp_file = "C:/Users/gabri/Desktop/dissertacao/SAO_RAFAEL.inp" #"C:/Users/ianca/Desktop/DADOS DE ENTRADA/REDE_COMPLETA.inp"
original_wn = wntr.network.WaterNetworkModel(inp_file)

[...]

# Obtaining tube roughness
pipe_roughness = original_wn.query_link_attribute('roughness')

# Defining Failure Probability Based on Roughness
failure_probability = pipe_roughness / pipe_roughness.sum()

# Desired pour sizes
tamanho_vazamento =[0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95, 1.0] 
# DataFrame vazio para armazenar os resultados da simulação
resultados_df = pd.DataFrame()

np.random.seed(67823)  # semente

def simulacao_rede(tamanho_vazamento, failure_probability,i):
    # Create a copy of the original network for this simulation
    wn = copy.deepcopy(original_wn)

    # Desired failure percentage
    porcentagem_of_failure = 0.02 #0.015 # 0.001 em 0.001

    # Calculating the number of tubes to be damaged
    num_links = len(wn.link_name_list)
    tubos_failure = int(porcentagem_of_failure * num_links)

    # Randomly choosing tubes to fail based on probability
    pipes_to_fail =  np.random.choice(failure_probability.index, tubos_failure, replace=False, p=failure_probability.values)

    [...]

    # Simulation with the modified network
    sim = wntr.sim.EpanetSimulator(wn)
    wn.options.hydraulic.demand_model = 'PDA'
    wn.options.hydraulic.required_pressure = 50
    wn.options.hydraulic.minimum_pressure = 10
    results = sim.run_sim(file_prefix=generate_file_prefix(i))

[...]
# PARALLELIZATION PROCESS HERE:

for aa in tamanho_vazamento:
    if __name__ == '__main__':
        with concurrent.futures.ProcessPoolExecutor() as executor:
            resultss = [executor.submit(simulacao_rede, aa, failure_probability, i) for i in range(num_simulacoes)]

            # Collect results
            results_list = []
            for future in concurrent.futures.as_completed(resultss):
                index = resultss.index(future)
                result = future.result()
                resultss_list.append([index, result])
                print(result)  # Print result (optional)

            # Append current results to the all_results list with context of b
            all_resultss.extend(((b, index), result) for index, result in results_list)

Environment

ucchejbb commented 2 weeks ago

Have you tried the multiprocessing package?

See the mp_run_epanet and runepanet functions in https://github.com/USEPA/PPMtools/blob/master/PPMtools.py

I have used multiprocessing package to run simulations in parallel in the past. I have not used the concurrent.futures packages in the past.