RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

[windowshopr]

Python 3.7, Windows 10.

Reproducible code is below. As you'll see, it IS encased in the if __name__ == '__main__': but still getting the error. If you put ALL the code under the if __name__ == '__main__': you don't get the error, but then it doesn't seem to multiprocess? When you hit CTRL C to stop the program, it does shut down multiple processes, but when you look at your task manager/threads running, it's only running one at a time. Ideas?

from deap import base, creator
import random
from deap import tools
import multiprocessing

creator.create("FitnessMin", base.Fitness, weights=(-1.0,))
creator.create("Individual", list, fitness=creator.FitnessMin)

# Made this big so you can see the repeating multiprocessing error
IND_SIZE = 500000

toolbox = base.Toolbox()
toolbox.register("attribute", random.random)
toolbox.register("individual", tools.initRepeat, creator.Individual,
                 toolbox.attribute, n=IND_SIZE)
toolbox.register("population", tools.initRepeat, list, toolbox.individual)

def evaluate(individual):
    return sum(individual),

toolbox.register("mate", tools.cxTwoPoint)
toolbox.register("mutate", tools.mutGaussian, mu=0, sigma=1, indpb=0.1)
toolbox.register("select", tools.selTournament, tournsize=3)
toolbox.register("evaluate", evaluate)

# Multiprocessing
pool = multiprocessing.Pool()
toolbox.register("map", pool.map)

def main():
    pop = toolbox.population(n=50)

    CXPB, MUTPB, NGEN = 0.5, 0.2, 40

    # Evaluate the entire population
    fitnesses = list(map(toolbox.evaluate, pop))
    for ind, fit in zip(pop, fitnesses):
        ind.fitness.values = fit

    for g in range(NGEN):
        # Select the next generation individuals
        offspring = toolbox.select(pop, len(pop))
        # Clone the selected individuals
        offspring = list(map(toolbox.clone, offspring))

        # Apply crossover and mutation on the offspring
        for child1, child2 in zip(offspring[::2], offspring[1::2]):
            if random.random() < CXPB:
                toolbox.mate(child1, child2)
                del child1.fitness.values
                del child2.fitness.values

        for mutant in offspring:
            if random.random() < MUTPB:
                toolbox.mutate(mutant)
                del mutant.fitness.values

        # Evaluate the individuals with an invalid fitness
        invalid_ind = [ind for ind in offspring if not ind.fitness.valid]
        fitnesses = list(map(toolbox.evaluate, invalid_ind))
        for ind, fit in zip(invalid_ind, fitnesses):
            ind.fitness.values = fit

        # The population is entirely replaced by the offspring
        pop[:] = offspring

    return pop

if __name__ == '__main__':
    pop = main()
    print("Done")

[fmder]

You have to create your pool in the if name == main.

[windowshopr]

@fmder Thanks! I tried that now, but same issue as when it all goes under the if __name__ == '__main__':, in that it only appears to run 1 process, even with Pool() not defining 1 worker. I don't understand why it's doing that.

[windowshopr]

If you put the process pool OUTSIDE the name == main, it runs multiple workers, but with that error. Inside, it only runs one process, confirmed by watching thread performance during both.

[windowshopr]

I guess as a bandaid, I replace this:

    # Multiprocessing
    pool = multiprocessing.Pool()
    toolbox.register("map", pool.map)

...with this:

if __name__ == '__main__':

    # # Multiprocessing
    # pool = multiprocessing.Pool()
    # toolbox.register("map", pool.map)

    max_workers = 3

    with concurrent.futures.ProcessPoolExecutor(max_workers=max_workers) as executor:

        results = [executor.submit(main, ) for i in range(max_workers)]

        for f in concurrent.futures.as_completed(results):

            if f.result() != -1:

                pop = f.result()

...and it works, but would be nice to know that the program's params would work the same?