Running genomes in parallel

[jvanakker]

My app has different fundamentals from the provided example.

As far as I understand correctly, the provided example does the following:

• It goes through the entire population, where for each performs a series if network.run()
• After each series of network.run(), it calculates the fitness level, assigns it to the genome using network.nextGenome(fitnesslevel).
• When the whole population is done, it does network.epoch() to generate the new mutations, and repeats the process.

However, my application works like this:

• Each entity in the population does a network.run() at each timestep, as long as the simulation endures. At the end of the simulation, I have the final fitness level for each one in the population, and want to assign it to its genome, then do a network.epoch(). So far, that's more or less the same as the provided example. However:
• The simulations for each entity are running parallel to each other. (In my case in the same visual 2D environment).
• I can't assign the genome using network.nextGenome(currentFitnesslevel) after each network.run(), because I guess that messes up the assigned genomes in NEAT-swift, as the others are running in parallel for each timestep in the simulation. I think it causes all sorts of issues, like the next one to run the wrong genome the next timestep.

I think the solution would be to have a specific index/identifier for the genomes, and be able to call network.run() with this specific identifier, and also assign the fitness using that specific identifier. Before I dive into modifying the library, is there any way that I can already use your NEAT-swift in this way?

[troydeville]

I haven’t looked at the project in quite some time. I’ll review the issue this afternoon!

[jvanakker]

Maybe I should elaborate a bit more on "running in parallel". It doesn't have to do with actually running in parallel or multithreading. Rather, what I meant was that the simulation has to go through each entity in the population every timestep (or X amounts per second), to perform a network.run() to get control outputs. Then next timestep, each one should still use their same genome used the previous timestep.

That's the major difference; with your example implementation, for each one in the population it finishes running through the entire input dataset one by one, then update the fitness with network.nextGenome so the network knows it has to go to the next genome at the next run().

Right now there doesn't seem to be a way to let the network know what genome (or rather: population index) it should run, without mutating. For my use case, it should be able to go through the entire population multiple times, and only have the genomes mutate when I provide the fitness at the end of that particular run.

[troydeville]

I believe I understand what you're saying. Typically, there are a few ways to go about integrating this framework into your project.

One way is that you would have an existing training dataset. With this dataset, you would then iterate through each genome and simultaneously assigning the fitness to the genome being iterated through the dataset. After the entire iteration of the genome, you would then do NEAT ("network.epoch()").

Another way is that you would have a simulation created in which one genome at a time is tested in the simulation. So, the population size determines the amount of times the simulation is ran through per generation and each genome that runs through the simulation would be assigned a fitness score at the end of each simulation. After each genome in a given population is ran through the simulation then you would do the network.epoch() (which does NEAT). Each genome needs a fair chance to be tested so that competition between species is accurate. Also in the simulation case, inputs would be simulation variables and outputs would be actions that cause changes in the simulation. So only one genome should be tested in the simulation at a time. Which this framework should support as long as you sequence it in the correct order.

In your case, it looks like you're trying to do a simulation approach by using the existing training dataset method. Typically in a simulation, you would want to update the genome inputs per refresh cycle of the simulation. Side note, if your network gets big enough, it could potentially slow down the simulation, but that depends on your CPU and NEAT tries to minimize the network size naturally. It would have to be a rather large network to slow down the simulation or you could adjust the refresh time.

Key takeaway: each genome in a population must be tested and assigned a fitness score before network.epoch() is used.

Let me know if this didn't clear things up.

[jvanakker]

Thanks for your feedback! Alright, to be more concrete, to learn more about using NEAT, I've created a simple simulation environment with 'cars' on a track, exactly like this: https://www.youtube.com/watch?v=wL7tSgUpy8w

Like in the video, for each car I have 5 radar distances as inputs for the network, the net then provides a number of output values that I convert to use as simple boolean controls (steer left, steer right, go fast, go slow). There is no existing training dataset, as the input values are generated in realtime.

The only difference from the example video above, that he's not using NEAT (but his own highly similar variant), and he manually assigns fittest candidates after each run. For me that's not required, as I already have created a way to see how far the car has driven on the track before it crashes into a wall, which is equal to the fitness level. Then, using NEAT I should be able to reach similar results.

I've already integrated your library, but like you said it is designed so that each genome is run through an entire simulation at a time, while I'd like the whole population to run simultaneously during each generation, like shown in the video above. I think that your NEAT library can still be used for that, with some modifications.

Any thoughts? Thanks for your help so far!

[troydeville]

I see what you're saying about simultaneously now. And yes, the library can absolutely be modified to do that. I'll look into it. There shouldn't be much modification involved and would be a nice feature to have.

[troydeville]

Check out new fix and let me know if that solved your problem. The typical code reflects changes.

[jvanakker]

Thanks, I appreciate your help! I tried it, but it doesn't fix the problem yet.

For this use case, I'm iterating through the whole population every timestep in the simulation. So for each species, network.run() is called. Then at the next frame, it again iterates through the whole population to do a network.run(). Every frame/timestep, it should still be using the same genome. And so forth, and so forth. Until the simulation is done. Only after that, I move on to add the new genomes to the species and go to the next generation.

The main problem in short: multiple iterations through the whole population need to be done, while still using the same genome all those times. With the current changes, there still doesn't seem to be a way to do this. Yes, it does keep the genomes the same until calling nextGenomeStepTwo(), but it doesn't allow for more than one iteration through the population, as far as I understand correctly.

[troydeville]

There needs to be a NEAT-Swift v2.0 rewrite. But I did create a quick fix. The way you have to do it is by updating the fitness per genome per frame of the simulation. You could create a class that has variables for your sprite or whatever you're using in which the variables in this class would contain a fitness value. So you'd update this value continuously, then after the simulation is over you'd perform nextGenomeStepTwo(). So in each frame of the simulation you would iterate through every genome and update it's input to get the output. You'd also assign its current fitness into nextGenomeStepOne(fitness). The idea is to have a fitness that is constantly updating.

I created another similar version of NEAT called ANT, but I'm not sure I put on here the latest version.

[jvanakker]

I think that's what I'm doing now.

Every car in the population is simulating it's state independently every frame, including keeping track of its fitnesslevel. Then, as for the neural control loop I perform the following, simplified:

for i in 1...self.populationSize
{
let car = self.cars[i-1]
if(car.isCrashed == false) // car is not crashed into wall
{
let output = self.network!.run(inputs: inputData, inputCount: inputData.count, outputCount: 5)
self.network!.nextGenomeStepOne(Double(car.fitnessLevel!))
}
}

Then, based on the output, I steer that particular simulated car.

However, at the next frame I'm repeating the same iteration again. This is where NEAT-Swift runs in to problems (at leat, I suspect, because there are no visual improvements to the population at all, after multiple generations). This is because it can't know the correct population index for the run() and nextGenomeStepOne() during the 2nd iteration where it should still use the same genome.

I think I will try modifying NEAT-Swift so I can specify the population index with run() as well as when updating the fitness value.

If I succeed, I'll make sure to put it on a Github fork with pull request, so you can check it out. Also I'll put the neural car simulation on Github. It's a simple Swift macOS application that uses Apple SpriteKit, without any external dependencies other than NEAT-Swift.

[EDIT] I just noticed you already have 'looping' logic inside nextGenomeId(), where currentGenomeKeyId goes back to 0 after reaching the last in the population. If that works correctly, I guess my integration of NEAT-Swift should already work, in theory... To be continued.

[troydeville]

You can try modifying it, but it uses BTree which makes things complicated. I don't use that in other projects.

I created a simulation on my own today using SpriteKit and it works for multiple genomes. It should work. My simulation involves a rocket landing on a platform with a population of 256. Each 256 genomes are tested in a frame and the fitness level is incremented to it's current fitness level. Once all the rockets are destroyed, run out of fuel, or land the target, then I call the 2nd step. So it's working on my end at the moment.

[jvanakker]

Status update:

Confirmed, your last version actually work correctly! There was a flaw in my last implementation, I only did the following for the 'living' cars:

let output = self.network!.run(inputs: inputData, inputCount: inputData.count, outputCount: 5)
self.network!.nextGenomeStepOne(Double(car.fitnessLevel!))

However, I also had to do this for the cars that were already crashed, in order to keep the currentGenomeKeyId cycle correct in NEAT-Swift: self.network!.nextGenomeStepOne(Double(car.fitnessLevel!))

After doing that, it now performs flawlessly! In order to prevent other issues I also use: private let queue = DispatchQueue(label: "com.jva.neuralracer",attributes: .concurrent) together with queue.async(flags: .barrier), in order to keep everything safe with my separate update thread. This way the framerate of SpriteKit always stays stable at 60 fps, no matter if I add a population of 10 or 600. (I don't actually do the neural control every frame, rather every 0.1 second to reduce the load).

They cars now actually learn to steer around the track amazingly well, it's a very nice demo showing how great NEAT works. Having bigger populations does increase chances of getting good results much quicker after only a few generations.

I know it's off-topic here, but I will continue playing with NEAT by:

• Adding/removing inputs for the net
• Besides track progress, also rewarding higher fitness for going around the track with a higher speed
• Changing configuration values in NConfiguration.swift
• Generating dynamic tracks/environments instead of a fixed reference image. Currently I use NSBitmapImageRep containing the reference image for a race track, on which I do various pixel-position lookups for calculating the cars radar distances etc. Moving over from this system to a dynamic environment, I can improve testing of a trained network.

After some generations, I still do sometimes experience a crash, Fatal error: Unexpectedly found nil while unwrapping an Optional value: file NeuralRacerSwift/NNetwork.swift, line 97, but this may be caused by something on my side. At least it's not related to the issue of this thread.

I consider this issue closed!

[troydeville]

Awesome!

Can you open a new issue about "After some generations, I still do sometimes experience a crash, Fatal error: Unexpectedly found nil while unwrapping an Optional value: file NeuralRacerSwift/NNetwork.swift, line 97, but this may be caused by something on my side. At least it's not related to the issue of this thread."

Or once you get the error again.