Changing the "activation_mutate_rate" throws an error

[Kuselokusi]

Hi, I have recently attempted to change the "activation_mutate_rate" in my config file however I get the error: "neat.activations.InvalidActivationFunction: No such activation function: '[softplus,' "

But this does not make sense because this is a valid activation function. And this activation function works when "activation_mutate_rate = 0.0" which make this error quite strange.

Please see my config file below

[NEAT]
# changed
# now 78% was 80% fitness used on training set
fitness_criterion     = max
fitness_threshold     = 182 
pop_size              = 220 
reset_on_extinction   = True

[DefaultGenome]
# node activation options 
# changed
activation_default      = softplus 
activation_mutate_rate  = 0.02
activation_options      = [softplus, sigmoid] 

# node aggregation options
aggregation_default     = sum
aggregation_mutate_rate = 0.0
aggregation_options     = sum

# node bias options
bias_init_mean          = 0.0
bias_init_stdev         = 1.0
bias_max_value          = 60.0
bias_min_value          = -60.0
bias_mutate_power       = 0.5
bias_mutate_rate        = 0.7
bias_replace_rate       = 0.1

# genome compatibility options
compatibility_disjoint_coefficient = 1.0
compatibility_weight_coefficient   = 0.5

# connection add/remove rates
conn_add_prob           = 0.5
conn_delete_prob        = 0.5

# connection enable options
# changed
enabled_default         = True
enabled_mutate_rate     = 0.05 

feed_forward            = False
initial_connection      = full

# node add/remove rates
# changed
node_add_prob           = 0.5 
node_delete_prob        = 0.5 

# network parameters
# changed
num_hidden              = 0
num_inputs              = 3 
num_outputs             = 1

# node response options
response_init_mean      = 1.0
response_init_stdev     = 0.0
response_max_value      = 60.0
response_min_value      = -60.0
response_mutate_power   = 0.0
response_mutate_rate    = 0.0
response_replace_rate   = 0.0

# connection weight options
# changed
weight_init_mean        = 0.0
weight_init_stdev       = 1.0
weight_max_value        = 60
weight_min_value        = -60
weight_mutate_power     = 0.5
weight_mutate_rate      = 0.9
weight_replace_rate     = 0.1

[DefaultSpeciesSet]
compatibility_threshold = 3.0

[DefaultStagnation]
species_fitness_func = max
max_stagnation       = 20
species_elitism      = 2

[DefaultReproduction]
elitism            = 2
survival_threshold = 0.2

[scolemann]

You do not need the bracket or commas. It should look like this: activation_options = softplus sigmoid

Without digging into the code my assumption is that since the activation mutation rate is 0.0 it only uses the activation_default softplus and thus never tries to read activation_options.

[Kuselokusi]

Thank you, I didn't realise that was the mistake I was making. Is the use of two activation functions with a mutation something that is advisable when applying neat to a particular problem. I ask because if I were to use an activation_mutate_rate of 0.01 and give two activation functions in the options, how does someone know when the activation function has been mutated over the generations?...can this information be extracted somehow using code?

[scolemann]

You can evaluate each individual genome and see what nodes it has. You can then traverse the nodes and see what the activation and aggregation type is on each.

The mutation rate is probably of less concern than node_add_prob and node_delete_prob which are set at .5. This means you have a 50% chance of adding or deleting a new node and it's default will default to the activation default function.

Depending on the complexity of the problem you are trying to solve you may need more or less nodes and the closer you can get to that from the start the better. The more nodes you have the longer your search time will be.

Having multiple activation functions is fine and you can include those that you think will work well with your problem space. You can also set the default to random so that the initial population starts with different node activation types.