synapse-alpha / mirror-neuron

Experiments on bittensor reward models to find exploits
BSD 2-Clause "Simplified" License
1 stars 0 forks source link

Dendrite update #68

Closed steffencruz closed 1 year ago

steffencruz commented 1 year ago

Makes DummyDendritePool more customizable. Notably, individual dendrites can now have individual failure rates and also fitnesses. These values can be set when instantiating, and they are stored in uid-indexed dictionaries. Fitness is defined as the probability of the correct answer being returned, as shown below:

        # if an answer is requested, return correct answer with some probability
        if not is_question and random.random() < self.fitness.get(int(uid), 0):
            index = self.questions.index(messages[-1])
        else:
            index = random.randint(0, len(self.dataset)-1)

Also adds an extra plot: heatmap without exponential smoothing. This is useful to see the raw rewards and scores and visually confirm that dendrites are failing as often as expected

steffencruz commented 1 year ago

A note on visually confirming the dendrite fail_rates. The plot below, taken from wandb shows the rewards versus simulation steps for each uid.

Screenshot 2023-05-09 at 12 44 25

Based on this plot I think that everything is fine with the reward allocation to failing dendrites. You can see in the plot attached (no EWM) that UID 1 3 and 5 have more sparsity (should have 0.1, 0.3, 0.5). You can also see from the tooltip that I am filling UIDs with NaN when their are either not queried or are not successful. This seems to affect the EWM window and also the overall statistics since NaNs are ignored and the mean only considers the values that are numeric.

The config file contains the new dendrite pool:

    dendrite_pool:
      name: DummyDendritePool
      args:
        data_path: nq_open
        baseline_fail_rate: 0.0
        fail_rate:
          1: 0.1
          3: 0.3
          5: 0.5
          7: 1.0
        baseline_fitness: 0.0
        fitness:
          2: 0.2
          4: 0.4
          6: 0.6
          8: 0.8

Which sets individual uids to different failure rates. In

steffencruz commented 1 year ago

Closes #52 and #57