search

ApolloResearch / rib

Library for methods related to the Local Interaction Basis (LIB)
MIT License
2 stars 0 forks source link

WIP: Adds ability to set off-diagonal weights in modular mlp #331

Closed LuciusApollo closed 2 months ago

LuciusApollo commented 5 months ago

Off diagonal modular MLP

Description

Adds the ability to initialise weights at chosen layers in modular mlp to non-zero random values. Mainly modifies rib/models/modular_mlp.py, and adds the new attributes along with some other small updates to the default configuration file rib_scripts/rib_build/block_diagonal.yaml. Also adds "build modular mlp" as an entry to launch.json.

Related Issue

Motivation and Context

We wanted to add the ability to make slightly more challenging MLP toy models to test RIB on.

How Has This Been Tested?

RIB graphs in the neuron basis with squared edges seem to come out looking as they should, meaning no off-diagonal edges save for those in layers with off-diagonal weights. Graphs in centred Jacobian basis looked sane-ish to the bare eye, but weren't actually checked for modularity. Using the debugger with the new setting has not been tested.

Does this PR introduce a breaking change?

stefan-apollo commented 2 months ago

We decided not to implement this