search

timaeus-research / devinterp

Tools for studying developmental interpretability in neural networks.
77 stars 14 forks source link

readme

DevInterp

PyPI version Python version Contributors Docs

A Python Library for Developmental Interpretability Research

DevInterp is a python library for conducting research on developmental interpretability, a novel AI safety research agenda rooted in Singular Learning Theory (SLT). DevInterp proposes tools for detecting, locating, and ultimately controlling the development of structure over training.

Read more about developmental interpretability.

:warning: This library is still in early development. Don't expect things to work on a first attempt. We are actively working on improving the library and adding new features.

Installation

To install devinterp, simply run pip install devinterp. (Note: This has PyTorch as a dependency.)

Minimal Example


from devinterp.slt.sampler import  sample, LLCEstimator
from devinterp.optim import SGLD
from devinterp.utils import default_nbeta

# Assuming you have a PyTorch Model assigned to model, and DataLoader assigned to trainloader
llc_estimator = LLCEstimator(..., nbeta=default_nbeta(trainloader))
sample(model, trainloader, ..., callbacks = [llc_estimator])

llc_mean = llc_estimator.get_results()["llc/mean"]

Advanced Usage

To see DevInterp in action, check out our example notebooks:

For more advanced usage, see the Diagnostics notebook Open In Colab and for a quick guide on picking hyperparameters, see the above Grokking Demo Open In Colab or the the Calibration notebook. Open In Colab. Documentation can be found here. Docs

For papers that either inspired or used the DevInterp package, click here.

Known Issues

If you run into issues not mentioned here, please first check the github issues, then ask in the DevInterp Discord, and only then make a new github issue.

Contributing

See CONTRIBUTING.md for guidelines on how to contribute.

Credits & Citations

This package was created by Timaeus. The main contributors to this package are Stan van Wingerden, Jesse Hoogland, George Wang, and William Zhou. Zach Furman, Matthew Farrugia-Roberts, Rohan Hitchcock, and Edmund Lau also made valuable contributions or provided useful advice.

If this package was useful in your work, please cite it as:

@misc{devinterpcode,
  title = {DevInterp},
  author = {van Wingerden, Stan and Hoogland, Jesse and Wang, George and Zhou, William},
  year = {2024},
  howpublished = {\url{https://github.com/timaeus-research/devinterp}},
}