package up development branch

1. Made source code in development branch pip packageable

organized source code into a library called s5dev
setup pyproject.toml
updated requirements.txt

Why. Packaging this branch allows other projects to take advantage of the models developed in this branch for further development and application.

Caveats. I have restricted python version to 3.12.1 because that is what I have tested with. We should be able to relax the python version down to >=3.9.2, but would need to do additional testing and package version to verify.

2. Moved `GPT2Embeddings` module from `LMBackbone` to `SimpleLMHeadModel`

Why. Factoring out the exact embedding module and knowledge of input / output allows more reuse of LMBackbone module. LMBackbone now consists solely of the hidden blocks, plus final dropout and layer norm layers.

Testing

Created a fresh virtual environment and installed package and dependencies via

pip install -e .
pip install -r requirements.txt

Ran the small model

python train.py -o output_dir_name -c configs/hyena_S5/associative_recall_20_10_S5.yaml

and verified sensible validation and test metrics:

lindermanlab / S5