Manual SwiGLU implementation

Modalities / modalities

Modalities, a PyTorch-native framework for distributed and reproducible foundation model training.

MIT License

59 stars 5 forks source link

Closed mali-git closed 3 months ago

mali-git commented 3 months ago

This PR adds a manual SwiGLU implementation. The original one from xops was imcompatible with activation checkpointing (see issue #14)

General changes:

Breaking changes:

renaming of fused_swiglu to swiglu in ActivationType (see here for the respective config changes)

le1nux commented 3 months ago

fixes #14