Bug Fix: float32 -> 32-true

fkodom / yet-another-retnet

A simple but robust PyTorch implementation of RetNet from "Retentive Network: A Successor to Transformer for Large Language Models" (https://arxiv.org/pdf/2307.08621.pdf)

MIT License

100 stars 15 forks source link

Bug Fix: float32 -> 32-true #15

Closed fkodom closed 1 year ago

fkodom commented 1 year ago

Change in how input projections are implemented. seem to converge faster
Fix incorrect float32 precision string