[Question] How did you implement 1-bit tensor?

kyegomez / BitNet

Implementation of "BitNet: Scaling 1-bit Transformers for Large Language Models" in pytorch

https://discord.gg/qUtxnK2NMf

MIT License

1.69k stars 155 forks source link

[Question] How did you implement 1-bit tensor? #7

Closed Beomi closed 9 months ago

Beomi commented 1 year ago

Hi, I found this repository when I'm plan to impelment BitNet.

BitLinear use 1-bit, but, since pytorch native dtype does not supports 1bit tensor, So i thought I need to implement via custom cuda kernel.(bit packing and unpacking)

How did u implement 1bit tensor implementation? (I can't find cuda files yet)

eisneim commented 12 months ago

i'll be interested to know this as well🤔

github-actions[bot] commented 10 months ago

Stale issue message