search

huggingface / peft

🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.
https://huggingface.co/docs/peft
Apache License 2.0
16.33k stars 1.61k forks source link

Gumbel-sigmoid (poly) #1983

Closed adws2 closed 1 month ago

adws2 commented 3 months ago

I don't understand the implementation of the Gumbel-Sigmoid part.

How can Eq.2 in poly paper become below?

skill_logits = RelaxedBernoulli(temperature=1., logits=skill_logits).rsample()

Also, isn't z_{ij} binary? It doesn't seem to be binary in peft code.

BenjaminBossan commented 3 months ago

@TaoSunVoyage could you please take a look?

TaoSunVoyage commented 3 months ago

@adws2 for this part, i followed the implementation of the original polytropon repository. https://github.com/McGill-NLP/polytropon/blob/d567ea838cb8b76b75c5e3135ac1a132ec77ebbc/src/polytropon/adapters.py#L159

github-actions[bot] commented 1 month ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.