search

D-X-Y / AutoDL-Projects

Automated deep learning algorithms implemented in PyTorch.
MIT License
1.56k stars 281 forks source link

Regarding Gumbel-Softmax #14

Closed xz6014 closed 5 years ago

xz6014 commented 5 years ago

Hi there,

Just wondering how was the gumbel-softmax implemented in your scripts. Was a hard one-hot sample generated at each iteration of the searching phase, or a soft one? I noticed there's an option in the torch.nn.functional.gumbel_softmax() of setting 'hard' to true to generate a one-hot sample. Was this the approach that you took?

Many thanks,

X

D-X-Y commented 5 years ago

It is a hard one-hot sample generated at each iteration per GPU (If you use multiple GPUs). Our implementation is similar to that of the hard mode of torch.nn.functional.gumbel_softmax(). Please note that torch.nn.functional.gumbel_softmax() has bugs, so please be careful if you are using that function. See here https://github.com/pytorch/pytorch/pull/20179

xz6014 commented 5 years ago

Thank you very much for your clarification. It's greatly appreciated.