For loops in AliasMethod

[dandelin]

https://github.com/HobbitLong/CMC/blob/master/NCE/alias_multinomial.py#L8

Hi!

While reading your code, I've noticed that for loops in the initialization function of AliasMethod causes a lot of computation.

However, the only entry (https://github.com/HobbitLong/CMC/blob/master/NCE/NCEAverage.py#L13) instantiating the class is passing torch.ones, which results in ones self.prob and zeros self.alias in AliasMethod.

What could go wrong if I let them just ones and zeros instead of running for loops while initializing AliasMethod?

Thanks for sharing the code :) (and RepDistiller too!)

[HobbitLong]

Hi @dandelin,

Thank you for finding it out. I think you can directly initialize them with ones and zeros instead.

Did you compare the difference of execution time between these two initializations? I thought the initialization function is only called once, so maybe the computation overhead is tolerable.

[dandelin]

@HobbitLong I'm trying a dataset larger than the ImageNet, so the initializing took some time (>15 minutes). After the fix, the initialization was done immediately. And yes, you are right since the actual training of CMC takes much longer than the initialization :) but it irritated me a lot while tuning the code so I left the issue.

+) Also I noticed that the memory bank took a few gigabytes on GPU RAM, so I moved them to a CPU RAM and it works fine for me with almost no overhead. Although I added additional lines of code to move read features to GPU RAM and store inferred features back to CPU RAM.

++) Are you planning releasing your experiments on NYU-Depth-V2? It will be helpful to understand the implementation of >2 views case.

[HobbitLong]

@dandelin,

1) Nice! Thanks for your experiment of moving memory back to CPU. How much is the relative decrease of speed? 2) Probably not now due to my limited capacity, but some hints are available here #9 and also our paper.

[dandelin]

@HobbitLong I did not measure the latency, but there was no difference in the feeling. Again, thanks for sharing the code :) closing the issue.