Open organisciak opened 4 years ago
Bump
Looks like I did fix this locally a few months ago--just pushed to the main branch. Using the version with the product to keep the results identical.
@organisciak reopening because it's at least co-culpable in the error at https://github.com/htrc/htrc-feature-reader/issues/42--out of memory problems on HPC. Rolling back to the old version doesn't raise OOM errors, reading the np.average
code, , it looks it broadcasts out the weights to a big array and then swaps them.
I'm going to try a different approach for the time being, which is to call np.matmul instead of np.dot. It appears that's the recommendation for 2d x 2d arrays anyway, so perhaps it will compile better.
If not, it would possible to use the numpy average code without the final division which might provide a marginal speedup anyway, while keeping results essentially the same as the old dot product code.
I've been trying to debug our
vectorization.py
code for converting EF files to SRP and GloVe. It has trouble parallelizing on my system, because all the cores quickly get used up.It seems to be an issue with the dot product, which is used in this repo as well as my GloVe code for taking a weighted sum of word vectors. https://github.com/bmschmidt/pySRP/blob/3d873fabdf05269ed904638657b27cb12c3b8177/SRP/SRP.py#L213
After some comparison, it seems that a weighted average with
np.average
works faster, even with one process (when it doesn't clog up all CPUs). If you want a sum, it's possible to multiply the result by the sum of weights - for cosine distance or when using normed vectors, it shouldn't matter though.Here's the fix that works faster for me:
Or to make the vectors identical to the dot product
This may be specific to my system, because I know that linear algebra is hard to set up and there may be some BLAS magic that Numpy is trying and failing. Perhaps you can test on another system? If the speedup from the fix isn't universal, can I request an argument to stable_transform to switch to it when parallelizing?