Optimize sparse * random dense matrix multiply in LsiModel

[ghost]

Progress tracking bug.

• [x] Fork https://github.com/brianmingus/gensim
• [x] Find initial work https://gist.github.com/brianmingus/ff6c3e9816c2cb7de0efc17d9bed06e6
• [x] Get most up-to-date CSparse. This was extracted from SuiteSparse-5.1.0 and placed in gensim/gensim/CSparse
• [x] Find example gensim Cython compilation and refactor our setup.py code to comply
• [x] Refactor the python code into a gensim.csparse module
• [ ] Refactor the test code into a gensim-style test that ensures the new code produces identical results and is faster
  • [x] Figure out why pmultiply is returning an array of 0s
  • [ ] Get the test to actually run
• [x] Current import structure is gensim.csparse.psparse is this ideal
• [x] Modify LsiModel to use pmultiply
  • [ ] Test that LsiModel works
• [x] Look into other uses of sparsetools, see what can be removed
• [ ] Fix import structure. csparse/gaxpy seems to clobber csparse/psparse/pmultiply

[menshikh-iv]

Hi @brianmingus, glad to see you here :+1:

About Cython compilation:

• You should add your file to ext_module https://github.com/RaRe-Technologies/gensim/blob/1afaacc8017ff2b51e8677b8b36cd3b9671b20be/setup.py#L252-L262 (include_dirs=[...] will be different in your case)

• You should translate/compile your code as

  • cython path/to/your/great.pyx - translate .pyx to .c
  • python setup.py build_ext --inplace - compile .c to .so (or something else, if you use not linux)

[ghost]

Just remembered that I have some local modifications to cs.h which will hopefully fix the integration.

/* #ifdef MATLAB_MEX_FILE */
/* #undef csi */
/* #define csi mwSignedIndex */
/* #endif */
/* #ifndef csi */
/* #define csi ptrdiff_t */
/* #endif */

// FORCE use of 32 bit int offsets, because thats what scipy uses so we can
// avoid a copy.
#define csi int32_t

[menshikh-iv]

Related PR #1896