spencebeecher
commented
7 years ago Hi @tejasshah93 - Great question!
There isnt anything custom like that out of the box. However it is pretty easy to implement your own custom distance metrics. By default, pysparnn uses sparse matricies + cosine similarity. This is where the library really shines. I am not aware of many python based libraries that do sparse search accurately and efficiently (https://github.com/ekzhu/datasketch and https://github.com/src-d/minhashcuda both look good).
However here is an example (already in the code) of using an alternative distance. https://github.com/facebookresearch/pysparnn/blob/master/examples/dense_matrix.ipynb
Here is an example (that should mostly work) that will let you use custom distances. scipy.spatial.distance.cdist provides a variety of distance types. Caution - I have found these distance functions to be much slower than what is currently in pysparnn. This is probably reasonable for testing but writing customized & efficient code is probably the best way forward. More examples of custom distances in pysparnn can be found here: https://github.com/facebookresearch/pysparnn/blob/master/pysparnn/matrix_distance.py
class SlowEuclideanDistance(MatrixMetricSearch):
def __init__(self, records_features, records_data):
super(SlowEuclideanDistance, self).__init__(records_features,
records_data,
is_similarity=False)
self.matrix = self.matrix.toarray()
@staticmethod
def _transform_value(self, v):
return v
@staticmethod
def features_to_matrix(features):
"""
Args:
val: A list of features to be formatted.
Returns:
The transformed matrix.
"""
return np.array(features, ndmin=2)
@staticmethod
def vstack(matrix_list):
"""
Args:
val: A list of features to be formatted.
Returns:
The transformed matrix.
"""
return np.vstack(matrix_list)
def _transform_value(self, v):
return v
def _similarity(self, a_matrix):
"""Euclidean distance"""
# need to handle fipping argmin k to positive
return scipy.sparse.csr_matrix(scipy.spatial.distance.cdist(
a_matrix.toarray(),
self.matrix, 'euclidean'))
tejasshah93
Hi,
PySparNN looks great! I'd like to know if there is a provision for using custom distances as a metric while searching for nearest neighbors.
E.g. For sklearn.neighbors.NearestNeighbors, the documentation mentions:
Do we have such a provision for custom distances in PySparNN? Thanks.