Enhance String Comparison Utilities

After looking over its documentation in more detail, I've seen that splink has some nice utilities for visualising string comparison metrics and creating comparison levels from templates.

In particular, I've found that defining comparison levels from thresholds of a similarity score is something that I regularly need to do, so having a template that constructs this from a set of thresholds would be handy.

@NickCrews I'd be happy to make a start on implementations of these, along with relevant documentation on their usage, if you think they would be useful generally.

As a side-note, I'm also interested in the concept of term-frequency adjustments, but am unsure how it would extend to strings that are sets of terms. Is this something you've come across?

Here's something I've been developing for a defining a LevelComparer using a comparison function and a set of thresholds. The basic idea is to use a factory function to create a MatchLevel subclass and dynamically create the cases required from the given UDF and thresholds. I haven't thought of a neat way to do this yet. Perhaps the simplest way is to require the user to specify the columns from the paris table

from __future__ import annotations

from mismo.compare._match_level import LevelComparer, MatchLevel
from typing import Callable, Any, Literal

def match_levels_factory(name, **levels) -> type:
    """A factory function for creating a MatchLevel class

    Examples
    --------
    >>> NameMatchLevel = match_levels_factory('NameMatchLevel', EXACT=0, NEAR=1, ELSE=2)
    >>> isinstance(NameMatchLevel, MatchLevel)
    True
    """
    return type(name, (MatchLevel,), levels)

class ThresholdComparer(LevelComparer):
    """Assigns a MatchLevel based on the threshold of a user-defined comparison function."""
    def __init__(self, name: str,
                 comparison_func: Callable[[Any, Any], float],
                 thresholds: list[float],
                 *,
                 representation: Literal["string", "integer"] = "integer",
                 add_null_level: bool = True,
                 add_exact_level: bool = True,):
        self.thresholds = thresholds
        self.fname = comparison_func.__name__
        levels = {...}
        match_level = match_levels_factory(self.fname+"Level", **levels,  **threshold_levels)
        cases = [...]
        super().__init__(name=name,
                         levels=match_level,
                         cases=cases,
                         representation=representation)

    def __repr__(self):
        return f"ThresholdComparer(name={self.name}, func={self.fname}, thresholds={self.thresholds})"

NickCrews / mismo

Enhance String Comparison Utilities #66