Specify stop words to remove for key collision + fingerprint method

[plopenrefine]

Hello,

for my specific needs, i need to make a small adjustment to the way the key collision + fingerprint algorithm works. In a step of creating the clusters, the algorithm removes all punctuation and control characters. (In this step, i need to also remove some stop words: "press", "and", "co", "the", "a" for example).

It would be great if the clustering algorithm be made modular so that it could accept more values that the algorithm ignores during the cluster creation

The way that the key collision + fingerprint method works to create clusters is:

remove leading and trailing whitespace change all characters to their lowercase representation remove all punctuation and control characters (in this step i need to add some stop words that will not be taken into account) normalize extended western characters to their ASCII representation (for example "gödel" → "godel") split the string into whitespace-separated tokens sort the tokens and remove duplicates join the tokens back together

I do not want to remove the stopwords from the column i try to cluster before the algorithm takes place, i just need to find the clusters, and see the original values of the clustered values, but without having

Taylor Francis Taylor and Francis in two different clusters

I need them in the same cluster, as one value Taylor Francis

[wetneb]

Makes sense. I can see two approaches:

• make the fingerprinting function more configurable (more generally we need a better way for clusterers to expose parameters)
• make it possible to cluster not just a column, but the result of evaluating any expression on a column (just like facets can be customized with their own expressions). In this way, a lot of the preprocessing for fingerprinting could be done in an expression language already. One could express (some of) the existing preprocessing as a default expression that users could tweak. This is harder than the first option, since it requires significant changes to be able to forward clustering decisions back to the grid (remember which cell generated which expression value).

[plopenrefine]

i understand that adding this feature in the gui may take some time, which line should one alter in current version's code, so as to add the following stopwords along with the punctuation? "a", "and", "co", etc Thank you in advance

[wetneb]

You could add that here: https://github.com/OpenRefine/OpenRefine/blob/c76e2b9a461ed5b353ebf5c80e0e0cad2163331c/main/src/com/google/refine/clustering/binning/FingerprintKeyer.java#L93

[plopenrefine]

could someone show me an example of how the above line should be altered?

if i use say the following line, and alter it so as to replace the and with "" an empty string, shall i not achieve what i want?

https://github.com/OpenRefine/OpenRefine/blob/c76e2b9a461ed5b353ebf5c80e0e0cad2163331c/main/src/com/google/refine/clustering/binning/FingerprintKeyer.java#L57