String as value type of DenseMatrix

[daphne-eu]

In GitLab by @pdamme on Mar 18, 2022, 11:42

So far, DAPHNE supports strings only for scalars, but not for the values in a matrix or frame. However, data sets with string columns are commonplace and play an important role in both machine learning and database applications. Therefore, we must support those.

The task is to implement (C++) a specialization of DenseMatrix for the string value type. The major challenge is that each value could have its individual length. Instead of storing each value as a separately allocated string, a more compact and cache-friendly structure is desired. For instance, one could store the concatenation of all string values in one large buffer, and make the entries of the matrix pointers into that buffer. The data structure shall implement the typical interface of a matrix in DAPHNE, including get/set/append for individual values, direct access to the underlying buffers, and the creation of (zero-copy) views (row/column segments) into the data.

In addition to the data representation itself, a handful of important operations on a matrix of strings shall be implemented, whereby the operations can be chosen depending on personal interests and team size. Interesting candidates include:

• elementwise comparisons (to another matrix and/or scalar): (in)equality, lexicographical order, SQL-style LIKE with wildcard %
• dictionary encoding (assigning unique numerical codes to distinct strings in a matrix)
• elementwise concatenation
• reading/writing string columns in CSV files (extension of the existing CSV reader/writer)
• ...

[daphne-eu]

In GitLab by @pdamme on Mar 22, 2022, 18:27

mentioned in commit 783dd44680f7c8483ab768b7c38dd952f90a05f8

[akroviakov]

Assigned to @akroviakov.

[pdamme]

@akroviakov: I recommend specializing for DenseMatrix<const char *>, since we use const char * (not std::string) to represent string scalars at the moment. But if that turns out to be difficult, we can talk about it.