Input data schema validation

[jiqicn]

Investigate possible ways of validating input data against the ontology.

[jiqicn]

Nodes, Components, and Workflows on KNIME Hub

1. Table validator and Table validator (reference)

• These two nodes ensure that the result table structure is mostly identical to the reference specification.
• Validation is done by resorting of columns, the insertion of missing columns (filled with missing values), and optional removal of additional columns.
• It's also possible to config if each column is required and if the data type or the domain should be checked/converted.
• The main difference between these two nodes is that the former accepts the reference specification via the configuration of the node, while the latter can use another input table as the reference.

2. R and Python Integration

• It's also possible to write some R/Python code with the R/Python Integration nodes (most possibly the Python Script node for Python and the R Snippet node for R).
• The drawback is that it requires the users to configure and manage the R/Python environment themselves (see this link in case of Python and this link in case of R).

3. Interactive Data Cleaning Component

• This is not something purely designed for validation, but also for data cleaning.
• This brings a new direction of thinking, that is to allow interactive data validation within GUIs.

[dafnevk]

The Table validator nodes seem very useful and close to what we need. It seems there is no compatibility with the CSV on the web specification, which is a more fomalized and interoprable way of describing the CSV table than with a reference table or within the configuration.

[dafnevk]

Here is an existing csv-on-the-web validator in Ruby that could be used for a customized node.

Also a python implementation (not well documented and seems to be python 2) and an R implementation.

[dafnevk]

In case we want to use the Ruby implementation, here is a ruby wrapper node for Knime. This works with jruby, a java implementation of Ruby. It's not clear to me how this wrapper nodes handles the ruby environment and dependencies, I think jruby and all dependencies already need to be installed on the system. See also documentation of the jruby node.

[jiqicn]

I also found a CSV on the Web validator in java that might be useful for developing the customized node.

At bottom of this note, you can find implementations of csvw validator in different languages (Python, Ruby, Javascript, Web, and R).

[dafnevk]

Note that the csvlint (ruby) implementation also has a webservice with an API: http://csvlint.io/documentation

[dafnevk]

Another python implementation, built in Clariah: https://github.com/clariah/cow

[jiqicn]

Deploy KNIME extension

As this note said, there are two ways of sharing the extension.

One is to build a local update site for the extension. The ideal situation is to become a contributor to the KNIME community, but that requires many efforts (see this link). It's also possible to have the local update site shared in different ways (e.g. github), but in that case, a dropin will be more convenient than a local update site.

The second way is to wrap the extension as a dropin, which is actually a .jar file. When deploying the extension, users simply needs to put the dropin file in the dropins folder of the local KNIME installation and restart KNIME.