Data sheet manipulation operations

[geoo89]

Introduces operations on data sheets, specifically: Concatenation, filtering and sorting.

In content index sheets, data sheets can be defined by referencing a spreadsheet and a data model, and each row of the spreadsheet is parsed into instances of the data model. The resulting data is stored in an index which is referenced by the sheet_name (or new_name, if present).

It is now possible to create new entries in this index by taking one (filter, sort) or multiple (concat) existing entries and/or spreadsheet references, and specifying an operation to be performed on in the input. The output is stored in the index under the reference specified by new_name.

Important note: This breaks backward compatibility in one instance: Previous, it was possible to concatenate sheets implicitly by providing a list of sheets in the sheet_name column when defining a data sheet. Now the concatenation has to be made explicit by specifying concat as operation.type. Without this explicit operation, only the first sheet will be taken and ther others ignored, and a warning will be issued.

Also see https://docs.google.com/document/d/1Onx2RhNoWKW9BQvFrgTc5R5hcwDy1OMsLKnNB7YxQH0/edit

Fixes #103

[istride]

I can see three instances in ParentText Malaysia, where we use implicit concatenation of data sheets; none in South Africa. I can only check those two because those are the only ones we have archived a snapshot of the spreadsheets as CSVs, which I can grep over.

Would it be easy enough to avoid the breaking change? My suggestion is that we still consider the operation type to be 'concatenation', but only if:

• the type is 'data_sheet'
• there are multiple sheets listed
• no explicitly defined operation type

We log a warning if we encounter this situation to help the migration process. This would be a transition measure - a sunset period - after which, all source sheets must comply with the new rule of specifying an explicit operation type.

[fagiothree]

@istride FYI there are instances of concatenation in all parentText deployments and it is also used for the SRH chatbot

[geoo89]

I pushed a commit so that backward compatibility wrt implicit concatenation is maintained, and instead a deprecation warning is issued.