Currently the Bueno plugin handles the case where a single file is responsible for determining the schema and sampling metadata.
We need to support a list of files. The Plugin should build up the schema from the union of input files, and append one or more rows for each file parsed. If any row values are missing, we should fill them with null values.
At the end of a transload operation, we should report the following to the user:
Nothing if null values were not inserted
If null values are inserted, report a warning: the number of rows, the number of columns, the number of null values inserted, the percentage of rows and values which are null. A high percentage indicates that the files have very different contents, and may require different plugins or special handling logic in the existing plugin.
Currently the Bueno plugin handles the case where a single file is responsible for determining the schema and sampling metadata.
We need to support a list of files. The Plugin should build up the schema from the union of input files, and append one or more rows for each file parsed. If any row values are missing, we should fill them with null values.
At the end of a transload operation, we should report the following to the user: