Currently, data.csv, data.json, and data.txt data sources use either path or glob (data.json) config params. This limits the applicability of the data block -- with data.json, it is possible to load multiple files, but not with data.csv. The variability between the files should not matter -- Fabric data blocks return a json object already, so they can accommodate multiple files with misaligned data structures.
Design
For data.csv and data.txt data sources, use glob instead of path, and add ability to read multiple files
Background
Currently,
data.csv,data.json, anddata.txtdata sources use eitherpathorglob(data.json) config params. This limits the applicability of the data block -- withdata.json, it is possible to load multiple files, but not withdata.csv. The variability between the files should not matter -- Fabric data blocks return a json object already, so they can accommodate multiple files with misaligned data structures.Design
For
data.csvanddata.txtdata sources, useglobinstead ofpath, and add ability to read multiple files