Folder Input Source type

[marc-portier]

content for one set could actually be coming from a folder of similar json or xml files that all capture 1 record

typical case: dump of *json files from e.g. IMIS

processing these all in one go as one 'set' seems needed

[marc-portier]

work involved inside sources.py

• change the SourceFactory : preceed the mime-type checking with detecting os.path.isdir() so to defer to the new FolderSource
• implement the FolderSource so it lists all files in the folder and offer those as different records in the set:
  • consider defering to the SourceFactory based on mimetype to actually grab the content
  • either require all files to be same mimetype, or allow mixed format
  • probably unnecessary to consider recurseively running through nested folders
  • ATTENTION: Think wisely about the interaction between the two needed aspects:
  • iterator (i.e. __iter__() and __next__()) and
  • context manager (i.e. _enter__() and __exit()__)
• be sure to add tests - providing a folder tests/in/data-xyz/*.csv or other files that gets used in a new or changed template

[marc-portier]

note: when reusing the different XXXFileSource implementations be aware that those too all return iterators (through arrays or over wrapped struct objects) - so the overal iteration over the folder of these should simply behave as a concatenation of all those lists --> after ending the iteration in one, we should simply go over to the next