It seems that the luigi process under the hood does not use file extensions (or overwrites them to whatever the function template is). So we will need a careful way to sense the file type without loading the whole file into memory. From what I can tell, sensing the { at the start of a JSON should be a safe bet.
It seems that the luigi process under the hood does not use file extensions (or overwrites them to whatever the function template is). So we will need a careful way to sense the file type without loading the whole file into memory. From what I can tell, sensing the { at the start of a JSON should be a safe bet.