*Minor bug-fix / future-proofing: seems not actually a current bug, but based on different behavior in airflow, also seems likely enough to become one in the future if not protected against
In building the python/airflow data-handling side of the data_frame loader for Polyphemus, I noticed that the pandas.read_table() file read-in did not automatically detect and adjust for csv vs tsv files. Possibly (not confirmed) from a pandas version difference with an adjustment to default behavior. In any case, implies a need to be explicit about what we want to happen. This PR does that.
In 'Fetch Data' step of the 'Visualization from File' workflow:
Adds determination of whether to use separator of "," or "\\t" based on the file extension, and passes this along in the read-in, pandas.read_table(open_file, sep = separator)
*Minor bug-fix / future-proofing: seems not actually a current bug, but based on different behavior in airflow, also seems likely enough to become one in the future if not protected against
In building the python/airflow data-handling side of the data_frame loader for Polyphemus, I noticed that the
pandas.read_table()file read-in did not automatically detect and adjust for csv vs tsv files. Possibly (not confirmed) from a pandas version difference with an adjustment to default behavior. In any case, implies a need to be explicit about what we want to happen. This PR does that.In 'Fetch Data' step of the 'Visualization from File' workflow:
separatorof","or"\\t"based on the file extension, and passes this along in the read-in,pandas.read_table(open_file, sep = separator)