search

USF-IMARS / imars-etl

:cloud: Tools for `extract` and `load` for IMaRS ETL (Extract, Transform, Load) operations
0 stars 0 forks source link

replace usage of find regex in imars-dags #5

Closed 7yl4r closed 6 years ago

7yl4r commented 6 years ago

imars-dags currently uses a find regex string along with xargs to load files from a directory into imars-objects via a call to imars-etl. Instead, imars-dags could call imars-etl load with the directory path, and imars-etl could find the file(s) using "path_format".

This could be done using the find_regex strings in imars_etl.filename.data and the re module, but it would be even better to (re)use path_format by utilizing features from #3, so maybe that should come first.

7yl4r commented 6 years ago

potential issue: this will attempt to read metadata from the filepaths, but all other metadata would be identical for all files loaded.

This is okay, but seems like it could lead to a lot of accidental false data getting entered.