USF-IMARS / imars-etl

:cloud: Tools for `extract` and `load` for IMaRS ETL (Extract, Transform, Load) operations
0 stars 0 forks source link

imars_etl as hook(s) #31

Open 7yl4r opened 5 years ago

7yl4r commented 5 years ago

_moved here from USF-IMARS/imarsdags/issues/89

So imars_etl now uses hooks, but what if imars_etl was also used as a hook.

Sounds crazy and pointless, right? But hear me out: imars_etl wraps other hooks so that you can use multiple hooks as redundancy, fallbacks, etc but where is that configuration stored? Right now I think it is hard-coded. Why not store that configuration as a hook itself?

Benefits:

  1. pretty use of hooks in airflow
  2. offload config storage worries in imars_etl
  3. potentially useful in general

Drawbacks:

  1. more of my time spent optimizing rather than feature-izing
  2. ad-hoc hook-chaining would require creating an airflow connection? hook chains connections would really need be planned (& added) in advance.
7yl4r commented 5 years ago

This would replace the --object_store and (NYI) --meta_db parameters & that crufty "fallback chain" stuff in BaseHookHandler with something like --conn_id etl_connection_1.

imars_etl as a hook would be much more elegant.