search

USF-IMARS / imars-etl

:cloud: Tools for `extract` and `load` for IMaRS ETL (Extract, Transform, Load) operations
0 stars 0 forks source link

API load_dir w/o duplicates_ok does not raise #8

Closed 7yl4r closed 5 years ago

7yl4r commented 6 years ago

--duplicates_ok is the flag to tell load not to fail when a duplicate is detected, implying that it should raise an error when a duplicate is found and duplicates_ok==False. The airflow logs below indicate this is not the case. 

[2018-05-21 14:54:45,026] {logging_mixin.py:84} INFO - loading {'directory': '/srv/imars-objects/airflow_tmp/proc_wv2_unzip_na_20180201T000000_unzip_dir', 'product_type_name': 'eph_wv2_m1bs', 'verbose': 3, 'json': '{"status_id":3, "area_id":5}', 'load_format': '{dag_id}_%Y%m%dT%H%M%S_{tag}'}

[2018-05-21 14:54:45,222] {base_task_runner.py:98} INFO - Subtask: [2018-05-21 14:54:45,220] {load.py:108} INFO - 0 files loaded, 64 skipped, 0 duplicates.
[2018-05-21 14:54:45,223] {base_task_runner.py:98} INFO - Subtask: [2018-05-21 14:54:45,220] {python_operator.py:90} INFO - Done. Returned value was: None
7yl4r commented 5 years ago

--directory is dead and so is this issue :skull: