search

datopian / ckanext-aircan

The custom extension for notifying(triggering) the Airflow DAG about the data to be uploaded to DataStore
GNU Affero General Public License v3.0
6 stars 5 forks source link

[dag_name param][s]: change dag_name param to dagname #17

Closed mbeilin closed 4 years ago

mbeilin commented 4 years ago

Change ckan.airflow.cloud.dag_name to ckan.airflow.cloud.dagname so it can be retrieved from .env correctly, otherwise the parameter is None (very weird...).

shubham-mahajan commented 4 years ago

@mbeilin As I can check with the method in ckanext-envvars (https://github.com/okfn/ckanext-envvars/blob/8602376a92a77b4f2d41763b6823d041469fa08e/ckanext/envvars/plugin.py#L13). This should not happen and it is very weird, but I will debug this in my free time and will check the root cause.

hannelita commented 4 years ago

@mbeilin So every var that contains a _ is causing problems? Or was it just for dag_name?

mbeilin commented 4 years ago

@mbeilin So every var that contains a _ is causing problems? Or was it just for dag_name?

@hannelita only for dag_name :(

hannelita commented 4 years ago

Weird! On #14 we start sending the dag_name via URL when we invoke aircan with aircan_submit, so we may also have resolved this issue in another format. Are you still using dagname?

mbeilin commented 4 years ago

Weird! On #14 we start sending the dag_name via URL when we invoke aircan with aircan_submit, so we may also have resolved this issue in another format. Are you still using dagname?

yes, here: https://github.com/datopian/ckanext-aircan/blob/fcf478e7e54c26aab4c90f95d0433d0bbab31383/ckanext/aircan_connector/gcp_handler.py#L61

hannelita commented 4 years ago

@mbeilin Hm, then I think we have some inconsistencies now. https://github.com/datopian/ckanext-aircan/blob/master/ckanext/aircan_connector/action.py#L119 I'm dynamically inputting this key on config, since I collect it from the URL now. If you do not pass anything on the URL (i.e. http://ckan:5000/api/3/action/aircan_submit?dag_name=ckan_api_load_gcp), it should be None. Are you using any other endpoint to trigger the DAG? If yes, we definitely must refactor the actions and how we collect the parameters for triggering the DAG