search

openx / ox-bqpipeline

python framework for authoring BigQuery Pipelines for scheduling
Apache License 2.0
1 stars 1 forks source link

Feature Request: improve export to gcs naming convention to include table name. #7

Open jaketf opened 5 years ago

jaketf commented 5 years ago

Currently the naming convention is:

gs://<prefix>/<jobName>/jobRunTime=<%Y%m%dT%H%M%s>"/<jobName>-*.csv

For exports of actual BigQuery Tables, we should change to:

gs://<prefix>/<jobName>/jobRunTime=<%Y%m%dT%H%M%s>"/<bqTableName>-*.csv

For direct export of query results to GCS, we should change to:

gs://<prefix>/<jobName>/jobRunTime=<%Y%m%dT%H%M%s>"/<sqlFileNameWithoutDotSql>-*.csv
jaketf commented 5 years ago

@freddyip I think this would be a good first issue for you to tackle. It's very straight forward / small change but it will get you used to maintaining ox-bqpipeline module.