azkaban / azkaban-plugins

Plugins for Azkaban.
https://azkaban.github.io
Apache License 2.0
130 stars 178 forks source link

Run Sql JobType added #292

Open aycaacar opened 6 years ago

aycaacar commented 6 years ago

runsql jobtype

The job can be used for running sql statements against Vertica and Mysql.

Parameter Value Comment
db Parameter for the database (*) Vertica or Mysql (Default: Vertica)
type runsql
file name_of_sql_file
working.dir /path/to/sql/file Optional (Default: Current directory of '.job' file)
param.extra Parameter for sql statement Optional (One or more)

* db value will be used for getting related values from private.properties file. One must change db-param in the next section with this db value.)

Parameter Value Comment

| db.db-param.user | User of the database | | | db.db-param.pass | Pass of the user | | | db.db-param.host | Host address of the database | | | db.db-param.db | Name of the database | | | db.db-param.type | Type of the database | Vertica or Mysql | | db.db-param.backupservernode | Addresses of back up server nodes | Use only for Vertica (Comma separated) |

* Parameters other than jobtype.class can be defined multiple times with a different db-param value.

An example job and property files:

Test.job
---------
    db=test
    type=runsql
    file=test.sql
    working.dir=/home/test/
Test2.job
----------
    type=runsql
    file=test.sql
    working.dir=/home/vertica/test/
Test3.job
----------
    db=prod
    type=runsql
    file=prod.sql
    param.x=5
    param.y=text
private.properties
-------------------
    db.prod.user=admin
    db.prod.pass=admin
    db.prod.host=host1
    db.prod.db=vertica-prod
    db.prod.type=vertica
    db.prod.backupservernode=host2, host3, host4

    db.test.user=admin
    db.test.pass=admin
    db.test.host=host5
    db.test.db=mysql-test
    db.test.type=mysql

    db.vertica.user=admin
    db.vertica.pass=admin
    db.vertica.host=host5
    db.vertica.db=vertica-test
    db.vertica.type=vertica

How to Use Parameters ?

After defining _param.paramname parameter of a job, all substrings of related sql statement matching with _{{paramname}} will be replaced by the value of the parameter.