feature : api

[seoyeong200]

spark-postgresql connection

• place executable postgre-sql connecto jar file to spark designated directory
• rebuild spark cluster base image
• add POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD env variable to .env.transform
• add connector configuration when creating spark session .config('spark.jars.packages', '/opt/spark/jars/postgresql-42.7.3.jar')\

  jdbcDF.select("*")\
  .write \
  .format("jdbc") \
  .option("url", "jdbc:postgresql://postgreDB:5432/bookDB") \
  .option("driver", "org.postgresql.Driver") \
  .option("dbtable", "bookDB") \
  .option("user", username) \
  .option("password", password) \
  .save() 

**note that hostname of jdbc connection url is postgreDB(not localhost), due to 'localhost' points to the container itself which execute spark application, not the postgresql server. it's due to the characteristic of docker compose, it automatically places all services on the same network.