NYCPlanning / db-safegraph

SafeGraph ETL
4 stars 1 forks source link

Python helper functions in recipes/_helper to standardize Athena processes #19

Closed SPTKL closed 3 years ago

SPTKL commented 3 years ago

Aws

class Aws:
     def __init__(self, region_name, ....): 
      """ 
      initialize class Aws
      """ 
      self.s3_client = XXXX
      self.athena_client = XXXX
      self.s3_resource = XXXX
      self.session = XXXXX

Aws.execute_query

def execute_query(self, query:str, database:str, output_location:str='s3://recovery-data-partnership/tmp/') -> str:
"""
Given query, database, output location, execute the query, 
and print out temporary output location
"""
return queryLoc

Aws.move_file

def move_file(self, source_location:str, target_location:str):
"""
given source_location, move file from source_location to
target_location and remove the original file at source_location
"""
SPTKL commented 3 years ago

resources: