search

ccao-data / data-architecture

Codebase for CCAO data infrastructure construction and management
https://ccao-data.github.io/data-architecture/
5 stars 3 forks source link

Revert "Configure high availability on all dbt tables" #476

Closed dfsnow closed 1 month ago

dfsnow commented 1 month ago

Reverts ccao-data/data-architecture#462. We're getting the following failure on Spark tables:

File "<stdin>", line 387, in <module>
  File "<stdin>", line 239, in model
  File "<stdin>", line 384, in <lambda>
  File "<stdin>", line 292, in ref
  File "<stdin>", line 378, in get_spark_df
  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 700, in table
    return DataFrame(self._jsparkSession.table(tableName), self._sc, self._jconf)
  File "/opt/amazon/spark/python/lib/py4j-0.10.9.3-src.zip/py4j/java_gateway.py", line 1321, in __call__
    return_value = get_return_value(
  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 117, in deco
    raise converted from None
pyspark.sql.utils.AnalysisException: Path does not exist: s3://ccao-athena-ctas-us-east-1/reporting/ratio_stats_input/abb06ce1-9e51-40e6-bdc5-aaceb2c5e2cc