dbt docs generate command fails when a custom macro is added. The same command works well without the custom macro. dbt compile command works fine even when the custom macro is added. Here is the detailed log and the issue.
03:37:52 - root - Running the docs generate command, command: /home/ec2-user/anaconda3/envs/python3/bin/dbt docs generate --profiles-dir ./
03:37:55 Running with dbt=1.2.0
03:37:55 Found 12 models, 0 tests, 0 snapshots, 0 analyses, 246 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics
03:38:42 Concurrency: 8 threads (target='notebook')
03:38:42
03:38:43 Done.
03:38:43 Building catalog
03:46:30 Encountered an error while generating catalog: Runtime Error
Runtime Error
GENERIC_INTERNAL_ERROR: java.lang.RuntimeException: java.lang.InterruptedException: sleep interrupted
03:46:30 dbt encountered 1 failure while writing the catalog
03:46:30 Catalog written to /home/ec2-user/SageMaker/test/raghu-folder/target/catalog.json
This is the catalog.json file:
{"metadata": {"dbt_schema_version": "https://schemas.getdbt.com/dbt/catalog/v1.json", "dbt_version": "1.2.0", "generated_at": "2022-09-02T03:46:30.948934Z", "invocation_id": "8c78b779-d267-46bc-9698-009a04824adc", "env": {}}, "nodes": {}, "sources": {}, "errors": ["Runtime Error\n Runtime Error\n GENERIC_INTERNAL_ERROR: java.lang.RuntimeException: java.lang.InterruptedException: sleep interrupted"]}
Environment:
- OS:Mac 12.5 and Amazon Linux 2 instance
- Python:3.7+
- dbt:1.1.1+
This is the custom macro I'm using:
{% macro generate_schema_name(custom_schema_name, node) -%}
{%- set default_schema = target.schema -%}
{%- if not custom_schema_name -%}
{{ default_schema }}
{%- else -%}
{{ custom_schema_name | trim }}
{%- endif -%}
{%- endmacro %}
Here is the detailed dbt.log doc for the same:
============================== 2022-09-13 03:24:11.505979 | 076edded-3158-45ad-8d24-5094e9056e95 ==============================
�[0m03:24:11.506010 [info ] [MainThread]: Running with dbt=1.2.0
�[0m03:24:11.507094 [debug] [MainThread]: running dbt with arguments {'write_json': True, 'use_colors': True, 'printer_width': 80, 'version_check': True, 'partial_parse': True, 'static_parser': True, 'profiles_dir': '/home/ec2-user/SageMaker/test/raghu-folder', 'send_anonymous_usage_stats': True, 'event_buffer_size': 100000, 'quiet': False, 'no_print': False, 'compile': True, 'which': 'generate', 'rpc_method': 'docs.generate', 'indirect_selection': 'eager'}
�[0m03:24:11.507296 [debug] [MainThread]: Tracking: tracking
�[0m03:24:11.537768 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7ff7039c6490>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7ff7039c6550>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7ff7039c68b0>]}
�[0m03:24:11.558151 [info ] [MainThread]: Unable to do partial parsing because a project config has changed
�[0m03:24:11.559242 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'partial_parser', 'label': '076edded-3158-45ad-8d24-5094e9056e95', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7ff703974f40>]}
�[0m03:24:11.572666 [debug] [MainThread]: Parsing macros/generate_schema_name.sql
�[0m03:24:11.574000 [debug] [MainThread]: Parsing macros/adapters/columns.sql
�[0m03:24:11.575668 [debug] [MainThread]: Parsing macros/adapters/freshness.sql
�[0m03:24:11.576122 [debug] [MainThread]: Parsing macros/adapters/metadata.sql
�[0m03:24:11.581870 [debug] [MainThread]: Parsing macros/adapters/relation.sql
�[0m03:24:11.583158 [debug] [MainThread]: Parsing macros/materializations/models/helpers.sql
�[0m03:24:11.584167 [debug] [MainThread]: Parsing macros/materializations/models/incremental/helpers.sql
�[0m03:24:11.591839 [debug] [MainThread]: Parsing macros/materializations/models/incremental/incremental.sql
�[0m03:24:11.601258 [debug] [MainThread]: Parsing macros/materializations/models/table/create_table_as.sql
�[0m03:24:11.605564 [debug] [MainThread]: Parsing macros/materializations/models/table/table.sql
�[0m03:24:11.608278 [debug] [MainThread]: Parsing macros/materializations/models/view/create_or_replace_view.sql
�[0m03:24:11.611780 [debug] [MainThread]: Parsing macros/materializations/models/view/create_view_as.sql
�[0m03:24:11.612435 [debug] [MainThread]: Parsing macros/materializations/models/view/view.sql
�[0m03:24:11.613498 [debug] [MainThread]: Parsing macros/materializations/seeds/helpers.sql
�[0m03:24:11.617228 [debug] [MainThread]: Parsing macros/materializations/snapshots/snapshot.sql
�[0m03:24:11.617921 [debug] [MainThread]: Parsing macros/adapters/apply_grants.sql
�[0m03:24:11.631123 [debug] [MainThread]: Parsing macros/adapters/columns.sql
�[0m03:24:11.640339 [debug] [MainThread]: Parsing macros/adapters/freshness.sql
�[0m03:24:11.643270 [debug] [MainThread]: Parsing macros/adapters/indexes.sql
�[0m03:24:11.645972 [debug] [MainThread]: Parsing macros/adapters/metadata.sql
�[0m03:24:11.653031 [debug] [MainThread]: Parsing macros/adapters/persist_docs.sql
�[0m03:24:11.657311 [debug] [MainThread]: Parsing macros/adapters/relation.sql
�[0m03:24:11.671380 [debug] [MainThread]: Parsing macros/adapters/schema.sql
�[0m03:24:11.673647 [debug] [MainThread]: Parsing macros/etc/datetime.sql
�[0m03:24:11.683765 [debug] [MainThread]: Parsing macros/etc/statement.sql
�[0m03:24:11.688086 [debug] [MainThread]: Parsing macros/generic_test_sql/accepted_values.sql
�[0m03:24:11.689540 [debug] [MainThread]: Parsing macros/generic_test_sql/not_null.sql
�[0m03:24:11.690405 [debug] [MainThread]: Parsing macros/generic_test_sql/relationships.sql
�[0m03:24:11.691274 [debug] [MainThread]: Parsing macros/generic_test_sql/unique.sql
�[0m03:24:11.692024 [debug] [MainThread]: Parsing macros/get_custom_name/get_custom_alias.sql
�[0m03:24:11.693501 [debug] [MainThread]: Parsing macros/get_custom_name/get_custom_database.sql
�[0m03:24:11.695047 [debug] [MainThread]: Parsing macros/get_custom_name/get_custom_schema.sql
�[0m03:24:11.697438 [debug] [MainThread]: Parsing macros/materializations/configs.sql
�[0m03:24:11.699688 [debug] [MainThread]: Parsing macros/materializations/hooks.sql
�[0m03:24:11.703563 [debug] [MainThread]: Parsing macros/materializations/models/incremental/column_helpers.sql
�[0m03:24:11.708690 [debug] [MainThread]: Parsing macros/materializations/models/incremental/incremental.sql
�[0m03:24:11.717505 [debug] [MainThread]: Parsing macros/materializations/models/incremental/is_incremental.sql
�[0m03:24:11.719092 [debug] [MainThread]: Parsing macros/materializations/models/incremental/merge.sql
�[0m03:24:11.734717 [debug] [MainThread]: Parsing macros/materializations/models/incremental/on_schema_change.sql
�[0m03:24:11.749702 [debug] [MainThread]: Parsing macros/materializations/models/table/create_table_as.sql
�[0m03:24:11.752636 [debug] [MainThread]: Parsing macros/materializations/models/table/table.sql
�[0m03:24:11.757819 [debug] [MainThread]: Parsing macros/materializations/models/view/create_or_replace_view.sql
�[0m03:24:11.761143 [debug] [MainThread]: Parsing macros/materializations/models/view/create_view_as.sql
�[0m03:24:11.763552 [debug] [MainThread]: Parsing macros/materializations/models/view/helpers.sql
�[0m03:24:11.764925 [debug] [MainThread]: Parsing macros/materializations/models/view/view.sql
�[0m03:24:11.770186 [debug] [MainThread]: Parsing macros/materializations/seeds/helpers.sql
�[0m03:24:11.787552 [debug] [MainThread]: Parsing macros/materializations/seeds/seed.sql
�[0m03:24:11.794279 [debug] [MainThread]: Parsing macros/materializations/snapshots/helpers.sql
�[0m03:24:11.806833 [debug] [MainThread]: Parsing macros/materializations/snapshots/snapshot.sql
�[0m03:24:11.818150 [debug] [MainThread]: Parsing macros/materializations/snapshots/snapshot_merge.sql
�[0m03:24:11.819910 [debug] [MainThread]: Parsing macros/materializations/snapshots/strategies.sql
�[0m03:24:11.835775 [debug] [MainThread]: Parsing macros/materializations/tests/helpers.sql
�[0m03:24:11.837602 [debug] [MainThread]: Parsing macros/materializations/tests/test.sql
�[0m03:24:11.841775 [debug] [MainThread]: Parsing macros/materializations/tests/where_subquery.sql
�[0m03:24:11.843564 [debug] [MainThread]: Parsing macros/utils/any_value.sql
�[0m03:24:11.844554 [debug] [MainThread]: Parsing macros/utils/bool_or.sql
�[0m03:24:11.845499 [debug] [MainThread]: Parsing macros/utils/cast_bool_to_text.sql
�[0m03:24:11.846553 [debug] [MainThread]: Parsing macros/utils/concat.sql
�[0m03:24:11.847532 [debug] [MainThread]: Parsing macros/utils/data_types.sql
�[0m03:24:11.853686 [debug] [MainThread]: Parsing macros/utils/date_trunc.sql
�[0m03:24:11.854780 [debug] [MainThread]: Parsing macros/utils/dateadd.sql
�[0m03:24:11.856000 [debug] [MainThread]: Parsing macros/utils/datediff.sql
�[0m03:24:11.857239 [debug] [MainThread]: Parsing macros/utils/escape_single_quotes.sql
�[0m03:24:11.858295 [debug] [MainThread]: Parsing macros/utils/except.sql
�[0m03:24:11.859135 [debug] [MainThread]: Parsing macros/utils/hash.sql
�[0m03:24:11.860214 [debug] [MainThread]: Parsing macros/utils/intersect.sql
�[0m03:24:11.861085 [debug] [MainThread]: Parsing macros/utils/last_day.sql
�[0m03:24:11.862909 [debug] [MainThread]: Parsing macros/utils/length.sql
�[0m03:24:11.863876 [debug] [MainThread]: Parsing macros/utils/listagg.sql
�[0m03:24:11.866281 [debug] [MainThread]: Parsing macros/utils/literal.sql
�[0m03:24:11.867226 [debug] [MainThread]: Parsing macros/utils/position.sql
�[0m03:24:11.868295 [debug] [MainThread]: Parsing macros/utils/replace.sql
�[0m03:24:11.869513 [debug] [MainThread]: Parsing macros/utils/right.sql
�[0m03:24:11.870603 [debug] [MainThread]: Parsing macros/utils/safe_cast.sql
�[0m03:24:11.871686 [debug] [MainThread]: Parsing macros/utils/split_part.sql
�[0m03:24:11.873699 [debug] [MainThread]: Parsing tests/generic/builtin.sql
�[0m03:24:12.090053 [debug] [MainThread]: 1603: static parser failed on incremental/gst/apc_hubspot_working_stage_dbt.sql
�[0m03:24:12.107656 [debug] [MainThread]: 1602: parser fallback to jinja rendering on incremental/gst/apc_hubspot_working_stage_dbt.sql
�[0m03:24:12.110046 [debug] [MainThread]: 1699: static parser successfully parsed precompute/gst/apc_hubspot_ent_marketing_leads_v6_dbt.sql
�[0m03:24:12.113665 [debug] [MainThread]: 1699: static parser successfully parsed precompute/gst/apc_hubspot_ent_marketing_leads_v4_dbt.sql
�[0m03:24:12.121032 [debug] [MainThread]: 1699: static parser successfully parsed precompute/gst/apc_hubspot_metrics_dbt.sql
�[0m03:24:12.129603 [debug] [MainThread]: 1699: static parser successfully parsed precompute/gst/apc_hubspot_cost_metrics_qs_dbt.sql
�[0m03:24:12.135158 [debug] [MainThread]: 1699: static parser successfully parsed precompute/gst/apc_hubspot_sql_targets_mtd_dbt.sql
�[0m03:24:12.140645 [debug] [MainThread]: 1699: static parser successfully parsed precompute/gst/apc_hubspot_marketing_quote_contribution_dbt.sql
�[0m03:24:12.143542 [debug] [MainThread]: 1699: static parser successfully parsed precompute/gst/apc_hubspot_lead_processing_funnel_dbt.sql
�[0m03:24:12.146344 [debug] [MainThread]: 1699: static parser successfully parsed precompute/gst/apc_hubspot_lead_targets_mtd_dbt.sql
�[0m03:24:12.148773 [debug] [MainThread]: 1699: static parser successfully parsed precompute/gst/apc_hubspot_marketing_contribution_dbt.sql
�[0m03:24:12.151629 [debug] [MainThread]: 1699: static parser successfully parsed precompute/gst/apc_hubspot_working_stage_qs_v2_dbt.sql
�[0m03:24:12.155487 [debug] [MainThread]: 1699: static parser successfully parsed precompute/gst/apc_hubspot_marketing_demo_contribution_dbt.sql
�[0m03:24:12.204863 [warn ] [MainThread]: [�[33mWARNING�[0m]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 10 unused configuration paths:
models.dbt_precompute.precompute.e_invoice
models.dbt_precompute.precompute.clearexpert
models.dbt_precompute.precompute.taxcloud
models.dbt_precompute.incremental.cleartax
models.dbt_precompute.precompute.clearapp_v2
models.dbt_precompute.precompute.cleartax
models.dbt_precompute.precompute.clearsave
models.dbt_precompute.precompute.tds
models.dbt_precompute.incremental.e_invoice
models.dbt_precompute.precompute.ewb
�[0m03:24:12.227801 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'load_project', 'label': '076edded-3158-45ad-8d24-5094e9056e95', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7ff7038f2b20>]}
�[0m03:24:12.237785 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'resource_counts', 'label': '076edded-3158-45ad-8d24-5094e9056e95', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7ff70399ff70>]}
�[0m03:24:12.238222 [info ] [MainThread]: Found 12 models, 0 tests, 0 snapshots, 0 analyses, 246 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics
�[0m03:24:12.239114 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'runnable_timing', 'label': '076edded-3158-45ad-8d24-5094e9056e95', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7ff70302f460>]}
�[0m03:24:12.241010 [info ] [MainThread]:
�[0m03:24:12.242208 [debug] [MainThread]: Acquiring new athena connection "master"
�[0m03:24:12.243724 [debug] [ThreadPool]: Acquiring new athena connection "list_awsdatacatalog_gst_prod"
�[0m03:24:12.254680 [debug] [ThreadPool]: Using athena connection "list_awsdatacatalog_gst_prod"
�[0m03:24:12.254982 [debug] [ThreadPool]: On list_awsdatacatalog_gst_prod: -- /* {"app": "dbt", "dbt_version": "1.2.0", "profile_name": "dbt_athena", "target_name": "notebook", "connection_name": "list_awsdatacatalog_gst_prod"} */
WITH views AS (
select
table_catalog as database,
table_name as name,
table_schema as schema
from "awsdatacatalog".INFORMATION_SCHEMA.views
where table_schema = LOWER('gst_prod')
), tables AS (
select
table_catalog as database,
table_name as name,
table_schema as schema
from "awsdatacatalog".INFORMATION_SCHEMA.tables
where table_schema = LOWER('gst_prod')
-- Views appear in both `tables` and `views`, so excluding them from tables
EXCEPT
select * from views
)
select views.*, 'view' AS table_type FROM views
UNION ALL
select tables.*, 'table' AS table_type FROM tables
�[0m03:24:12.255131 [debug] [ThreadPool]: Opening a new connection, currently in state init
�[0m03:24:50.339249 [debug] [ThreadPool]: SQL status: OK -1 in 38.08 seconds
�[0m03:24:50.605972 [debug] [ThreadPool]: On list_awsdatacatalog_gst_prod: Close
�[0m03:24:50.644508 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'runnable_timing', 'label': '076edded-3158-45ad-8d24-5094e9056e95', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7ff703902af0>]}
�[0m03:24:50.645498 [info ] [MainThread]: Concurrency: 8 threads (target='notebook')
�[0m03:24:50.645944 [info ] [MainThread]:
�[0m03:24:50.652511 [debug] [Thread-1 ]: Began running node model.dbt_precompute.apc_hubspot_metrics_dbt
�[0m03:24:50.653439 [debug] [Thread-1 ]: Acquiring new athena connection "model.dbt_precompute.apc_hubspot_metrics_dbt"
�[0m03:24:50.652856 [debug] [Thread-2 ]: Began running node model.dbt_precompute.apc_hubspot_working_stage_dbt
�[0m03:24:50.654447 [debug] [Thread-2 ]: Acquiring new athena connection "model.dbt_precompute.apc_hubspot_working_stage_dbt"
�[0m03:24:50.654727 [debug] [Thread-2 ]: Began compiling node model.dbt_precompute.apc_hubspot_working_stage_dbt
�[0m03:24:50.655171 [debug] [Thread-2 ]: Compiling model.dbt_precompute.apc_hubspot_working_stage_dbt
�[0m03:24:50.655496 [debug] [Thread-1 ]: Began compiling node model.dbt_precompute.apc_hubspot_metrics_dbt
�[0m03:24:50.667094 [debug] [Thread-1 ]: Compiling model.dbt_precompute.apc_hubspot_metrics_dbt
�[0m03:24:50.669573 [debug] [Thread-1 ]: Writing injected SQL for node "model.dbt_precompute.apc_hubspot_metrics_dbt"
�[0m03:24:50.696116 [debug] [Thread-1 ]: finished collecting timing info
�[0m03:24:50.700647 [debug] [Thread-2 ]: Writing injected SQL for node "model.dbt_precompute.apc_hubspot_working_stage_dbt"
�[0m03:24:50.700962 [debug] [Thread-1 ]: Began executing node model.dbt_precompute.apc_hubspot_metrics_dbt
�[0m03:24:50.701561 [debug] [Thread-1 ]: finished collecting timing info
�[0m03:24:50.702268 [debug] [Thread-1 ]: Finished running node model.dbt_precompute.apc_hubspot_metrics_dbt
�[0m03:24:50.702985 [debug] [Thread-2 ]: finished collecting timing info
�[0m03:24:50.703240 [debug] [Thread-4 ]: Began running node model.dbt_precompute.apc_hubspot_ent_marketing_leads_v4_dbt
�[0m03:24:50.704237 [debug] [Thread-4 ]: Acquiring new athena connection "model.dbt_precompute.apc_hubspot_ent_marketing_leads_v4_dbt"
�[0m03:24:50.703614 [debug] [Thread-2 ]: Began executing node model.dbt_precompute.apc_hubspot_working_stage_dbt
�[0m03:24:50.703415 [debug] [Thread-5 ]: Began running node model.dbt_precompute.apc_hubspot_ent_marketing_leads_v6_dbt
�[0m03:24:50.704582 [debug] [Thread-4 ]: Began compiling node model.dbt_precompute.apc_hubspot_ent_marketing_leads_v4_dbt
�[0m03:24:50.705576 [debug] [Thread-4 ]: Compiling model.dbt_precompute.apc_hubspot_ent_marketing_leads_v4_dbt
�[0m03:24:50.704854 [debug] [Thread-2 ]: finished collecting timing info
�[0m03:24:50.705383 [debug] [Thread-5 ]: Acquiring new athena connection "model.dbt_precompute.apc_hubspot_ent_marketing_leads_v6_dbt"
�[0m03:24:50.717381 [debug] [Thread-2 ]: Finished running node model.dbt_precompute.apc_hubspot_working_stage_dbt
�[0m03:24:50.716709 [debug] [Thread-4 ]: Writing injected SQL for node "model.dbt_precompute.apc_hubspot_ent_marketing_leads_v4_dbt"
�[0m03:24:50.717960 [debug] [Thread-5 ]: Began compiling node model.dbt_precompute.apc_hubspot_ent_marketing_leads_v6_dbt
�[0m03:24:50.718668 [debug] [Thread-7 ]: Began running node model.dbt_precompute.apc_hubspot_working_stage_qs_v2_dbt
�[0m03:24:50.718923 [debug] [Thread-5 ]: Compiling model.dbt_precompute.apc_hubspot_ent_marketing_leads_v6_dbt
�[0m03:24:50.719514 [debug] [Thread-7 ]: Acquiring new athena connection "model.dbt_precompute.apc_hubspot_working_stage_qs_v2_dbt"
�[0m03:24:50.726556 [debug] [Thread-7 ]: Began compiling node model.dbt_precompute.apc_hubspot_working_stage_qs_v2_dbt
�[0m03:24:50.729425 [debug] [Thread-5 ]: Writing injected SQL for node "model.dbt_precompute.apc_hubspot_ent_marketing_leads_v6_dbt"
�[0m03:24:50.729667 [debug] [Thread-7 ]: Compiling model.dbt_precompute.apc_hubspot_working_stage_qs_v2_dbt
�[0m03:24:50.733870 [debug] [Thread-7 ]: Writing injected SQL for node "model.dbt_precompute.apc_hubspot_working_stage_qs_v2_dbt"
�[0m03:24:50.734303 [debug] [Thread-4 ]: finished collecting timing info
�[0m03:24:50.734527 [debug] [Thread-4 ]: Began executing node model.dbt_precompute.apc_hubspot_ent_marketing_leads_v4_dbt
�[0m03:24:50.734885 [debug] [Thread-4 ]: finished collecting timing info
�[0m03:24:50.735129 [debug] [Thread-7 ]: finished collecting timing info
�[0m03:24:50.735390 [debug] [Thread-5 ]: finished collecting timing info
�[0m03:24:50.736033 [debug] [Thread-4 ]: Finished running node model.dbt_precompute.apc_hubspot_ent_marketing_leads_v4_dbt
�[0m03:24:50.736331 [debug] [Thread-7 ]: Began executing node model.dbt_precompute.apc_hubspot_working_stage_qs_v2_dbt
�[0m03:24:50.737513 [debug] [Thread-7 ]: finished collecting timing info
�[0m03:24:50.738091 [debug] [Thread-7 ]: Finished running node model.dbt_precompute.apc_hubspot_working_stage_qs_v2_dbt
�[0m03:24:50.736604 [debug] [Thread-5 ]: Began executing node model.dbt_precompute.apc_hubspot_ent_marketing_leads_v6_dbt
�[0m03:24:50.738312 [debug] [Thread-5 ]: finished collecting timing info
�[0m03:24:50.738801 [debug] [Thread-5 ]: Finished running node model.dbt_precompute.apc_hubspot_ent_marketing_leads_v6_dbt
�[0m03:24:50.739850 [debug] [Thread-3 ]: Began running node model.dbt_precompute.apc_hubspot_cost_metrics_qs_dbt
�[0m03:24:50.740209 [debug] [Thread-1 ]: Began running node model.dbt_precompute.apc_hubspot_lead_processing_funnel_dbt
�[0m03:24:50.740734 [debug] [Thread-3 ]: Acquiring new athena connection "model.dbt_precompute.apc_hubspot_cost_metrics_qs_dbt"
�[0m03:24:50.742681 [debug] [Thread-3 ]: Began compiling node model.dbt_precompute.apc_hubspot_cost_metrics_qs_dbt
�[0m03:24:50.741293 [debug] [Thread-2 ]: Began running node model.dbt_precompute.apc_hubspot_marketing_contribution_dbt
�[0m03:24:50.742218 [debug] [Thread-8 ]: Began running node model.dbt_precompute.apc_hubspot_marketing_demo_contribution_dbt
�[0m03:24:50.742028 [debug] [Thread-1 ]: Acquiring new athena connection "model.dbt_precompute.apc_hubspot_lead_processing_funnel_dbt"
�[0m03:24:50.742506 [debug] [Thread-4 ]: Began running node model.dbt_precompute.apc_hubspot_marketing_quote_contribution_dbt
�[0m03:24:50.741069 [debug] [Thread-6 ]: Began running node model.dbt_precompute.apc_hubspot_lead_targets_mtd_dbt
�[0m03:24:50.742954 [debug] [Thread-3 ]: Compiling model.dbt_precompute.apc_hubspot_cost_metrics_qs_dbt
�[0m03:24:50.743116 [debug] [Thread-7 ]: Began running node model.dbt_precompute.apc_hubspot_sql_targets_mtd_dbt
�[0m03:24:50.743632 [debug] [Thread-2 ]: Acquiring new athena connection "model.dbt_precompute.apc_hubspot_marketing_contribution_dbt"
�[0m03:24:50.745694 [debug] [Thread-8 ]: Acquiring new athena connection "model.dbt_precompute.apc_hubspot_marketing_demo_contribution_dbt"
�[0m03:24:50.745881 [debug] [Thread-1 ]: Began compiling node model.dbt_precompute.apc_hubspot_lead_processing_funnel_dbt
�[0m03:24:50.746907 [debug] [Thread-4 ]: Acquiring new athena connection "model.dbt_precompute.apc_hubspot_marketing_quote_contribution_dbt"
�[0m03:24:50.747551 [debug] [Thread-6 ]: Acquiring new athena connection "model.dbt_precompute.apc_hubspot_lead_targets_mtd_dbt"
�[0m03:24:50.761006 [debug] [Thread-6 ]: Began compiling node model.dbt_precompute.apc_hubspot_lead_targets_mtd_dbt
�[0m03:24:50.759469 [debug] [Thread-3 ]: Writing injected SQL for node "model.dbt_precompute.apc_hubspot_cost_metrics_qs_dbt"
�[0m03:24:50.761539 [debug] [Thread-3 ]: finished collecting timing info
�[0m03:24:50.760314 [debug] [Thread-8 ]: Began compiling node model.dbt_precompute.apc_hubspot_marketing_demo_contribution_dbt
�[0m03:24:50.761887 [debug] [Thread-8 ]: Compiling model.dbt_precompute.apc_hubspot_marketing_demo_contribution_dbt
�[0m03:24:50.760807 [debug] [Thread-4 ]: Began compiling node model.dbt_precompute.apc_hubspot_marketing_quote_contribution_dbt
�[0m03:24:50.761155 [debug] [Thread-6 ]: Compiling model.dbt_precompute.apc_hubspot_lead_targets_mtd_dbt
�[0m03:24:50.758426 [debug] [Thread-7 ]: Acquiring new athena connection "model.dbt_precompute.apc_hubspot_sql_targets_mtd_dbt"
�[0m03:24:50.773164 [debug] [Thread-7 ]: Began compiling node model.dbt_precompute.apc_hubspot_sql_targets_mtd_dbt
�[0m03:24:50.773377 [debug] [Thread-7 ]: Compiling model.dbt_precompute.apc_hubspot_sql_targets_mtd_dbt
�[0m03:24:50.760584 [debug] [Thread-1 ]: Compiling model.dbt_precompute.apc_hubspot_lead_processing_funnel_dbt
�[0m03:24:50.766472 [debug] [Thread-8 ]: Writing injected SQL for node "model.dbt_precompute.apc_hubspot_marketing_demo_contribution_dbt"
�[0m03:24:50.766836 [debug] [Thread-4 ]: Compiling model.dbt_precompute.apc_hubspot_marketing_quote_contribution_dbt
�[0m03:24:50.772886 [debug] [Thread-6 ]: Writing injected SQL for node "model.dbt_precompute.apc_hubspot_lead_targets_mtd_dbt"
�[0m03:24:50.759858 [debug] [Thread-2 ]: Began compiling node model.dbt_precompute.apc_hubspot_marketing_contribution_dbt
�[0m03:24:50.761737 [debug] [Thread-3 ]: Began executing node model.dbt_precompute.apc_hubspot_cost_metrics_qs_dbt
�[0m03:24:50.787857 [debug] [Thread-1 ]: Writing injected SQL for node "model.dbt_precompute.apc_hubspot_lead_processing_funnel_dbt"
�[0m03:24:50.788868 [debug] [Thread-7 ]: Writing injected SQL for node "model.dbt_precompute.apc_hubspot_sql_targets_mtd_dbt"
�[0m03:24:50.790753 [debug] [Thread-8 ]: finished collecting timing info
�[0m03:24:50.795275 [debug] [Thread-4 ]: Writing injected SQL for node "model.dbt_precompute.apc_hubspot_marketing_quote_contribution_dbt"
�[0m03:24:50.795780 [debug] [Thread-6 ]: finished collecting timing info
�[0m03:24:50.796030 [debug] [Thread-2 ]: Compiling model.dbt_precompute.apc_hubspot_marketing_contribution_dbt
�[0m03:24:50.798088 [debug] [Thread-3 ]: finished collecting timing info
�[0m03:24:50.799046 [debug] [Thread-8 ]: Began executing node model.dbt_precompute.apc_hubspot_marketing_demo_contribution_dbt
�[0m03:24:50.809535 [debug] [Thread-8 ]: finished collecting timing info
�[0m03:24:50.810615 [debug] [Thread-8 ]: Finished running node model.dbt_precompute.apc_hubspot_marketing_demo_contribution_dbt
�[0m03:24:50.800385 [debug] [Thread-6 ]: Began executing node model.dbt_precompute.apc_hubspot_lead_targets_mtd_dbt
�[0m03:24:50.811040 [debug] [Thread-6 ]: finished collecting timing info
�[0m03:24:50.811636 [debug] [Thread-6 ]: Finished running node model.dbt_precompute.apc_hubspot_lead_targets_mtd_dbt
�[0m03:24:50.799437 [debug] [Thread-7 ]: finished collecting timing info
�[0m03:24:50.812222 [debug] [Thread-7 ]: Began executing node model.dbt_precompute.apc_hubspot_sql_targets_mtd_dbt
�[0m03:24:50.800728 [debug] [Thread-4 ]: finished collecting timing info
�[0m03:24:50.812631 [debug] [Thread-4 ]: Began executing node model.dbt_precompute.apc_hubspot_marketing_quote_contribution_dbt
�[0m03:24:50.807126 [debug] [Thread-2 ]: Writing injected SQL for node "model.dbt_precompute.apc_hubspot_marketing_contribution_dbt"
�[0m03:24:50.799639 [debug] [Thread-1 ]: finished collecting timing info
�[0m03:24:50.813074 [debug] [Thread-1 ]: Began executing node model.dbt_precompute.apc_hubspot_lead_processing_funnel_dbt
�[0m03:24:50.809050 [debug] [Thread-3 ]: Finished running node model.dbt_precompute.apc_hubspot_cost_metrics_qs_dbt
�[0m03:24:50.812789 [debug] [Thread-4 ]: finished collecting timing info
�[0m03:24:50.812395 [debug] [Thread-7 ]: finished collecting timing info
�[0m03:24:50.813234 [debug] [Thread-1 ]: finished collecting timing info
�[0m03:24:50.813838 [debug] [Thread-4 ]: Finished running node model.dbt_precompute.apc_hubspot_marketing_quote_contribution_dbt
�[0m03:24:50.814635 [debug] [Thread-7 ]: Finished running node model.dbt_precompute.apc_hubspot_sql_targets_mtd_dbt
�[0m03:24:50.815487 [debug] [Thread-1 ]: Finished running node model.dbt_precompute.apc_hubspot_lead_processing_funnel_dbt
�[0m03:24:50.815886 [debug] [Thread-2 ]: finished collecting timing info
�[0m03:24:50.816674 [debug] [Thread-2 ]: Began executing node model.dbt_precompute.apc_hubspot_marketing_contribution_dbt
�[0m03:24:50.816831 [debug] [Thread-2 ]: finished collecting timing info
�[0m03:24:50.817278 [debug] [Thread-2 ]: Finished running node model.dbt_precompute.apc_hubspot_marketing_contribution_dbt
�[0m03:24:50.819074 [debug] [MainThread]: Connection 'master' was properly closed.
�[0m03:24:50.819240 [debug] [MainThread]: Connection 'model.dbt_precompute.apc_hubspot_lead_processing_funnel_dbt' was properly closed.
�[0m03:24:50.819359 [debug] [MainThread]: Connection 'model.dbt_precompute.apc_hubspot_marketing_contribution_dbt' was properly closed.
�[0m03:24:50.819469 [debug] [MainThread]: Connection 'model.dbt_precompute.apc_hubspot_marketing_quote_contribution_dbt' was properly closed.
�[0m03:24:50.819577 [debug] [MainThread]: Connection 'model.dbt_precompute.apc_hubspot_ent_marketing_leads_v6_dbt' was properly closed.
�[0m03:24:50.819684 [debug] [MainThread]: Connection 'model.dbt_precompute.apc_hubspot_sql_targets_mtd_dbt' was properly closed.
�[0m03:24:50.819790 [debug] [MainThread]: Connection 'model.dbt_precompute.apc_hubspot_cost_metrics_qs_dbt' was properly closed.
�[0m03:24:50.819892 [debug] [MainThread]: Connection 'model.dbt_precompute.apc_hubspot_marketing_demo_contribution_dbt' was properly closed.
�[0m03:24:50.819997 [debug] [MainThread]: Connection 'model.dbt_precompute.apc_hubspot_lead_targets_mtd_dbt' was properly closed.
�[0m03:24:50.829319 [info ] [MainThread]: Done.
�[0m03:24:50.832870 [debug] [MainThread]: Acquiring new athena connection "generate_catalog"
�[0m03:24:50.833106 [info ] [MainThread]: Building catalog
�[0m03:24:50.836563 [debug] [ThreadPool]: Acquiring new athena connection "awsdatacatalog.information_schema"
�[0m03:24:50.851046 [debug] [ThreadPool]: Using athena connection "awsdatacatalog.information_schema"
�[0m03:24:50.851387 [debug] [ThreadPool]: On awsdatacatalog.information_schema: -- /* {"app": "dbt", "dbt_version": "1.2.0", "profile_name": "dbt_athena", "target_name": "notebook", "connection_name": "awsdatacatalog.information_schema"} */
select * from (
(
with tables as (
select
tables.table_catalog as table_database,
tables.table_schema as table_schema,
tables.table_name as table_name,
case
when views.table_name is not null
then 'view'
when table_type = 'BASE TABLE'
then 'table'
else table_type
end as table_type,
null as table_owner,
null as table_comment
from "awsdatacatalog".INFORMATION_SCHEMA.tables
left join "awsdatacatalog".INFORMATION_SCHEMA.views
on tables.table_catalog = views.table_catalog
and tables.table_schema = views.table_schema
and tables.table_name = views.table_name
),
columns as (
select
table_catalog as table_database,
table_schema as table_schema,
table_name as table_name,
column_name as column_name,
ordinal_position as column_index,
data_type as column_type,
comment as column_comment
from "awsdatacatalog".INFORMATION_SCHEMA.columns
),
catalog as (
select
tables.table_database,
tables.table_schema,
tables.table_name,
tables.table_type,
tables.table_comment,
columns.column_name,
columns.column_index,
columns.column_type,
columns.column_comment,
tables.table_owner
from tables
join columns
on tables."table_database" = columns."table_database"
and tables."table_schema" = columns."table_schema"
and tables."table_name" = columns."table_name"
)select * from catalog where lower("table_schema") = lower('gst_prod'))
)
�[0m03:24:50.851571 [debug] [ThreadPool]: Opening a new connection, currently in state init
�[0m03:34:06.453748 [debug] [ThreadPool]: Athena adapter: Error running SQL: -- /* {"app": "dbt", "dbt_version": "1.2.0", "profile_name": "dbt_athena", "target_name": "notebook", "connection_name": "awsdatacatalog.information_schema"} */
select * from (
(
with tables as (
select
tables.table_catalog as table_database,
tables.table_schema as table_schema,
tables.table_name as table_name,
case
when views.table_name is not null
then 'view'
when table_type = 'BASE TABLE'
then 'table'
else table_type
end as table_type,
null as table_owner,
null as table_comment
from "awsdatacatalog".INFORMATION_SCHEMA.tables
left join "awsdatacatalog".INFORMATION_SCHEMA.views
on tables.table_catalog = views.table_catalog
and tables.table_schema = views.table_schema
and tables.table_name = views.table_name
),
columns as (
select
table_catalog as table_database,
table_schema as table_schema,
table_name as table_name,
column_name as column_name,
ordinal_position as column_index,
data_type as column_type,
comment as column_comment
from "awsdatacatalog".INFORMATION_SCHEMA.columns
),
catalog as (
select
tables.table_database,
tables.table_schema,
tables.table_name,
tables.table_type,
tables.table_comment,
columns.column_name,
columns.column_index,
columns.column_type,
columns.column_comment,
tables.table_owner
from tables
join columns
on tables."table_database" = columns."table_database"
and tables."table_schema" = columns."table_schema"
and tables."table_name" = columns."table_name"
)select * from catalog where lower("table_schema") = lower('gst_prod'))
)
�[0m03:34:06.454296 [debug] [ThreadPool]: Athena adapter: Error running SQL: macro get_catalog
�[0m03:34:06.454574 [debug] [ThreadPool]: On awsdatacatalog.information_schema: Close
�[0m03:34:06.455144 [warn ] [MainThread]: Encountered an error while generating catalog: Runtime Error
Runtime Error
GENERIC_INTERNAL_ERROR: java.lang.RuntimeException: java.lang.InterruptedException: sleep interrupted
�[0m03:34:06.479104 [error] [MainThread]: dbt encountered 1 failure while writing the catalog
�[0m03:34:06.481398 [info ] [MainThread]: Catalog written to /home/ec2-user/SageMaker/test/raghu-folder/target/catalog.json
�[0m03:34:06.481854 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7ff7039c6490>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7ff700e851c0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7ff700e85250>]}
�[0m03:34:07.319110 [debug] [MainThread]: Connection 'generate_catalog' was properly closed.
�[0m03:34:07.319681 [debug] [MainThread]: Connection 'awsdatacatalog.information_schema' was properly closed.
It looks like the java.lang.RuntimeError crops up when I'm trying to run this query.
select * from (
(
with tables as (
select
tables.table_catalog as table_database,
tables.table_schema as table_schema,
tables.table_name as table_name,
case
when views.table_name is not null
then 'view'
when table_type = 'BASE TABLE'
then 'table'
else table_type
end as table_type,
null as table_owner,
null as table_comment
from "awsdatacatalog".INFORMATION_SCHEMA.tables
left join "awsdatacatalog".INFORMATION_SCHEMA.views
on tables.table_catalog = views.table_catalog
and tables.table_schema = views.table_schema
and tables.table_name = views.table_name
),
columns as (
select
table_catalog as table_database,
table_schema as table_schema,
table_name as table_name,
column_name as column_name,
ordinal_position as column_index,
data_type as column_type,
comment as column_comment
from "awsdatacatalog".INFORMATION_SCHEMA.columns
),
catalog as (
select
tables.table_database,
tables.table_schema,
tables.table_name,
tables.table_type,
tables.table_comment,
columns.column_name,
columns.column_index,
columns.column_type,
columns.column_comment,
tables.table_owner
from tables
join columns
on tables."table_database" = columns."table_database"
and tables."table_schema" = columns."table_schema"
and tables."table_name" = columns."table_name"
)select * from catalog where lower("table_schema") = lower('gst_prod'))
)
raghub1
commented
1 year ago
dbt docs generatecommand fails when a custom macro is added. The same command works well without the custom macro.dbt compilecommand works fine even when the custom macro is added. Here is the detailed log and the issue.This is the catalog.json file:
{"metadata": {"dbt_schema_version": "https://schemas.getdbt.com/dbt/catalog/v1.json", "dbt_version": "1.2.0", "generated_at": "2022-09-02T03:46:30.948934Z", "invocation_id": "8c78b779-d267-46bc-9698-009a04824adc", "env": {}}, "nodes": {}, "sources": {}, "errors": ["Runtime Error\n Runtime Error\n GENERIC_INTERNAL_ERROR: java.lang.RuntimeException: java.lang.InterruptedException: sleep interrupted"]}Environment:
This is the custom macro I'm using:
Here is the detailed dbt.log doc for the same:
It looks like the java.lang.RuntimeError crops up when I'm trying to run this query.
Can anybody please help ?