Plan fails due to missing resources?

[aleenprd]

 logger.info("Defining infrastructure")
    role = Role(name="TITAN_TEST_ROLE")

    warehouse = Warehouse(
        name="TITAN_TEST_WAREHOUSE",
        warehouse_size="xsmall",
        auto_suspend=60,
    )

    database = Database(
        name="TITAN_TEST_DB",
        comment="This is a database used for testing purposes of titan[core].",
    )

    schema = Schema(
        name="TITAN_TEST_SCHEMA",
        comment="This is a schema used for testing purposes of titan[core].",
    )

    grants = [
        Grant(priv="usage", to=role, on=warehouse),
        Grant(priv="usage", to=role, on=database),
        Grant(priv="usage", to=role, on=schema),
        RoleGrant(role=role, to_role="SYSADMIN"),
    ]

    logger.info("Creating a new blueprint")
    bp = Blueprint(
        run_mode="create-or-update",
        resources=[role, warehouse, database, schema, *grants],
        dry_run=True, # apply() will return a list of SQL commands that would be executed without applying them.
    )

    logger.info("Planning the infrastructure")
    plan = bp.plan(connection)  # Returns list[ResourceChange]

    logger.info("Preview of plan")
    print_plan(plan)

    if input("Do you want to apply the plan? (y/n): ").lower() == "y":
        bp.apply(connection, plan)
        logger.info("Infrastructure applied successfully")
    else:
        logger.info("Infrastructure not applied")

Running this will error due to:

[redacted:ACCOUNTADMIN] > SHOW SCHEMAS LIKE 'TITAN_TEST_SCHEMA' IN DATABASE TITAN_TEST_DB    (err 2043, 0.11s)
[redacted:ACCOUNTADMIN] > SHOW GRANTS OF ROLE TITAN_TEST_ROLE    (1 rows, 0.09s)
{URN(urn::redacted:account/redacted): {'_pointer': True, 'name': Resource:redacted}, URN(urn::redacted:role/TITAN_TEST_ROLE): {'name': 'TITAN_TEST_ROLE', 'owner': 'USERADMIN', 'comment': None}, URN(urn::redacted:warehouse/TITAN_TEST_WAREHOUSE): {'name': 'TITAN_TEST_WAREHOUSE', 'owner': 'SYSADMIN', 'warehouse_type': 'STANDARD', 'warehouse_size': 'XSMALL', 'max_cluster_count': None, 'min_cluster_count': None, 'scaling_policy': None, 'auto_suspend': 60, 'auto_resume': True, 'initially_suspended': None, 'resource_monitor': None, 'comment': None, 'enable_query_acceleration': False, 'query_acceleration_max_scale_factor': None, 'max_concurrency_level': 8, 'statement_queued_timeout_in_seconds': 0, 'statement_timeout_in_seconds': 172800}, URN(urn::redacted:database/TITAN_TEST_DB): {'name': 'TITAN_TEST_DB', 'transient': False, 'owner': 'SYSADMIN', 'data_retention_time_in_days': 1, 'max_data_extension_time_in_days': 14, 'default_ddl_collation': None, 'comment': 'This is a database used for testing purposes of titan[core].'}, URN(urn::redacted:schema/TITAN_TEST_DB.TITAN_TEST_SCHEMA): {'name': 'TITAN_TEST_SCHEMA', 'transient': False, 'managed_access': False, 'data_retention_time_in_days': 1, 'max_data_extension_time_in_days': 14, 'default_ddl_collation': None, 'owner': 'SYSADMIN', 'comment': 'This is a schema used for testing purposes of titan[core].'}, URN(urn::redacted:grant/TITAN_TEST_ROLE?priv=USAGE&on=warehouse/TITAN_TEST_WAREHOUSE): {'priv': 'USAGE', 'on': 'TITAN_TEST_WAREHOUSE', 'on_type': 'WAREHOUSE', 'to': Resource:TITAN_TEST_ROLE, 'grant_option': False, 'owner': 'SYSADMIN', '_privs': ['USAGE']}, URN(urn::redacted:grant/TITAN_TEST_ROLE?priv=USAGE&on=database/TITAN_TEST_DB): {'priv': 'USAGE', 'on': 'TITAN_TEST_DB', 'on_type': 'DATABASE', 'to': Resource:TITAN_TEST_ROLE, 'grant_option': False, 'owner': 'SYSADMIN', '_privs': ['USAGE']}, URN(urn::redacted:grant/TITAN_TEST_ROLE?priv=USAGE&on=schema/TITAN_TEST_SCHEMA): {'priv': 'USAGE', 'on': 'TITAN_TEST_SCHEMA', 'on_type': 'SCHEMA', 'to': Resource:TITAN_TEST_ROLE, 'grant_option': False, 'owner': 'SYSADMIN', '_privs': ['USAGE']}, URN(urn::redacted:role_grant/TITAN_TEST_ROLE?role=SYSADMIN): {'role': Resource:TITAN_TEST_ROLE, 'to_role': 'SYSADMIN', 'to_user': None}}
Traceback (most recent call last):
  File "redacted/app/main.py", line 56, in <module>

  File "redacted/virtualenvs/titan-core-kmpMIK0i-py3.10/lib/python3.10/site-packages/titan/blueprint.py", line 779, in plan
    remote_state = self.fetch_remote_state(session, manifest)
  File "/Users/redacted/Library/Caches/pypoetry/virtualenvs/titan-core-kmpMIK0i-py3.10/lib/python3.10/site-packages/titan/blueprint.py", line 586, in fetch_remote_state
    raise MissingResourceException(
titan.blueprint.MissingResourceException: Resource urn::redacted:schema/TITAN_TEST_SCHEMA required by urn::redacted:grant/TITAN_TEST_ROLE?priv=USAGE&on=schema/TITAN_TEST_SCHEMA not found or failed to fetch

Should't Titan create all of these resources???

[teej]

This is an issue with the way Titan handles deferred name resolution. Titan assumes you intend for the schema TITAN_TEST_SCHEMA to live inside of the database TITAN_TEST_DB since you have one and only one database specified. Titan automatically connects those two resources together, but that breaks the grant.

You can work around this bug by explicitly connecting the schema to the database. There are a few ways but the most straightforward is this 1-line change:

schema = Schema(
    name="TITAN_TEST_SCHEMA",
    comment="This is a schema used for testing purposes of titan[core].",
    database=database,
)

[aleenprd]

I see. I believe this worked a couple of months before though, no? I only just picked it up again but updated to latest version.

I think it should also be best practice for me to be more specific in the declaration as well though.

Thanks for the quick reply!

[aleenprd]

@teej Hey Teej, perhaps you know what I am doing wrong here (same type of error).I structured my repo like this so I am using names to pass references between resources, not the objects themselves.

Error: titan.blueprint.MissingResourceException: Resource urn::redacted:role/TITAN_TEST_DB_ROLE required by urn::redacted:table/TITAN_TEST_DB.TITAN_TEST_SCHEMA.TITAN_TEST_TABLE not found or failed to fetch

I am declaring them like:

titan_test_db = Database(
    name="TITAN_TEST_DB",
    comment="This is a database used for testing of titan[core].",
    transient=True,
    owner="SYSADMIN",
    data_retention_time_in_days=None,
    max_data_extension_time_in_days=None,
    #tags={"group": "titan", "purpose": "testing"},
)

titan_test_schema = Schema(
    name="TITAN_TEST_SCHEMA",
    comment="This is a schema used for testing purposes of titan[core].",
    database = "TITAN_TEST_DB",
    transient=True,
    owner="SYSADMIN",
    data_retention_time_in_days=None,
    max_data_extension_time_in_days=None,
    #tags={"group": "titan", "purpose": "testing"},
)

titan_test_table = Table(
    name="TITAN_TEST_TABLE",
    columns=[
        {"name": "col1", "data_type": "STRING"},
        {"name": "col2", "data_type": "NUMBER"},
        {"name": "col3", "data_type": "TIMESTAMP"}
    ],
    owner="TITAN_TEST_DB_ROLE",   
    comment="This is a sample table used to test titan[core].",
    #tags={"group": "titan", "purpose": "test"},
    schema="TITAN_TEST_SCHEMA",
    database="TITAN_TEST_DB",
)

titan_test_db_role = DatabaseRole(
    name="TITAN_TEST_DB_ROLE",
    database="TITAN_TEST_DB",
    comment="This role is for database-specific access control used to test titan[core].",
    owner="USERADMIN",
    #tags={"group": "titan", "purpose": "testing"},
)    

role_to_role_grants = [
    RoleGrant(role="TITAN_TEST_DB_ROLE", to_role="SECURITYADMIN"),
    RoleGrant(role="TITAN_TEST_ROLE", to_role="SECURITYADMIN"),
]

role_to_user_grants = [
    RoleGrant(role="TITAN_TEST_DB_ROLE", to_user="TITAN_TEST_USER"),
    RoleGrant(role="TITAN_TEST_ROLE", to_user="TITAN_TEST_USER"),
]

  logger.info("Concatenating all resources...")
  # NB for Teej: these are imported from app/resources
    resources = (
        database_roles.all_database_roles
        + databases.all_databases
        + grants.all_grants
        + network_policies_and_rules.all_network_policies
        + network_policies_and_rules.all_network_rules
        + role_grants.all_role_grants
        + roles.all_roles
        + schemas.all_schemas
        + tables.all_tables
        + users.all_users
        + views.all_views
        + warehouses.all_warehouses
    )

    logger.info("Creating the blueprint...")
    bp = Blueprint(
        run_mode="create-or-update",
        dry_run=True,
        resources=resources
    )

    logger.info("Planning the infrastructure...")
    plan = bp.plan(connection)

Is it simply not possible to chain them like that, using just names? Or is something else off?

[teej]

I think this is an issue with database roles. I'll see if I can repro.