peay
commented
1 year ago Currently, an instance is always created, and passed to:
def get_partition_tags(
repo_def: RepositoryDefinition,
partition_set_name: str,
partition_name: str,
instance: Optional[DagsterInstance] = None,
):
partition_set_def = repo_def.get_partition_set_def(partition_set_name)
partition = partition_set_def.get_partition(partition_name, dynamic_partitions_store=instance)
try:
with user_code_error_boundary(
PartitionExecutionError,
lambda: f"Error occurred during the evaluation of the `tags_for_partition` function for {_get_target_for_partition_execution_error(partition_set_def)}",
):
tags = partition_set_def.tags_for_partition(partition)
return ExternalPartitionTagsData(name=partition.name, tags=tags)
except Exception:
return ExternalPartitionExecutionErrorData(
serializable_error_info_from_exc_info(sys.exc_info())
)Maybe partition_set_def = repo_def.get_partition_set_def(partition_set_name) could be taken out of there, and checked to determine whether the partitions are dynamic? Only then would a DagsterInstance be created.
clairelin135
sryza
Dagster version
1.1.18
What's the issue?
We deploy a user code deployment as a container on kubernetes with the Helm chart. This container can't access the PostgreSQL database for Dagster for two reasons:
This was fine before since it didn't need any such access.
With 1.1.18, when we attempt to start a backfill, Dagster seems to invoke
ExternalPartitionTagson the gRPC server of the user code deployment, which fails when trying to create a Dagster instance as it cannot connect to the database, and prevents the backfill from starting.The user code deployment has the following exception chain when attempting to start a backfill from Dagit (here, with network policies lifted):
Single runs triggered from Dagit or schedules still work fine.
What did you expect to happen?
When dynamic partitions are not used, we should ideally not need the user code deployment to be able to connect to the Dagster instance.
The fact that the Dagster database connection string may be passed on to the user code deployment should be documented so that possible security implications can be taken into account.
How to reproduce?
Add a
NetworkPolicycovering a user code deployment that blocks egress trafficDeployment type
Dagster Helm chart
Deployment details
No response
Additional information
See https://dagster.slack.com/archives/C01U954MEER/p1676652897955709
cc @clairelin135
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.