search

dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.71k stars 1.48k forks source link

[docs] - K8s AWS auth and roles #5232

Open dagsterbot[bot] opened 3 years ago

dagsterbot[bot] commented 3 years ago

Dagster Documentation Gap

K8s AWS auth and roles

Conversation excerpt

Full Slack conversation

Conversation U02J848FL9F: Folks, the helm chart renders the runLauncher ``` run_launcher: module: dagster_celery_k8s class: CeleryK8sRunLauncher config: dagster_home: env: DAGSTER_HOME instance_config_map: env: DAGSTER_K8S_INSTANCE_CONFIG_MAP postgres_password_secret: env: DAGSTER_K8S_PG_PASSWORD_SECRET broker: "" backend: "rpc://" ``` So it sets up the celery worker container ... and it has the DAGSTER_K8S_PG_PASSWORD_SECRET and the DAGSTER_K8S_INSTANCE_CONFIG_MAP ... When we launch a pipeline the run container complains about both ``` Error 2: Post processing at path root:postgres_password_secret of original value {'env': 'DAGSTER_K8S_PG_PASSWORD_SECRET'} failed: dagster.config.errors.PostProcessingError: You have attempted to fetch the environment variable "DAGSTER_K8S_PG_PASSWORD_SECRET" which is not set. In order for this execution to succeed it must be set in this environment. ``` U02J848FL9F: This is really confusing , we are not setting up the environment of the run containers but we do not even know where to do that... U02J848FL9F: And we do not see where in the helm chart this has to be set up U02J848FL9F: U015C9U9RLK: Hi <@U02J848FL9F> this is an odd bit of config that should certainly be improved. Could you look at the result of `kubectl get configmaps` it will have a result `<name>-pipeline-env`, defaulting to `dagster-pipeline-env` but its overridable. Then use that configmap in your pipeline run config (entered either in code via a PresetDefinition, or in the Dagit playground) ``` execution: celery-k8s: config: env_config_maps: - "<NAME>-pipeline-env" ``` This configmap contains the env vars that your container is missing. U02J848FL9F: This did work and thank you . I am trying to make a mental model here of how this works and this will help solving the next issue. The steps fail and it seems it is not able to push logs ( at the end of the step ) to S3. I have set up ``` compute_logs: module: dagster_aws.s3.compute_log_manager class: S3ComputeLogManager config: bucket: "xxxxxx-dev-null" prefix: "dagster-test-" ``` And it fails with S3 credentials issue `botocore.exceptions.NoCredentialsError: Unable to locate credentials` I though want the run container to execute via an IAM role that allows for writes to the said S3 bucket. U02J848FL9F: Do I now set up the role through the run-config ( configuring the executors is the right lingo I think ) too and if yes how ? Something like ``` annotations = { "" = <aws_iam_role.dagster_poc.name> ``` I do not see any complete example of setting up execute containers with the right configurations, maps, annotations etc ... That said the celery workers ( that presumably launch these containers ) have been set up with the required annotation that allows for S3 access to the said compute log bucket and should arguably be propagating their set up to the containers they launch. ``` Annotations: : dagster-poc-yyyyyyyyyyyy ``` In fact we need this role to be available to all the run steps ( access to different resources of our stack ) and thus to all pods executed from a celery worker. This issues is not restricted to just the compute logs I would assume. U02J848FL9F: yep, similar issue when io manager is set to `s3_pickle_io_manager` ``` botocore.exceptions.NoCredentialsError: Unable to locate credentials File "/usr/local/lib/python3.8/site-packages/dagster/core/errors.py", line 184, in user_code_error_boundary yield File "/usr/local/lib/python3.8/site-packages/dagster/core/execution/resources_init.py", line 289, in single_resource_event_generator resource_def.resource_fn(context) File "/usr/local/lib/python3.8/site-packages/dagster_aws/s3/io_manager.py", line 114, in s3_pickle_io_manager pickled_io_manager = PickledObjectS3IOManager(s3_bucket, s3_session, s3_prefix=s3_prefix) File "/usr/local/lib/python3.8/site-packages/dagster_aws/s3/io_manager.py", line 17, in __init__ self.s3.head_bucket(Bucket=self.bucket) ``` Want this pods to launch under an IAM role that allows access to configured buckets.... U02J848FL9F: So I tried this set up ``` @solid( tags = { 'dagster-k8s/config': { 'container_config': { 'resources': { 'requests': { 'cpu': '250m', 'memory': '64Mi' }, 'limits': { 'cpu': '500m', 'memory': '2560Mi' }, }, }, 'pod_template_spec_metadata': { 'annotations': { "": "dagster-poc-20211014204833791300000001"} }, }, }, ) def not_much(): return ``` And it did get the annotation on the run pod ``` Annotations: : dagster-poc-20211014204833791300000001 : eks.privileged ``` It still complains about missing `botocore.exceptions.NoCredentialsError: Unable to locate credentials` Stack Trace: ``` File "/usr/local/lib/python3.8/site-packages/dagster/core/errors.py", line 184, in user_code_error_boundary yield File "/usr/local/lib/python3.8/site-packages/dagster/core/execution/resources_init.py", line 289, in single_resource_event_generator resource_def.resource_fn(context) File "/usr/local/lib/python3.8/site-packages/dagster_aws/s3/io_manager.py", line 114, in s3_pickle_io_manager pickled_io_manager = PickledObjectS3IOManager(s3_bucket, s3_session, s3_prefix=s3_prefix) File "/usr/local/lib/python3.8/site-packages/dagster_aws/s3/io_manager.py", line 17, in __init__ ``` U02J848FL9F: That actually makes sense .. boto3 ``` If you are running on Amazon EC2 and no credentials have been found by any of the providers above, Boto3 will try to load credentials from the instance metadata service. In order to take advantage of this feature, you must have specified an IAM role to use when you launched your EC2 instance. ``` I think I am missing where to specify that role within dagster set up..... U015C9U9RLK: One option is to create a secret in your cluster with the `AWS_ACCESS_KEY_ID` etc. variables, then use `env_secrets` in run launcher config (or executor config, if it differs per run). ``` env_secrets (Optional[List[str]]): A list of custom Secret names from which to draw environment variables (using ``envFrom``) for the Job. Default: ``[]``. ``` See: I’m assuming you’re on EKS? If so there are a few other options. In our clusters, we use iam roles for service accounts U015C9U9RLK: <@U018K0G2Y85> docs K8s AWS auth and roles

Message from the maintainers:

Are you looking for the same documentation content? Give it a :thumbsup:. We factor engagement into prioritization.

erinkcochran87 commented 2 years ago

@johannkm D'you know if this is still relevant?

johannkm commented 2 years ago

@erinkcochran87 Not the most pressing but yeah