cdklabs / cdk-stacksets

Apache License 2.0
82 stars 13 forks source link

Unable to deploy Glue job assets #300

Open hurtadoca opened 9 months ago

hurtadoca commented 9 months ago

I am trying to create a stack set containing a Glue job, I have a conflict in how the script asset is handled by the Job construct and how the stackset assets are managed during the deployment.

When using the Job construct the code asset expects to receive a single file.

from aws_cdk import aws_glue_alpha as glue
from cdk_stacksets import StackSetStack

dirname = os.path.dirname(__file__)

class GlueJobStack(StackSetStack):

    def __init__(self, scope: Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

...

        glue_job = glue.Job(
            self,
            "GlueJob",
            job_name="glue_job",
            executable=glue.JobExecutable.python_etl(
                glue_version=glue.GlueVersion.V3_0,
                python_version=glue.PythonVersion.THREE,
                script=glue.Code.from_asset(
                    path=os.path.join(dirname,
                                      "glue_jobs/script.py"
                                      ),
                    deploy_time=True,
                )
            ),
            worker_type=glue.WorkerType.G_1_X,
            worker_count=10,
            role=glue_role
        )

When executing the cdk synth command I get the error: Asset path must be either a .zip file or a directory

If I try using a path to the glue_jobs folder instead I get the error: Code path ./glue_jobs/ is a directory. Only files are supported. This is because the glue.Code.from_asset method expects a single file, not a directory.

Another approach:

        job_asset = assets.Asset(
            self,
            "JobAsset",
            path=os.path.join(dirname,
                              "glue_jobs/script.py"
                              )
        )

...

I have tried using an Asset construct first but I get the same Asset path must be either a .zip file or a directory error if my asset refers to a single file. Again, if I reference the folder containing the python job script and in the job definition glue.Code. from_bucket instead, the synth and deploy process works but the job is created using the zip file (binary file) as the source so the code is illegible and can't be executed.

josh-demuth commented 8 months ago

thanks for reporting this. to confirm, this works as a regular stack deployment? i will test this out in a bit with the updates that were made in https://github.com/cdklabs/cdk-stacksets/pull/325.