toryas / serverless-glue

This is a plugin for Serverless framework that provide the possibility to deploy AWS Glue Jobs and Triggers
25 stars 30 forks source link

Triggers required since version 2.X.X? #25

Closed ThMrtns closed 2 years ago

ThMrtns commented 2 years ago

Hi! I updated do latest version of glue and some things of the config - like moving it out from custom-part in serverless.yml to the root - works for the jobs. But it seems, that defining triggers is required now. Would it be possible to make the trigger-section optional? We use special lambda functions to start the gluejobs. This would be awesome.

On the other hand, when defining placeholders as a workaroung like this:


  triggers:
    - name: trigger-placeholder-a
      actions:
        - name: gluejob-a
    - name: trigger-placeholder-b
      actions:
        - name: gluejob-b

I got following error during deploy: TypeError: Cannot read property 'split' of undefined at ServerlessService.

What am I doing wrong?

toryas commented 2 years ago

Hi @ThMrtns Triggers are optional, If it is being required as mandatory, let me know so I will correct it.

can you share your yml to find the error you mention

ThMrtns commented 2 years ago

Hi @toryas , thanks for your fast reply. It doesnt matter if I define any triggers or not, I got the above mentioned error: TypeError: Cannot read property 'split' of undefined at ServerlessService.

The Glue-Part of my severless.yml looks like below. (deploy without Glue works without problems. The variables from the .env also are succesfully used at other places in the yml)

Glue:
  bucketDeploy: ${env:S3_GLUEJOBS_BUCKET_NAME}-${self:provider.stage}
  s3Prefix: gluejobs/
  tempDirBucket: ${env:S3_GLUEJOBS_BUCKET_NAME}-${self:provider.stage}
  tempDirS3Prefix: tempdir/
  jobs:
    - name: ${env:GLUEJOB_A_NAME}-${self:provider.stage}
      script: src/gluejobs/gluejob_a.py
      tempDir: true
      type: spark
      glueVersion: python3-3.0
      role: ${env:ROLE_GLUEJOB_NAME}-${self:provider.stage}
      WorkerType: G.2X
      NumberOfWorkers: 10
      MaxConcurrentRuns: 10
      Timeout: 180

Best wishes and thx in advance!

toryas commented 2 years ago

you must change script to scriptPath

ThMrtns commented 2 years ago

Arrrgh, I compared it so often and still overlooked. Thanks for your eyes!