Python Library Path

[politoedo]

Hi, thanks a lot for the plugin. I was wondering if it was already possible to specify the s3 uri of wheel python libraries when the python job needs these libraries in sls template.

[toryas]

today is not possible specify python libraries. I can add this feature soon. could you test this functionality when it is ready

[politoedo]

Sure! It would be nice (although I do not know if possible) if the upload of libraries (by specifing path) would be handled by the deploy itself.

[toryas]

I implemented default arguments in version 2.1.0, in this arguments you can define a path for py lib in the argument extraPyFiles. This will help you with that for now, I will continue working on a feature to load the libraries with the plugin

[politoedo]

Wonderful! I'll test tomorrow

---

From: toryas @.> Sent: Wednesday, December 15, 2021 9:35:59 PM To: toryas/serverless-glue @.> Cc: Edoardo Polito @.>; Author @.> Subject: Re: [toryas/serverless-glue] Python Library Path (Issue #12)

I implemented default arguments in version 2.1.0, in this arguments you can define a path for py lib in the argument extraPyFiles. This will help you with that for now, I will continue working on a feature to load the libraries with the plugin

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/toryas/serverless-glue/issues/12#issuecomment-995193722, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ARCY7XOLDNAA2VGOANMDMN3URD327ANCNFSM5JY66JUA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[politoedo]

My first test was having some troubles around connection parameter. As it is marked optional, I just deleted it (also because I skip it on glue console configuration panel) Though, the deploy wanted me to pass something nonetheless:

An error occurred: {} - Connection should not be null or empty in Connections.

So I had to pass

      Connections: 
        - 'None'

After that, everything went smooth and deploy was successful.

[politoedo]

Update: actually the job won't run at all.

Failed to execute with exception Could not find connection for the given criteria Failed to get catalog connections given names: None (Service: AWSGlueJobExecutor; Status Code: 400; Error Code: InvalidInputException;

I usually leave connections blank when setting up my GlueJob from web console. I believe this error:

An error occurred: {} - Connection should not be null or empty in Connections.

has forced me to specify a connection, and that connection of course does not exist.

[toryas]

Can you share your serverless.yml?

[politoedo]

service: google-ingestion

frameworkVersion: '2'

provider:
  name: aws
  region: eu-west-1
  stage: ${opt:stage}

custom:
  env: ${file(./env/${self:provider.stage}.yaml)}

Glue:
  bucketDeploy: ${self:custom.env.code_bucket}  # Required
  s3Prefix: ${self:custom.env.code_prefix}/ # optional, default = 'glueJobs/'
  tempDirBucket: ${self:custom.env.code_bucket} # optional, default = '{serverless.serviceName}-{provider.stage}-gluejobstemp'
  tempDirS3Prefix: ${self:custom.env.code_prefix}/temp # optional, default = ''. The job name will be appended to the prefix name, leading '/'

  jobs:
    - name: ${self:custom.env.job_name}_${self:provider.stage} # Required
      scriptPath: ${self:custom.env.script_path} # Required script will be named with the name after '/' and uploaded to s3Prefix location
      tempDir: true # Optional true | false
      type: pythonshell # spark / pythonshell # Required
      glueVersion: python3-0.9 # Required python3-1.0 | python3-2.0 | python2-1.0 | python2-0.9 | scala2-1.0 | scala2-0.9 | scala2-2.0
      role: ${self:custom.env.role_arn} # Required
      MaxConcurrentRuns: 1 # Optional
      Connections: 
        - None
      DefaultArguments: # Optional
        class: GlueApp  # Optional
        extraPyFiles: ${self:custom.env.wr},${self:custom.env.google_api},${self:custom.env.google_auth_http},${self:custom.env.google_auth_oauth},${self:custom.env.oauth2},${self:custom.env.pandas}

plugins:
    - serverless-glue

[toryas]

I cant found the error, could you talk me on discord? toryas#9550

[politoedo]

sent you a friend request

---

From: toryas @.> Sent: Thursday, December 16, 2021 6:28:49 PM To: toryas/serverless-glue @.> Cc: Edoardo Polito @.>; Author @.> Subject: Re: [toryas/serverless-glue] Python Library Path (Issue #12)

I cant found the error, could you talk me on discord? toryas#9550

— Reply to this email directly, view it on GitHubhttps://github.com/toryas/serverless-glue/issues/12#issuecomment-996027642, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ARCY7XMJG3TYLCULNYC6IPLURIOVDANCNFSM5JY66JUA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you authored the thread.Message ID: @.***>

[toryas]

issue covered with version 2.1.1