Digdag scheduler can't use secret.

hiroyuki-sato commented 7 years ago

It seems that digdag scheduler can't use secret value. The server and local modes work fine.

Digdag: 0.9.12

File layout

.
|-- body.txt
`-- mail.dig

mail.dig

_export:
  mail:
    host: smtp.host
    from: from@mail.add.re.ss
    port: 587
    tls: true
    debug: true
    username: "user"

timezone: "Asia/Tokyo"

#schedule:
#  minutes_interval>: 1

+task1:
  mail>: body.txt
  to: ["dest@mail.add.re.ss"]
  subject: "digdag test"

body.txt

This is a test
This is a test
This is a test
This is a test
This is a test
This is a test
This is a test
This is a test

scheduler mode (NG)

/tmp/digdag.conf

digdag.secret-encryption-key = dGVzdDEyMzR0ZXN0MTIzNA==

digdag scheduler -m -c /tmp/digdag.conf
2017-06-27 22:11:20 +0900: Digdag v0.9.12
2017-06-27 22:11:21 +0900 [INFO] (main): secret encryption engine: aesgcm
2017-06-27 22:11:21 +0900 [INFO] (main): Added new revision 1
2017-06-27 22:11:21 +0900 [INFO] (main): XNIO version 3.3.6.Final
2017-06-27 22:11:21 +0900 [INFO] (main): XNIO NIO Implementation Version 3.3.6.Final
2017-06-27 22:11:21 +0900 [INFO] (main): Starting server on 127.0.0.1:65432
2017-06-27 22:11:22 +0900 [INFO] (main): Bound on 127.0.0.1:65432 (api)

Can't register secret value

digdag secret --project mail --set mail.password
2017-06-27 22:11:25 +0900: Digdag v0.9.12
error: Resource not found: project not found: mail

If I set secret with --local, I got the authentication error

digdag secret --local --set mail.password

2017-06-27 22:16:02 +0900 [ERROR] (0027@[0:default]+mail+task1): Task +mail+task1 failed.
535 5.7.0 Error: authentication failed: authentication failure
 (authentication failed)

local mode (OK)

digdag secret --local --set mail.password

digdag run -a mail

server mode (OK)

/tmp/digdag.conf

digdag.secret-encryption-key = dGVzdDEyMzR0ZXN0MTIzNA==

digdag server -c /tmp/digdag.conf

digdag push mail
2017-06-27 22:08:35 +0900: Digdag v0.9.12
Creating .digdag/tmp/archive-2205706131168390170.tar.gz...
  Archiving body.txt
  Archiving mail.dig
Workflows:
  mail.dig
Uploaded:
  id: 1
  name: mail
  revision: 76189a48-46a7-4385-81fb-20550f0c2702
  archive type: db
  project created at: 2017-06-27T13:08:37Z
  revision updated at: 2017-06-27T13:08:37Z

Use `digdag workflows` to show all workflows.

digdag secret --project mail --set mail.password
2017-06-27 22:09:05 +0900: Digdag v0.9.12
mail.password:
Secret 'mail.password' set

digdag start mail mail --session '2017-06-27'

saorio commented 7 years ago

I also couldn't set password. (scheduler mode)

$ digdag secrets --project xxx--set @secrets.json
2017-10-06 04:24:44 +0000: Digdag v0.9.7
error: RESTEASY004655: Unable to invoke request (processing)
> Connection refused (Connection refused) (connect)

hiroyuki-sato commented 7 years ago

Hello, @saorio Thank you for reporting this issue. I think it is better to use server mode in your case.

Vanuan commented 5 years ago

@hiroyuki-sato Would server mode run scheduled tasks?

Vanuan commented 5 years ago

Can't use secrets in server mode either:

digdag server -o data -b 0.0.0.0 &
digdag push scheduled-workflow
digdag secrets --project scheduled-workflow --set pg.password

2019-06-04 16:49:23 +0000 [ERROR] (XNIO-1 task-13): UT005023: Exception handling request to /api/projects/1/secrets/pg.password
org.jboss.resteasy.spi.UnhandledException: java.lang.UnsupportedOperationException
    at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:78)
    at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:222)
    at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:179)

Vanuan commented 5 years ago

Caused by: java.lang.UnsupportedOperationException: null
    at io.digdag.core.database.DisabledSecretCrypto.encryptSecret(DisabledSecretCrypto.java:11)
    at io.digdag.core.database.DatabaseSecretControlStore$LockedControl.setProjectSecret(DatabaseSecretControlStore.java:83)
    at io.digdag.core.database.DatabaseSecretControlStore.lambda$setProjectSecret$0(DatabaseSecretControlStore.java:46)
    at io.digdag.core.database.BasicDatabaseStoreManager.transaction(BasicDatabaseStoreManager.java:163)
    at io.digdag.core.database.DatabaseSecretControlStore.setProjectSecret(DatabaseSecretControlStore.java:45)
    at io.digdag.server.rs.ProjectResource.lambda$putProjectSecret$17(ProjectResource.java:728)
    at io.digdag.core.database.ThreadLocalTransactionManager.begin(ThreadLocalTransactionManager.java:251)

Vanuan commented 5 years ago

It looks like I needed to provide configuration file:

digdag server -o data -b 0.0.0.0 -c digdag.conf

treasure-data / digdag