search

Azure / batch-shipyard

Simplify HPC and Batch workloads on Azure
MIT License
277 stars 121 forks source link

Assistance with environment variables for credentials #305

Closed gcormier closed 5 years ago

gcormier commented 5 years ago

Similar to the other SHIPYARD_ environment variables, it would be great if we could have the credentials as environment variables.

SHIPYARD_BATCH_ACCOUNT_KEY SHIPYARD_BATCH_ACCOUNT_URL SHIPYARD_BATCH_STORAGE_ACCOUNT SHIPYARD_BATCH_STORAGE_KEY SHIPYARD_BATCH_STORAGE_ENDPOINT

alfpark commented 5 years ago

Can you expand on your specific scenario on why you require these credentials?

gcormier commented 5 years ago

These would be as an alternative to the credentials that sit inside config.yaml.

alfpark commented 5 years ago

You can use the SHIPYARD_AAD_* for this purpose. Or alternatively, store your credential.yaml file (or individual fields) in keyvault: https://github.com/Azure/batch-shipyard/blob/master/docs/74-batch-shipyard-azure-keyvault.md.

gcormier commented 5 years ago

Wow, I totally glazed over the SHIPYARD_AAD_* !

gcormier commented 5 years ago

Sorry, still having problems finding which config value corresponds to which environment variable.

credentials: batch: account_key: ??? account_service_url: ??? storage: mystorageaccount: account: ??? account_key: ??? endpoint: SHIPYARD_AAD_ENDPOINT

alfpark commented 5 years ago

With AAD, you don't need to populate any secrets, please see: https://github.com/Azure/batch-shipyard/blob/master/docs/11-batch-shipyard-configuration-credentials.md, specifically the aad section. Please read the entire doc however, as there are certain properties that are required when using AAD (e.g., resource_group under storage and management).

Additionally, this section is helpful (even without using keyvault): https://github.com/Azure/batch-shipyard/blob/master/docs/74-batch-shipyard-azure-keyvault.md#authenticating-with-aad-and-azure-keyvault

gcormier commented 5 years ago

Okay, I've gone through it and done a bit of jumping around, and I'm getting an error.

Error : ValueError: You need to provide an account name and either an account_key or sas_token when creating a storage service.

Docs state:

(required unless aad or account_key_keyvault_secret_id is specified) account_key is the storage account key

My credentials.yaml:

credentials:
  batch:
    resource_group: batch-shipyard
    account_service_url: https://abcbatch.eastus.batch.azure.com
  storage:
    mystorageaccount:
      resource_group: batch-shipyard
      account: xyz

Environment (obfuscated)

export SHIPYARD_AAD_DIRECTORY_ID=123..
export SHIPYARD_AAD_AUTH_KEY=456..
export SHIPYARD_AAD_APPLICATION_ID=789..

Since I have the env defined, that should be the global AAD section, in which case I should not need to specify account_key in the yaml file.

alfpark commented 5 years ago

Looks like this is a case of bad errors and missing documentation, apologies.

You need to also populate:

credentials:
  # other stuff
  management:
    subscription_id: 123...
gcormier commented 5 years ago

That worked, I'm up and running! Thanks very much for the assistance!

I doubt a subscription ID is anything sensitive, but it probably couldn't hurt to move that to an environment variable as well?

alfpark commented 5 years ago

Apparently this is already supported, but I forgot to doc it. Use the environment variable SHIPYARD_SUBSCRIPTION_ID.

Edit: It's actually documented here: https://github.com/Azure/batch-shipyard/blob/master/docs/20-batch-shipyard-usage.md#shared-options