KeyError in settings.py

[markpearl]

Problem Description

My credentials.yaml seems to be facing issues when trying to run the following shipyard command: SHIPYARD_CONFIGDIR=config ./shipyard pool add

{
    "credentials": {
      "docker_registry": {
        "genomicsazuredocker.azurecr.io": {
          "username": "agcancregistry", 
          "password": "=="
        }
      }, 
      "storage": {
        "mystorageaccount": {
          "account": "agcanrnaseqdiag", 
          "endpoint": "core.windows.net", 
          "account_key": "==", 
          "resource_group": "agcan-rnaseq"
        }
      }, 
      "batch": {
        "account_service_url": "https://agcanbatch.canadaeast.batch.azure.com", 
        "account_key": "==", 
        "resource_group": "agcan-rnaseq"
    }
  }
}

After running the command, I receive the KeyError issues and the following stacktrace:

Traceback (most recent call last):
  File "/home/mjpearl/batch-shipyard/convoy/settings.py", line 1854, in credentials_storage
    conf = config['credentials']['storage'][ssel]
  File "/home/mjpearl/batch-shipyard/.shipyard/lib/python3.6/site-packages/ruamel/yaml/comments.py", line 747, in __getitem__
    return ordereddict.__getitem__(self, key)
KeyError: 'agcanrnaseqdiag'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mjpearl/batch-shipyard/shipyard.py", line 3130, in <module>
    cli()
  File "/home/mjpearl/batch-shipyard/.shipyard/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/mjpearl/batch-shipyard/.shipyard/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/mjpearl/batch-shipyard/.shipyard/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/mjpearl/batch-shipyard/.shipyard/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/mjpearl/batch-shipyard/.shipyard/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/mjpearl/batch-shipyard/.shipyard/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/mjpearl/batch-shipyard/.shipyard/lib/python3.6/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/mjpearl/batch-shipyard/.shipyard/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/mjpearl/batch-shipyard/shipyard.py", line 1536, in pool_add
    ctx.initialize_for_batch()
  File "/home/mjpearl/batch-shipyard/shipyard.py", line 302, in initialize_for_batch
    fs_storage=False)
  File "/home/mjpearl/batch-shipyard/shipyard.py", line 566, in _init_config
    convoy.fleet.populate_global_settings(self.config, fs_storage)
  File "/home/mjpearl/batch-shipyard/convoy/fleet.py", line 366, in populate_global_settings
    sc = settings.credentials_storage(config, bs.storage_account_settings)
  File "/home/mjpearl/batch-shipyard/convoy/settings.py", line 1859, in credentials_storage
    'exists.').format(ssel))
ValueError: Could not find storage account alias agcanrnaseqdiag in credentials:storage configuration. Please ensure the storage account alias exists.

My yaml file seems correct when referencing it to the yaml template as per the batchyard installation. Any idea what could be causing this issue?

[alfpark]

The storage account is aliased by a link:

{
      "storage": {
        "mystorageaccount": {
          "account": "agcanrnaseqdiag", 
          "endpoint": "core.windows.net", 
          "account_key": "==", 
          "resource_group": "agcan-rnaseq"
        }
      }
}

mystorageaccount is the alias you need to refer to, not agcanrnaseqdiag. If you wish to refer to the storage account name, then change mystorageaccount to whatever is appropriate.

[markpearl]

@alfpark I'm actually using yaml files in the config directory.

Are you saying that the yaml file needs to look like the following?

storage: agcanrnaseqdiag: account: agcanrnaseqdiag account_key: == endpoint: core.windows.net resource_group: agcan-rnaseq

[alfpark]

It depends on how you're referring to it in config.yaml. If you are using agcanrnaseqdiag, then yes, you'll need to modify the accompanying storage config.

[alfpark]

Closing, please re-open if you feel like the issue has not been resolved.

[markpearl]

Hi Fred,

I'm currently running snakemake using Shipyard and I seem to be having permission issues. It's an Ubuntu 18.04 machine and I've gone by the example in github discussed here:

https://github.com/Azure/azure-hpc/tree/master/LifeSciences/SnakemakeBurst

As per my snakemake file, it seems to be having permission issues when trying to reference the inputs defined in the rules at the beginning of the file.

[image: image.png]

I feel as though it's a permission issue, but there's not other way to install snakemake in Ubuntu without "sudo apt-get install snakemake".

Do you have any suggestions for this? I'm really struggling with this.

I've attached my snakemake file and all azure-batch configs.

Would really appreciate your guidance on this.

-Mark

On Thu, Mar 7, 2019 at 10:21 AM Fred Park notifications@github.com wrote:

Closing, please re-open if you feel like the issue has not been resolved.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Azure/batch-shipyard/issues/263#issuecomment-470567707, or mute the thread https://github.com/notifications/unsubscribe-auth/AqWy_9PfeJAaGnORQzVTt_OYcBkDix6Jks5vUS5ogaJpZM4bblcv .