Error during msticnb initialization

AlexSta2019 commented 3 years ago

Environment details:

Linux (ubuntu 20.04 - Standard D2s v3 (2 vcpus, 8 GiB memory))
Python 3.8.10
Installed all pre-reqs for running the Azure Automated Notebooks from https://github.com/Azure/Azure-Sentinel-Notebooks. I also added/installed msticpy[ml] & gir1.2-secret-1.

Steps to repro: After I launch AutomatedNotebooks-Manager.ipynb using papermill, the AutomatedNotebooks-IncidentTriage.ipynb is triggered and at some point the msticnb should be initialized. The code looks like this:

# Set up notebooklets
nb.init(qry_prov)
timespan = TimeSpan(start=datetime.now() - timedelta(days=7))

This is where I can the following error:

~/.local/lib/python3.8/site-packages/msticpy/common/provider_settings.py in _get_settings(config_section, provider_name, conf_group, name_map)
    176     if not conf_group:
    177         return ProviderArgs()
--> 178     setting_dict: ProviderArgs = ProviderArgs(conf_group.copy())
    179 
    180     for arg_name, arg_value in conf_group.items():

AttributeError: 'str' object has no attribute 'copy'

NOTE: the error message is longer than this, please let me know if I should paste the entire message.

ianhelle commented 3 years ago

Hi @AlexSta2019, Thanks for reporting this. I'm not 100% certain but it looks like this could be caused by a problem with your msticpconfig.yaml The conf_group in the context of the code above is the Args section of a TI or Data provider entry. This should always be a dict and should look like the settings in the following. I.e. Args should have one or more keys (like AuthKey, AppID) and that, in turn can either be a string (for directly storing the value) or another dict for storing values in env vars or in Azure Key Vault.

TIProviders:
  OTX:
    Args:
      AuthKey: sdfg7f89g7fs0d97gfdag
    Primary: True
    Provider: "OTX"
  VirusTotal:
    Args:
      AuthKey:
        EnvironmentVar: "VIRUSTOTAL_AUTH"
    Primary: True
    Provider: "VirusTotal"
  AnOtherProvider:
    Args:
      AuthKey:
        KeyVault: 
    Primary: True
    Provider: "VirusTotal"
  ...

Can you check your config to make sure it looks OK. If this is the problem I will add an issue (in microsoft/msticpy repo) to add a check for this in the config validation, so that it's detectable earlier and with a bit more clarity.

AlexSta2019 commented 3 years ago

Thank you, Ian! This was it! Most likely between many/countless tests in different environments I ended up using the wrong msticpyconfig.yaml. I had an environment variable (inside the config file) which was not available in my current environment. So many details to be careful about 😊.

From: Ian Hellen @.> Sent: Friday, September 10, 2021 6:21 PM To: microsoft/msticnb @.> Cc: Alexandru Stamate @.>; Mention @.> Subject: Re: [microsoft/msticnb] Error during msticnb initialization (#21)

Hi @AlexSta2019https://github.com/AlexSta2019, Thanks for reporting this. I'm not 100% certain but it looks like this could be caused by a problem with your msticpconfig.yaml The conf_group in the context of the code above is the Args section of a TI or Data provider entry. This should always be a dict and should look like the settings in the following. I.e. Args should have one or more keys (like AuthKey, AppID) and that, in turn can either be a string (for directly storing the value) or another dict for storing values in env vars or in Azure Key Vault.

TIProviders:

OTX:

Args:

  AuthKey: sdfg7f89g7fs0d97gfdag

Primary: True

Provider: "OTX"

VirusTotal:

Args:

  AuthKey:

    EnvironmentVar: "VIRUSTOTAL_AUTH"

Primary: True

Provider: "VirusTotal"

AnOtherProvider:

Args:

  AuthKey:

    KeyVault:

Primary: True

Provider: "VirusTotal"

...

Can you check your config to make sure it looks OK. If this is the problem I will add an issue (in microsoft/msticpy repo) to add a check for this in the config validation, so that it's detectable earlier and with a bit more clarity.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/microsoft/msticnb/issues/21#issuecomment-917034177, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ATKZ2KHDAWBXG4KU3V7VZ7LUBIWAFANCNFSM5DXN27OQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

ianhelle commented 3 years ago

Hmm - the missing env variable should show up when you run init_notebook during the settings validation. I checked the validation code and it should show up as a config error. You can also run this check manually:

import msticpy
msticpy.settings.validate_config()

or on a specific config file

import msticpy
msticpy.settings.validate_config(config_file="./mymsticpyconfig.yaml")

Let me know if this doesn't work. Still slightly mystified about why an unset env var triggered this error though.

microsoft / msticnb

Error during msticnb initialization #21