resource enumeration : NoRegisteredProviderFound for hardcoded API version '2018-02-14'

[59e5aaf4]

Hi,

There seem to be a hardcoded version issue in sscollector.pyz. I tried to patch the specific .shiv packaged file in my ~/.shiv/ with a brand new version but encountered import errors related to that version.

• sscollector cli - error, only fetches an equivalent of az resource list
• sscollector cli --azure, same as no parameter, raises the error below and only fecthes an equivalent of az resource list
• sscollector cli --aad - works, no error, now we're speaking, it works, 400MB and counting in the output folder

azure.core.exceptions.HttpResponseError: (NoRegisteredProviderFound) No registered resource provider found for location 'canadacentral' and API version '2018-02-14' for type 'workbooks'. The supported api-versions are '2018-06-01-preview, 2018-06-17-preview, 2020-02-12, 2020-10-20'. The supported locations are ', westeurope, southcentralus, eastus, northeurope, southeastasia, westus2, japaneast, australiaeast, koreacentral, francecentral, centralus, eastus2, eastasia, westus, canadacentral, centralindia, uksouth, ukwest, southafricanorth, northcentralus, brazilsouth, switzerlandnorth, norwayeast, australiasoutheast'.

Running sscollector.pyz

user@debian:~/git/stormspotter/stormcollector$ python3 sscollector.pyz cli --azure
2021-01-21 07:44:33.874 | INFO     | stormcollector.auth:_get_resource_creds_from_cli:73 - Authenticating to login.microsoftonline.com with CLI credentials.
2021-01-21 07:44:33.875 | INFO     | stormcollector.arm:query_arm:134 - Starting enumeration for ARM - https://management.azure.com
2021-01-21 07:44:36.276 | INFO     | stormcollector.arm:query_arm:142 - Enumerating subscription and resource groups for tenant x
2021-01-21 07:44:36.677 | INFO     | stormcollector.arm:_query_management_certs:103 - Enumerating management certs for subscription: x
2021-01-21 07:44:38.727 | WARNING  | stormcollector.arm:_query_management_certs:111 - Forbidden: Cannot enumerate management certs for x
2021-01-21 07:44:38.729 | INFO     | stormcollector.arm:_query_rbac:81 - Enumerating rbac permissions for subscription: x
2021-01-21 07:44:55.568 | INFO     | stormcollector.arm:_query_rbac:97 - Finishing rbac permissions for subscription: x
2021-01-21 07:44:56.099 | INFO     | stormcollector.arm:_query_subscription:55 - Querying for resources in subscription - x
2021-01-21 07:45:06.525 | ERROR    | asyncio.events:_run:81 - An error has been caught in function '_run', process 'MainProcess' (2793), thread 'MainThread' (140393654445888):
Traceback (most recent call last):

  File "/home/user/.shiv/sscollector_c820988d13859f1bb61185859fa4382c0b0b940dd43447e95de851c2d7e834d0/site-packages/stormcollector/arm.py", line 28, in _query_resource
    response = await client.resources.get_by_id(resource_id, api_version)
                     │      │                   │            └ '2018-02-14'
                     │      │                   └ '/subscriptions/191c06c2-637b-4a1e-8efa-143e899b7ab4/resourceGroups/rg-sentinel-secops/providers/microsoft.insights/workbooks...
                     │      └ <property object at 0x7fafef2b02c0>
                     └ <azure.mgmt.resource.resources.aio._resource_management_client.ResourceManagementClient object at 0x7fafede64490>

  File "/home/user/.shiv/sscollector_c820988d13859f1bb61185859fa4382c0b0b940dd43447e95de851c2d7e834d0/site-packages/azure/mgmt/resource/resources/v2020_06_01/aio/operations/_resources_operations.py", line 1470, in get_by_id
    raise HttpResponseError(response=response, error_format=ARMErrorFormat)
          │                          │                      └ <class 'azure.mgmt.core.exceptions.ARMErrorFormat'>
          │                          └ <azure.core.pipeline.transport._aiohttp.AioHttpTransportResponse object at 0x7fafd769c160>
          └ <class 'azure.core.exceptions.HttpResponseError'>

azure.core.exceptions.HttpResponseError: (NoRegisteredProviderFound) No registered resource provider found for location 'canadacentral' and API version '2018-02-14' for type 'workbooks'. The supported api-versions are '2018-06-01-preview, 2018-06-17-preview, 2020-02-12, 2020-10-20'. The supported locations are ', westeurope, southcentralus, eastus, northeurope, southeastasia, westus2, japaneast, australiaeast, koreacentral, francecentral, centralus, eastus2, eastasia, westus, canadacentral, centralindia, uksouth, ukwest, southafricanorth, northcentralus, brazilsouth, switzerlandnorth, norwayeast, australiasoutheast'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
           │         │     └ {'__name__': '__main__', '__doc__': None, '__package__': '', '__loader__': <zipimporter object "sscollector.pyz/">, '__spec__...
           │         └ <code object <module> at 0x7faff1827f50, file "sscollector.pyz/__main__.py", line 2>
           └ <function _run_code at 0x7faff18760d0>
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
         │     └ {'__name__': '__main__', '__doc__': None, '__package__': '', '__loader__': <zipimporter object "sscollector.pyz/">, '__spec__...
         └ <code object <module> at 0x7faff1827f50, file "sscollector.pyz/__main__.py", line 2>

  File "sscollector.pyz/__main__.py", line 3, in <module>

  File "sscollector.pyz/_bootstrap/__init__.py", line 241, in bootstrap

  File "sscollector.pyz/_bootstrap/__init__.py", line 36, in run

  File "/home/user/.shiv/sscollector_c820988d13859f1bb61185859fa4382c0b0b940dd43447e95de851c2d7e834d0/site-packages/main.py", line 124, in main
    asyncio.run(run(args))
    │       │   │   └ Namespace(aad=False, auth='cli', azure=True, backfill=False, cloud='PUBLIC', config=None, get_creds=<function Context.auth at...
    │       │   └ <function run at 0x7faff126a430>
    │       └ <function run at 0x7faff11abc10>
    └ <module 'asyncio' from '/usr/lib/python3.8/asyncio/__init__.py'>

  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
           │    │                  └ <coroutine object run at 0x7fafef24a3c0>
           │    └ <function BaseEventLoop.run_until_complete at 0x7faff0f26e50>
           └ <_UnixSelectorEventLoop running=True closed=False debug=False>
  File "/usr/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
    self.run_forever()
    │    └ <function BaseEventLoop.run_forever at 0x7faff0f26dc0>
    └ <_UnixSelectorEventLoop running=True closed=False debug=False>
  File "/usr/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
    │    └ <function BaseEventLoop._run_once at 0x7faff0f2a940>
    └ <_UnixSelectorEventLoop running=True closed=False debug=False>
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
    │      └ <function Handle._run at 0x7faff0fa3700>
    └ <Handle <TaskStepMethWrapper object at 0x7fafef1ca310>()>
> File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
    │    │            │    │           │    └ <member '_args' of 'Handle' objects>
    │    │            │    │           └ <Handle <TaskStepMethWrapper object at 0x7fafef1ca310>()>
    │    │            │    └ <member '_callback' of 'Handle' objects>
    │    │            └ <Handle <TaskStepMethWrapper object at 0x7fafef1ca310>()>
    │    └ <member '_context' of 'Handle' objects>
    └ <Handle <TaskStepMethWrapper object at 0x7fafef1ca310>()>

  File "/home/user/.shiv/sscollector_c820988d13859f1bb61185859fa4382c0b0b940dd43447e95de851c2d7e834d0/site-packages/stormcollector/arm.py", line 203, in query_arm
    tenant_dict["subscriptions"].append(await result)
    │                                         └ <coroutine object as_completed.<locals>._wait_for_one at 0x7fafedf5a140>
    └ {'id': '/tenants/x', 'tenant_id': 'x', 'tenant_category...

  File "/usr/lib/python3.8/asyncio/tasks.py", line 616, in _wait_for_one
    return f.result()  # May raise f.exception().
           │ └ <method 'result' of '_asyncio.Task' objects>
           └ <Task finished name='Task-14' coro=<_query_subscription() done, defined at /home/user/.shiv/sscollector_c820988d13859f1bb6118...

  File "/home/user/.shiv/sscollector_c820988d13859f1bb61185859fa4382c0b0b940dd43447e95de851c2d7e834d0/site-packages/stormcollector/arm.py", line 69, in _query_subscription
    res = await _query_resource(rm_client, resource.id)
                │               │          │        └ '/subscriptions/x/resourceGroups/x/providers/microsoft.insights/workbooks...
                │               │          └ <azure.mgmt.resource.resources.v2020_06_01.models._models_py3.GenericResourceExpanded object at 0x7fafd76365e0>
                │               └ <azure.mgmt.resource.resources.aio._resource_management_client.ResourceManagementClient object at 0x7fafede64490>
                └ <function _query_resource at 0x7fafef57c040>

  File "/home/user/.shiv/sscollector_c820988d13859f1bb61185859fa4382c0b0b940dd43447e95de851c2d7e834d0/site-packages/stormcollector/arm.py", line 45, in _query_resource
    return await _query_resource(
                 └ <function _query_resource at 0x7fafef57c040>

  File "/home/user/.shiv/sscollector_c820988d13859f1bb61185859fa4382c0b0b940dd43447e95de851c2d7e834d0/site-packages/stormcollector/arm.py", line 29, in _query_resource
    return response.as_dict()
           └ None

AttributeError: 'NoneType' object has no attribute 'as_dict'
2021-01-21 07:45:06.559 | INFO     | main:main:125 - --- COMPLETE: 32.68561935424805 seconds. ---
2021-01-21 07:45:06.560 | INFO     | main:main:127 - Zipping up output...
2021-01-21 07:45:06.562 | INFO     | main:main:129 - OUTPUT: /home/user/git/stormspotter/stormcollector/results_20210121-074433.zip

Version:

user@debian:~/git/stormspotter/stormcollector$ unzip -p sscollector.pyz environment.json | jq .
{
  "always_write_cache": false,
  "build_id": "c820988d13859f1bb61185859fa4382c0b0b940dd43447e95de851c2d7e834d0",
  "built_at": "2021-01-19 05:41:16",
  "hashes": {},
  "no_modify": false,
  "reproducible": false,
  "script": null,
  "shiv_version": "0.4.0",
  "preamble": null,
  "entry_point": "main:main",
  "compile_pyc": false,
  "extend_pythonpath": true,
  "root": null
}

[legra-ms]

Sorry for the delay in response.

The API versions for a resource is current (sorta) dynamically discovered. It starts with a basic "2018-02-14" and is expected to error out in most cases, and pick up a valid version in that error you're seeing. I'm not sure why it's failing for that type of resource but that entire function will go away sometime in the next week or so and be replaced with an accurate list of resource providers and their versions.