How do I use the boto3 library correctly?

[fake-name]

Hi there!

I'm flailing about trying to implement a salt-cloud plugin for AWS Lightsail, and I can't figure out how the heck to make all the import magic work.

Basically, I need access to the salt.utils.boto3.get_connection() call, in my plugin.

Searching the codebase for boto3, I can see it used extensively in various modules, but the ec2 cloud plugin seems to work entirely over HTTP(S) (I think?), which seems odd because I'm pretty sure you can control EC2 stuff using boto, and that seems much more "correct" then piecemeal HTTP stuff.

Anyways, if I import the boto3 module directly: import salt.utils.boto3, and then use it:

Traceback (most recent call last):
  File "/media/Storage/Scripts/salt/salt/cloud/clouds/lightsail.py", line 505, in _query
    conn = __get_connection()
  File "/media/Storage/Scripts/salt/salt/cloud/clouds/lightsail.py", line 494, in __get_connection
    conn = salt.utils.boto3.get_connection('lightsail')
  File "/media/Storage/Scripts/salt/salt/utils/boto3.py", line 207, in get_connection
    keyid, profile)
  File "/media/Storage/Scripts/salt/salt/utils/boto3.py", line 119, in _get_profile
    if not region and _option(service + '.region'):
  File "/media/Storage/Scripts/salt/salt/utils/boto3.py", line 100, in _option
    if value in __opts__:
NameError: global name '__opts__' is not defined

Apparently some of the mysterious import-magic is broken when it's used directly. Looking around for things that use it, it seems you have to set up the boto3... thing somehow with either a module level __init__() (how does that even work?), or including the setup call in your __virtual__() function.

The docs for the boto3.py say

import salt.utils.boto3

def __virtual__():
    # only required in 2015.2
    salt.utils.compat.pack_dunder(__name__)

    __utils__['boto.apply_funcs'](__name__, 'platform_name')

def test():
    conn = _get_conn()
    vpc_id = _cache_id('test-vpc')

Which is confusing, because I'm pretty sure the locations I've found using it refer to it as __utils__['boto3.assign_funcs'](__name__, 'platform_name'), so I guess the docs in boto3.py are just wrong? Anyways, putting either of the above (or the possible alternatives, boto3.assign_funcs, boto3.apply_funcs) in my __virtual__() call fails with a key error:

Traceback (most recent call last):
  File "/media/Storage/Scripts/salt/salt/loader.py", line 1676, in process_virtual
    virtual = getattr(mod, virtual_func)()
  File "/media/Storage/Scripts/salt/salt/cloud/clouds/lightsail.py", line 72, in __virtual__
    __utils__['boto3.apply_funcs'](__name__, 'lightsail')
  File "/media/Storage/Scripts/salt/salt/loader.py", line 1125, in __getitem__
    func = super(LazyLoader, self).__getitem__(item)
  File "/media/Storage/Scripts/salt/salt/utils/lazy.py", line 101, in __getitem__
    raise KeyError(key)
KeyError: 'boto3.apply_funcs'

Since elsewhere, the __utils__['boto3.assign_funcs'](__name__, 'platform_name') is located in a module level __init__() call, I tried that too, but you get the same key-error.

So, the above said, is it possible for my code to access the salt.utils.boto3 module, and if so, what's the correct way to do so? The enormous quantity of metaprogramming is breaking my tiny little brain.

For that matter, how does the stuff in __utils__ even work? It seems like it's mucking about in the available namespace on-the-fly, and functions and modules are getting dropped into the active local namespace somehow, which makes following anything almost impossible.

[gtmanfred]

__utils__ loads stuff from salt.utils into the dunder dictionary, and then injects the __opts__ dictionary in them. So you should be using __utils__['boto3.assign_funcs'] not apply_funcs like your error message is showing you using.

To learn more about this, you should check out salt.loader and the _module_dirs functionl, this lets us load from the salt/cache/extmods directories.

 File "/media/Storage/Scripts/salt/salt/cloud/clouds/lightsail.py", line 72, in __virtual__
   __utils__['boto3.apply_funcs'](__name__, 'lightsail')

Then you can specify the get_conn_funcname, which should be what is used to run the boto3.get_connection iirc.

Here is an example of the elasticsearch module

def __init__(opts):
    salt.utils.compat.pack_dunder(__name__)
    if HAS_BOTO3:
        __utils__['boto3.assign_funcs'](__name__, 'elasticache',
                  get_conn_funcname='_get_conn',
                  cache_id_funcname='_cache_id',
                  exactly_one_funcname=None)

[fake-name]

utils loads stuff from salt.utils into the dunder dictionary, and then injects the opts dictionary in them. So you should be using __utils__['boto3.assign_funcs'] not apply_funcs like your error message is showing you using.

I'm pretty sure I tried all 4 possible combinations of (boto, boto3 and assign_funcs, apply_funcs), and they all failed.

Which is confusing, because I'm pretty sure the locations I've found using it refer to it as __utils__['boto3.assign_funcs'](__name__, 'platform_name'), so I guess the docs in boto3.py are just wrong? Anyways, putting either of the above (or the possible alternatives, boto3.assign_funcs, boto3.apply_funcs) in my __virtual__() call fails with a key err.

Yep, doesn't work:

```

[DEBUG   ] Could not LazyLoad boto3.assign_funcs: 'boto3' __virtual__ returned False
[DEBUG   ] Error loading clouds.lightsail: __init__ failed
Traceback (most recent call last):
  File "/media/Storage/Scripts/salt/salt/loader.py", line 1468, in _load_module
    module_init(self.opts)
  File "/media/Storage/Scripts/salt/salt/cloud/clouds/lightsail.py", line 99, in __init__
    __utils__['boto3.assign_funcs'](
  File "/media/Storage/Scripts/salt/salt/loader.py", line 1125, in __getitem__
    func = super(LazyLoader, self).__getitem__(item)
  File "/media/Storage/Scripts/salt/salt/utils/lazy.py", line 101, in __getitem__
    raise KeyError(key)
KeyError: 'boto3.assign_funcs'
```

See: https://github.com/fake-name/salt/blob/lightsail/salt/cloud/clouds/lightsail.py#L96-L100

I tried the extended version you posted, and it doesn't work either.

I've been working off the elasticsearch and other AWS modules, and it was only after I couldn't get anything to work I asked here.

[gtmanfred]

@ryan-lane can you take a look at this and see if you can assist?

Thanks Daniel

[ryan-lane]

I don't think you can access execution or state modules from salt-cloud.

[ryan-lane]

I think most likely what you want to do is implement lightsail like one of the newer boto_ modules (see the recently refactored boto_sqs module, which now uses boto3), where you implement execution and state modules, rather than a salt-cloud plugin.

[fake-name]

@ryan-lane - AWS Lightsail is a VPS provider, no different then DigitalOcean, Vultr, or any of the other salt-cloud providers.

Additionally, I'm specifically interested in using it in a context where I'm already using other cloud providers, via salt-cloud, so if it's not a salt-cloud module, it's useless to me anyways.

[ryan-lane]

Ok. I'm not the right person to talk to then, as I don't use salt cloud.

On Aug 15, 2017 8:32 PM, "Connor Wolf" notifications@github.com wrote:

@ryan-lane https://github.com/ryan-lane - AWS Lightsail is a VPS provider, no different then DigitalOcean, Vultr, or any of the other salt-cloud providers. Additionally, I'm specifically interested in using it in a context where I'm already using other cloud providers, via salt-cloud, so if it's not a salt-cloud module, it's useless to me anyways.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/saltstack/salt/issues/42906#issuecomment-322550199, or mute the thread https://github.com/notifications/unsubscribe-auth/ABd5MkkuHt1-TmRkDZiST53Evfmrd2Knks5sYeRGgaJpZM4O18xk .

[fake-name]

Ok, no problem.

I guess the answer here is more or less just use boto directly, rather then trying to rely on the existing infrastructure? Unfortunately, the salt-cloud ec2 module seems to not use boto, but rather the amazon HTTP(s) interface for some reason, so there's no existing stuff I can base off of.

[ryan-lane]

I think someone from saltstack Inc needs to help out here.

On Aug 15, 2017 9:59 PM, "Connor Wolf" notifications@github.com wrote:

Ok, no problem.

I guess the answer here is more or less just use boto directly, rather then trying to rely on the existing infrastructure? Unfortunately, the salt-cloud ec2 module seems to not use boto, but rather the amazon HTTP(s) interface for some reason, so there's no existing stuff I can base off of.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/saltstack/salt/issues/42906#issuecomment-322572046, or mute the thread https://github.com/notifications/unsubscribe-auth/ABd5Mh-75NQIeRf3R4t7rx-SGj_16bq0ks5sYfiZgaJpZM4O18xk .

[gtmanfred]

Just fyi, i haven't forgotten about this, just working on other stuff at the moment, hopefully will have some time at some point to figure out how this could be used.

On Tue, Aug 15, 2017 at 3:28 PM, Ryan Lane notifications@github.com wrote:

I think someone from saltstack Inc needs to help out here.

On Aug 15, 2017 9:59 PM, "Connor Wolf" notifications@github.com wrote:

Ok, no problem.

I guess the answer here is more or less just use boto directly, rather then trying to rely on the existing infrastructure? Unfortunately, the salt-cloud ec2 module seems to not use boto, but rather the amazon HTTP(s) interface for some reason, so there's no existing stuff I can base off of.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/saltstack/salt/issues/42906#issuecomment-322572046, or mute the thread https://github.com/notifications/unsubscribe- auth/ABd5Mh-75NQIeRf3R4t7rx-SGj_16bq0ks5sYfiZgaJpZM4O18xk

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/saltstack/salt/issues/42906#issuecomment-322594952, or mute the thread https://github.com/notifications/unsubscribe-auth/AAssodlDHEA1YWY3prGgX47goaW1aRcjks5sYg16gaJpZM4O18xk .

[fake-name]

Any movement here? I'd really like lightsail integration.

[gtmanfred]

Sorry, one second. I think what you actually want is something like this.

# Import third party libs
try:
    #pylint: disable=unused-import
    import boto3
    #pylint: enable=unused-import
    from botocore.exceptions import ClientError
    logging.getLogger('boto3').setLevel(logging.CRITICAL)
    HAS_BOTO3 = True
except ImportError:
    HAS_BOTO3 = False

def __virtual__():
    '''
    Only load if boto libraries exist and if boto libraries are greater than
    a given version.
    '''
    if not HAS_BOTO3:
        return (False, 'The boto3_lightsail module could not be loaded: boto3 libraries not found')
    return True

def __init__(opts):
    salt.utils.compat.pack_dunder(__name__)
    if HAS_BOTO3:
        __utils__['boto3.assign_funcs'](__name__, 'lightsail`)

def test():
    conn = _get_conn()
    vpc_id = _cache_id('test-vpc')

__utils__ is not available when __virtual__ is run. utils isn't packed in until after that point, and then __init__ is run, so you want to use assign_funcs in there.

Thanks, Daniel

[fake-name]

One of the things I was hoping was that I'd be able to re-use the credential management stuff from the existing boto wrappers.

Should a salt-cloud plugin just not touch them entirely?

[gtmanfred]

you can use that, just put the boto3.assign_funcs() stuff in __init__, and it will add the _get_conn() stuff.

The import for boto3 is just to check if it is installed, it doesn't actually need to be used.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[fake-name]

This is still an active issue, though I'm strongly looking at switching to Ansible these days.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[fake-name]

Still no solution. I think I'm going to take the time to migrate to ansible at some point.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[sagetherage]

@fake-name looks like this hasn't been worked in some time and you are still having issues, so I will self-assign, but be aware that is to get information and a team-core assignment.

[fake-name]

At this point, I think this should be treated as basically a bug report about missing documentation.

[sagetherage]

I apologize it took me while to get back to this, thank you for your comment @fake-name and I will move it to get the bug documentation worked, thank you. @saltstack/docs-working-group

[barbaricyawps]

Will someone on this thread please confirm whether the boto3 library is still undocumented in Salt 3005+? Does @fake-name want to weigh in or did he actually migrate to Ansible?

[fake-name]

After all the issues I've had with salt-cloud, and it's apparently nearly-unmaintained status, I'm no longer interested in putting any energy into helping with saltstack.

I'd imagine the issue is still present, but I haven't bothered to check.

[alan-cugler]

@barbaricyawps I do not use anything that is actually connected to salt-cloud as it falls short on multiple stability points and breaks from many standard salt practices. I would recommend to the open-salt PM (not sure who that is) deprecating or marking as "wont fix" salt-cloud related documentation.

The best way for engineers to leverage salt + boto3 is to make salt modules with the boto3 library for their use cases as needed. Anytime I need boto3 for a client or internal Terminal Labs project I make those boto3 modules from scratch or develop off of one I have developed before.

Being a documentation focused discussion, I would propose make a "how to develop a salt modules on boto3" article with a couple examples. This would be the best way to enable engineers without taking on the boto3 technical debt in the salt team as the variables and options are extensive and complex. You would effectively have to be a aws expert for every facet of boto3 documentation which is unreasonable for a "jack of all trades" tool like salt.