search

ansible-collections / community.aws

Ansible Collection for Community AWS
GNU General Public License v3.0
187 stars 396 forks source link

Don't use S3 for file transfers in SSM connection plugin #1895

Open alicekaerast opened 1 year ago

alicekaerast commented 1 year ago

Summary

As noted in #1853 we believe that using S3 to transfer all of the small files required to run tasks is part of what is making the aws_ssm connection plugin so slow

We should provide an option to transfer files using the SSM session, similar to the piped method in the ssh connection

Issue Type

Feature Idea

Component Name

aws_ssm

Additional Information

I believe that one of the performance issues with the aws_ssm connection plugin is that all files are transferred using S3, and that there are multiple file transfers required for each task. Ansible needs to transfer the ansiball_z task files in addition to the files we actually need to transfer, and it doesn't re-use these

The ssh connection has a piped option that uses dd to transfer files. Whilst this is not compatible with Windows hosts, it does suggest that this way of transferring files could work here

https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/connection/ssh.py#L1285

I believe that implementing Linux and Windows equivalents of that piped mode would improve performance with the aws_ssm connection plugin, though maybe only for smaller files. In fact, the Windows equivalent is probably more important than the Linux version because for Linux hosts many people can instead use SSH over SSM and we have found that Windows hosts take a bigger performance hit than Linux hosts when running this plugin

Code of Conduct

ansibullbot commented 1 year ago

Files identified in the description:

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

ckingparra-st commented 3 months ago

I just submitted a similar issue. This can be achieved by setting up port forwarding and streaming file content over it. https://gist.github.com/lukeplausin/4b412d83fb1246b0bed6507b5083b3a7