user_agent incorrectly cast from Python list to string

[jaidisido]

Describe the bug We are leveraging the user_agent and user_agent_extra arguments in the botocore.config.Config to override/append a custom user agent.

Looking at CloudTrail logs, the result of the operation is a Python list that was cast to string and thus includes uncessary [:

userAgent”: “[Boto3/1.17.18 Python/3.9.2 Darwin/18.7.0 Botocore/1.20.18 my_user_agent/2.5.0]”

Notice the surrounding brackets which should not be there.

Steps to reproduce

botocore.config.Config(
        user_agent_extra=f"{my_user_agent}/{__version__}",
    )

Same result using user_agent argument

Expected behavior Correctly casting from a Python list to string:

userAgent”: “Boto3/1.17.18 Python/3.9.2 Darwin/18.7.0 Botocore/1.20.18 my_user_agent/2.5.0”

[kdaily]

Hi @jaidisido,

I'm trying to reproduce. Locally, changing the user agent with user_agent_extra works OK:

import boto3
import botocore

botocore.session.Session().set_debug_logger('')

uss = "hithere"

# Create a config
session_config = botocore.config.Config(
  user_agent_extra=f"{uss}/new_user_agent"
)

s3 = boto3.client('s3', config=session_config)

# Make an API call
response = s3.list_buckets()

Request sent:

2021-03-23 10:38:05,564 - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://s3.amazonaws.com/, headers={'User-Agent': b'Boto3/1.17.18 Python/3.8.5 Darwin/19.6.0 Botocore/1.20.18 hithere/new_user_agent', 'X-Amz-Date': b'20210323T173805Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAZZZZZZZZZZZZ/20210323/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=f25e721c2163c695ac4e0098e5581ed20330e81c7196c5b5409d5cedf6d98d6d'}>

Can you confirm:

1. What it looks like locally when you make this type of change, and
2. what CloudTrail does if you do not modify the header with user_agent_extra?

[jaidisido]

It seems to be somehow specific to the s3.get_bucket_location call:

import boto3
import botocore

botocore.session.Session().set_debug_logger('')

uss = "hithere"

# Create a config
session_config = botocore.config.Config(
  user_agent_extra=f"{uss}/new_user_agent"
)

s3 = boto3.client('s3', config=session_config)

# Make an API call
response = s3.get_bucket_location(Bucket="my_bucket")

The botocore log matches your output: 2021-03-23 19:02:24,529 - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://s3.amazonaws.com/my_bucket?location, headers={'User-Agent': b'Boto3/1.17.18 Python/3.9.2 Darwin/18.7.0 Botocore/1.20.18 hithere/new_user_agent'

But the attached CT log does have the [ within user-agent. Do you think the issue is with Cloud Trail instead?

I have anonymized parts of these outputs for security.

CT_user_agent.log

[kdaily]

Hi @jaidisido, thanks for the update. It seems likely that it's on CloudTrail, but I'll try to replicate. Can you confirm my second question:

What CloudTrail does if you do not modify the header with user_agent_extra?

[kdaily]

With my reproduction, I think I can confirm that the user agent appears as a list element even without user_agent_extra:

      "eventTime": "2021-03-24T17:49:46Z",
      "eventSource": "s3.amazonaws.com",
      "eventName": "ListObjects",
      "awsRegion": "us-west-2",
      "sourceIPAddress": "zzz.zzz.zzz.zzz",
      "userAgent": "[Boto3/1.17.18 Python/3.8.5 Darwin/19.6.0 Botocore/1.20.18]",
      "requestParameters": {
        "bucketName": "mybucket,
        "Host": "mybucket.s3.us-west-2.amazonaws.com",
        "encoding-type": "url"
      },

[kdaily]

I can also confirm that the user agent for the AWS CLI is not the only one that ends up as a list. I'll check on this with the CloudTrail team.

[manjunathsc]

Hi @jaidisido , @kdaily what is the reason of having unnecessary brackets attached with useragents. ? could you please help me to understand ? and for what useragents it get attached?