KAPE S3 Pre-signed PUT URL non-functional

[Brailagise]

Hello!

I am trying to implement KAPE to retrieve artifacts from remote hosts and upload them to an AWS S3 bucket using PUT pre-signed urls. I have set up the infrastructure to generate the pre-signed url but it appears that a matching content-type based on the content-type provided when the pre-signed url is made is required for a successful upload.

SUCCEEDS

After watching some of the traffic in Fiddler I can see that the initial test to make sure that the url is live passes a Content-Type: application/text and is successful if I dictate application/text as the content-type in the url generator. However after the .zip file is created it tries to push the actual resultant file to the url and it fails with a 403. Additionally it does not appear to set a Content-Type header at all when attempting to push the .zip file unlike when the test is performed initially.

FAILS

I would recommend possibly allowing a flag to bypass S3 url testing and ensuring that an appropriate Content-Type can be set in the headers for files that can be uploaded to an S3 pre-signed url. I would be happy to provide any additional information that may be of assistance.

[EricZimmerman]

s3url This allows for using a presigned URL to upload a collection to S3. Make sure you generate a PUT URL for this to work. The AWS CLI does not allow for making PUT URLs, so you may need to use the SDK to generate these. When using this option, you do not need any other options, as all the necessary info is embedded into the URL.

how did you generate these? theres also something with like a v4 signature thats needed, but this has been working since forever

[EricZimmerman]

id also not specify a type at all.

[Brailagise]

This is running in AWS Lambda using Python 3.11 under a context that has the appropriate GET / PUT permissions to the bucket. I have successfully been able to upload / get from the bucket with the restrictions I mentioned above.

Bulk of the actual code that handles that is below. If I specify no type I get a 403 forbidden regardless of what I try and upload. If I set the ContentType in the generate_presigned_url function it works as expected.

# Expected handler event content:
# {
#     "bucket_name": "<SNIP>",
#     "object_key": "EXAMPLEACQUISITION.ZIP",
#     "method": "PUT",
#     "expiry": 1000
# }

# Setup
import logging
import boto3
import json
from botocore.exceptions import ClientError

logger = logging.getLogger()

# Function handler method
def lambda_handler(event, context):
    logger.info("Starting URL Generation...")

    # Extract event content
    client = boto3.client('s3', region_name='us-east-1')
    method = event['method']
    bucket = event['bucket_name']
    key = event['object_key']
    expiry = event['expiry']

    url = None
    response = None

        url = generate_presigned_url(
            client,
            "put_object",
            {"Bucket": bucket, "Key": key,) #### I am using POSTMAN to test comparisons. If I leave it without "ContentType": "<Value>" I instantly get a 403 regardless of what I upload. If I specify application/zip for example I can upload a zip file as long as that is also dictated in the header as the content-type.
            expiry
      )

[EricZimmerman]

like i said, signature v4

https://github.com/EricZimmerman/KapeFiles/issues/519#issuecomment-1188999589

[EricZimmerman]

# Imports
import boto3
from botocore.client import Config

# Globals
bucket = 'cloudymccloudface'

# Get hostname
host = input('Enter hostname: ')
host = host + '.zip'

# Generate presigned URL
psu = boto3.client('s3', config=Config(signature_version='s3v4')).generate_presigned_url(ClientMethod='put_object', Params={'Bucket': bucket, 'Key': host})

# Display output
print(psu)

[Brailagise]

That worked perfectly! I think the confusion came in at the difference in signature_version='v4' which I believe is the default for boto3 from my reading and signature_version='s3v4'. Some light googling found some other github issue threads that seem to have the same confusion. Would this be worth mentioning in the KAPE docs? I would be happy to try and put together an update on the repo.

[EricZimmerman]

you can put a kapedocs PR together with a code example i suppose, or add it to the tips and tricks, but the KapeFiles repo had this issue in there with the solution too