search

okigan / awscurl

curl-like access to AWS resources with AWS Signature Version 4 request signing.
MIT License
755 stars 94 forks source link

Fix #106 (But may not compatible with #90) #110

Closed ToshihikoMakita closed 3 years ago

ToshihikoMakita commented 3 years ago

For details, refer to #106 issue threads.

ToshihikoMakita commented 3 years ago

Mistake for interpretation of --data-binary parameter.

okigan commented 3 years ago

@ToshihikoMakita double checking since you closed the issue - no changes needed for awscurl code base?

ToshihikoMakita commented 3 years ago

As I mentioned above, I misunderstood the --data-binary parameter. I'm modifying awacurl.py again and will test it with accessing S3 service, If it succeeds I will submit pull-request again. Thank you for your understanding.

ToshihikoMakita commented 3 years ago

Applying encoding="UTF-8" for file open when "Content-Type: application/json" works fine. Accessing AWS Elasticsearch by this method works perfectly. But I could not resolve the error when I tried to upload binary .jpg file to AWS S3 yet.

okigan commented 3 years ago

@ToshihikoMakita maybe applicable: the content type correctly returned from your service?

ToshihikoMakita commented 3 years ago

No, low level error occurred and I have no way to resolve now:

I have changed with open(filename, "rb") as post_data_file: when --data-bainary is specified.

[The change of awscurl.py]

if args.header is None:
    args.header = default_headers

if data is not None and data.startswith("@"):
    filename = data[1:]
    if args.data_binary:
        with open(filename, "rb") as post_data_file:
            data = post_data_file.read()
    elif "Content-Type: application/json" in args.header or "Content-Type: application/x-ndjson" in args.header:
        with open(filename, "r", encoding="UTF-8") as post_data_file:
            data = post_data_file.read()
    else:
        with open(filename, "r") as post_data_file:
            data = post_data_file.read()

[awss3curl-db.ps1]

aws sts assume-role --role-arn arn:aws:iam::999999999999:role/S3FullAccess --role-session-name "RoleSession1" --profile S3TestAccess > assume-role-output.txt
$json = ConvertFrom-Json -InputObject (Get-Content assume-role-output.txt -Raw)
$env:AWS_ACCESS_KEY_ID = $json.Credentials.AccessKeyId
$env:AWS_SECRET_ACCESS_KEY = $json.Credentials.SecretAccessKey
$env:AWS_SECURITY_TOKEN = $json.Credentials.SessionToken
Write-Host 'AWS_ACCESS_KEY_ID: ' $env:AWS_ACCESS_KEY_ID
Write-Host 'AWS_SECRET_ACCESS_KEY: ' $env:AWS_SECRET_ACCESS_KEY
Write-Host 'AWS_SECURITY_TOKEN: ' $env:AWS_SECURITY_TOKEN
Write-Host 'Key is valid until: ' $json.Credentials.Expiration
cmd /c awscurl --service s3 --region yy-yyyy-y $args

[The logs]

PS C:\Users\toshi\OneDrive\Documents\ElasticSearch\command-s3> ./awss3curl-db.ps1 -H 'Content-Type: image/jpeg' --data-binary -d "@20151118_133658.jpg" -X POST https://xxxxxxxx-access-internet-999999999999.s3-accesspoint.yy-yyyy-y.amazonaws.com/20151118_133658.jpg
AWS_ACCESS_KEY_ID:  AAAAAAAAAAAAAAAAAAAA
AWS_SECRET_ACCESS_KEY:  eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee
AWS_SECURITY_TOKEN:  zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
Key is valid until:  2021/04/28 9:35:35
Traceback (most recent call last):
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1049, in _send_output
    self.send(chunk)
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 971, in send
    self.sock.sendall(data)
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1173, in send
    return self._sslobj.write(data)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\adapters.py", line 439, in send
    resp = conn.urlopen(
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\util\retry.py", line 531, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\packages\six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1049, in _send_output
    self.send(chunk)
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 971, in send
    self.sock.sendall(data)
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1173, in send
    return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\Scripts\awscurl-script.py", line 33, in <module>
    sys.exit(load_entry_point('awscurl==0.21', 'console_scripts', 'awscurl')())
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\site-packages\awscurl-0.21-py3.9.egg\awscurl\awscurl.py", line 505, in main
    inner_main(sys.argv[1:])
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\site-packages\awscurl-0.21-py3.9.egg\awscurl\awscurl.py", line 483, in inner_main
    response = make_request(args.request,
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\site-packages\awscurl-0.21-py3.9.egg\awscurl\awscurl.py", line 135, in make_request
    return __send_request(uri, data, headers, method, verify)
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\site-packages\awscurl-0.21-py3.9.egg\awscurl\awscurl.py", line 329, in __send_request
    response = requests.request(method, uri, headers=headers, data=data, verify=verify)
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
PS C:\Users\toshi\OneDrive\Documents\ElasticSearch\command-s3>