Continued issue with Client basic HTTP auth as per #1699

[Chanonry]

Long story short

As before...

I was making calls with aiohttp client to a Companies House API (which requires basic HTTP authentication) and it works fine when I requested metadata but I request document download there is a redirect to amazon S3 and the request fails.

It appears that aiohttp passes authentication after redirection ?? This causes an issue as amazon requires that there should be only one auth method.

Error received includes - "Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified"

I've tested with the requests lib and it works fine for identical requests.

Expected behaviour

Only one auth mechanism should be used on the redirect

Actual behaviour

end response from S3 ends with an authentication error, not entirely sure why as #1699 should have fixed this ?

Steps to reproduce

Would need to have access to server redirecting to s3?

Your environment

macOS 10.12.6 python 3.5.4 aiohttp 2.3.6

[jtarang]

Can you show a example .. seems to me seems to me login is being passed. You might need to make a clientsession and add headers to it. If you need a example let me know .

[Chanonry]

Test code attached below. Just the essentials in this.

There are 2 url’s.

First one (commented out) is a request for filing history - which works.

Second one is request for a document to a url contained in the filing history and the initial request to the url is then redirected to an Amazon s3 bucket. This fails.

Hope this helps.

I am away for a couple of days but if you need anything else just ask.

Cheers

Andy

On 9 Jan 2018, at 07:03, Jasmit Tarang notifications@github.com wrote:

Can you show a example .. seems to me seems to me login is being passed. You might need to make a clientsession and add headers to it. If you need a example let me know .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aio-libs/aiohttp/issues/2610#issuecomment-356197868, or mute the thread https://github.com/notifications/unsubscribe-auth/AT6OHSFCRQimDd6UlkDVTzpCnO5wQV9Tks5tIw9agaJpZM4RDS81.

[jtarang]

Maybe I am going crazy but I dont see any code.

[Chanonry]

Can you read the .py file attached to my reply??

A

Sent from my iPhone

On 10 Jan 2018, at 19:24, Jasmit Tarang notifications@github.com wrote:

Maybe I am going crazy but I dont see any code.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[jtarang]

Nope don't see it . Unfortunately. Just paste it in a gist or just markdown.

[Chanonry]

Yeah fine. I will run it and add in output from success and fail so you have the full picture

A

Sent from my iPhone

On 11 Jan 2018, at 00:40, Jasmit Tarang notifications@github.com wrote:

Nope don't see it . Unfortunately. Just paste it in a gist or just markdown.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[Chanonry]

Did you get my email with test code and output ok ??

A

On 11 Jan 2018, at 00:40, Jasmit Tarang notifications@github.com wrote:

Nope don't see it . Unfortunately. Just paste it in a gist or just markdown.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aio-libs/aiohttp/issues/2610#issuecomment-356786218, or mute the thread https://github.com/notifications/unsubscribe-auth/AT6OHWlYPEJ9U2Pt2sXnPKampQ2ZjaBbks5tJVh3gaJpZM4RDS81.

[jtarang]

Nope . Just make a gist.

[Chanonry]

sent this last week

let me know if you get this email…. import asyncio import aiohttp

import auth # own authentication class

async def test_session(loop, url, auth): api_session = aiohttp.ClientSession(loop=loop, auth=auth) async with api_session.get(url) as response: return await response.read()

if name == 'main':

# metadata url
test_url = 'https://api.companieshouse.gov.uk/company/04025989/filing-history?category=accounts&items_per_page=100&start_index=0 <https://api.companieshouse.gov.uk/company/04025989/filing-history?category=accounts&items_per_page=100&start_index=0>'

# document url
# test_url = 'https://document-api.companieshouse.gov.uk/document/w3HCtbsw1OWNup0R3wbBxtbE3nixKhNwVYACspslbNA/content <https://document-api.companieshouse.gov.uk/document/w3HCtbsw1OWNup0R3wbBxtbE3nixKhNwVYACspslbNA/content>'

# api Oauth from secrets
oauth_path = '/Users/andyspence/Project/ccML_async/asyncImporter/config/'
oauth_file = 'ch_api_key.yaml'

# decrypt authorisation keys
decoder = auth.Auth()
BAuth = aiohttp.BasicAuth(decoder.read(oauth_path, oauth_file), '')

loop = asyncio.get_event_loop()

# fetch_task = asyncio.ensure_future(test_session(loop=loop, url=test_url, auth=BAuth))
# group = asyncio.gather(fetch_task, loop=loop, return_exceptions=False)
group = asyncio.gather(test_session(loop=loop, url=test_url, auth=BAuth), loop=loop, return_exceptions=False)
c = loop.run_until_complete(group)

print(c[0])

Run as above and Output is :

/Users/andyspence/anaconda3/bin/python3.6 /Users/andyspence/Project/ccML_async/asyncImporter/tests/trial_aioget.py Unclosed client session client_session: <aiohttp.client.ClientSession object at 0x106ce1828> Unclosed connector connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x104b9e630>, 676.943253907)]'] connector: <aiohttp.connector.TCPConnector object at 0x106ce15c0> b'{"total_count":18,"items":[{"action_date":"2015-12-31","date":"2016-07-27","description_values":{"made_up_date":"2015-12-31"},"paper_filed":true,"type":"AA","links":{"self":"/company/04025989/filing-history/MzE1MzcxMDI2OGFkaXF6a2N4","document_metadata":"https://document-api.companieshouse.gov.uk/document/w3HCtbsw1OWNup0R3wbBxtbE3nixKhNwVYACspslbNA https://document-api.companieshouse.gov.uk/document/w3HCtbsw1OWNup0R3wbBxtbE3nixKhNwVYACspslbNA”},……etc etc

Process finished with exit code 0

Switch url’s to the second test_url and run

Output is :

/Users/andyspence/anaconda3/bin/python3.6 /Users/andyspence/Project/ccML_async/asyncImporter/tests/trial_aioget.py Unclosed client session client_session: <aiohttp.client.ClientSession object at 0x10f18e828> Unclosed connector connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x10d93c630>, 961.485947615)]'] connector: <aiohttp.connector.TCPConnector object at 0x10f18e5c0> b'<?xml version="1.0" encoding="UTF-8"?>\nInvalidArgumentOnly one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specifiedAuthorizationBasic NE10N3hhczlXS0FpRjFIYThmRThRUGhnQlVwWm5LZlJUZVFBMFBHYjo=679BB15D96599042hJ1M6t8yznnspSDb7LQAmQ3wz54/zxVMrimdgg3k8CSMbUasOTherfocKKhZ0wRPEOMUVK8ld7I='

Process finished with exit code 0

On 16 Jan 2018, at 00:11, Jasmit Tarang notifications@github.com wrote:

Nope

[Chanonry]

OK, I assume you could not read the last email either so here is code and output in this format. Harder to read but ….

import asyncio import aiohttp

import auth # own authentication class

async def test_session(loop, url, auth): api_session = aiohttp.ClientSession(loop=loop, auth=auth) async with api_session.get(url) as response: return await response.read()

if name == 'main':

metadata url

test_url = 'https://api.companieshouse.gov.uk/company/04025989/filing-history?category=accounts&items_per_page=100&start_index=0'

document url

test_url = 'https://document-api.companieshouse.gov.uk/document/w3HCtbsw1OWNup0R3wbBxtbE3nixKhNwVYACspslbNA/content'

api Oauth from secrets

oauth_path = '/Users/andyspence/Project/ccML_async/asyncImporter/config/' oauth_file = 'ch_api_key.yaml'

decrypt authorisation keys

decoder = auth.Auth() BAuth = aiohttp.BasicAuth(decoder.read(oauth_path, oauth_file), '')

loop = asyncio.get_event_loop()

fetch_task = asyncio.ensure_future(test_session(loop=loop, url=test_url, auth=BAuth))

group = asyncio.gather(fetch_task, loop=loop, return_exceptions=False)

group = asyncio.gather(test_session(loop=loop, url=test_url, auth=BAuth), loop=loop, return_exceptions=False) c = loop.run_until_complete(group)

print(c[0])

Run as above and Output is :

/Users/andyspence/anaconda3/bin/python3.6 /Users/andyspence/Project/ccML_async/asyncImporter/tests/trial_aioget.py Unclosed client session client_session: <aiohttp.client.ClientSession object at 0x106ce1828> Unclosed connector connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x104b9e630>, 676.943253907)]'] connector: <aiohttp.connector.TCPConnector object at 0x106ce15c0> b'{"total_count":18,"items":[{"action_date":"2015-12-31","date":"2016-07-27","description_values":{"made_up_date":"2015-12-31"},"paper_filed":true,"type":"AA","links":{"self":"/company/04025989/filing-history/MzE1MzcxMDI2OGFkaXF6a2N4","document_metadata":"https://document-api.companieshouse.gov.uk/document/w3HCtbsw1OWNup0R3wbBxtbE3nixKhNwVYACspslbNA https://document-api.companieshouse.gov.uk/document/w3HCtbsw1OWNup0R3wbBxtbE3nixKhNwVYACspslbNA”},……etc etc

Process finished with exit code 0

Switch url’s to the second test_url and run

Output is :

/Users/andyspence/anaconda3/bin/python3.6 /Users/andyspence/Project/ccML_async/asyncImporter/tests/trial_aioget.py Unclosed client session client_session: <aiohttp.client.ClientSession object at 0x10f18e828> Unclosed connector connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x10d93c630>, 961.485947615)]'] connector: <aiohttp.connector.TCPConnector object at 0x10f18e5c0> b'<?xml version="1.0" encoding="UTF-8"?>\nInvalidArgumentOnly one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specifiedAuthorizationBasic NE10N3hhczlXS0FpRjFIYThmRThRUGhnQlVwWm5LZlJUZVFBMFBHYjo=679BB15D96599042hJ1M6t8yznnspSDb7LQAmQ3wz54/zxVMrimdgg3k8CSMbUasOTherfocKKhZ0wRPEOMUVK8ld7I='

Process finished with exit code 0

On 11 Jan 2018, at 00:40, Jasmit Tarang notifications@github.com wrote:

Nope don't see it . Unfortunately. Just paste it in a gist or just markdown.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aio-libs/aiohttp/issues/2610#issuecomment-356786218, or mute the thread https://github.com/notifications/unsubscribe-auth/AT6OHWlYPEJ9U2Pt2sXnPKampQ2ZjaBbks5tJVh3gaJpZM4RDS81.

[jtarang]

I can't really test this as its just not complete. Do they have a test website or something? Test creds?

Constants: requests: did you use your own auth class

If not that might be the issue there .

[Chanonry]

No test website for testing programmatic access - pita.

I will set up another account for this test application and we can hard code the access keys and test with both requests and aiohttp. That will removes/highlights errors elsewhere and demonstrates the anticipated responses.

I will set that up this am and email you today. If you don’t get it today let me know

A

On 18 Jan 2018, at 04:51, Jasmit Tarang notifications@github.com wrote:

I can't really test this as its just not complete. Do they have a test website or something? Test creds?

Constants: requests: did you use your own auth class

If not that might be the issue there .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aio-libs/aiohttp/issues/2610#issuecomment-358537003, or mute the thread https://github.com/notifications/unsubscribe-auth/AT6OHVmSY4esSzR_A-vi_jH701QZ22lPks5tLs20gaJpZM4RDS81.

[Chanonry]

Right, as per previous email, I have set up an account specifically for this test. API key is hardcoded here. Not ideal but I will delete later and at least it allows us to exclude other code.

I have refactored to 2 test urls and perform a get on both with requests and aiohttp.

Requests behaves as expected. test_1 url is metadata for one company and is returned correctly. test_2 url is a pdf doc, also returned correctly.

Aiohttp gets the correct response from test_1 but returns 'InvalidArgumentOnly one auth mechanism allowed’ for test_2 url.

Kind of it in a nutshell.

The redirect is to a Amazon aws to retrieve the document from an s3 bucket.

Test code:

""" test for the Caller class for document import"""

import asyncio import aiohttp import requests

registered with the API for this app only -> use full account auth if refactoring

KEY = 'ba6xUk6iXpVg_iozY8RST8MdRgZHj_4_yrEt8__g' PASS = ''

def requests_get(url): r = requests.get(url, auth=(KEY, PASS)) print(r.status_code) return r

async def aiohttp_get(loop, url, auth): with aiohttp.ClientSession(loop=loop, auth=auth) as api_session: print('api session context') print(BAuth)

    async with api_session.get(url) as response:
        r = await response.read()
        print(url)
        print(r)
        return

if name == 'main':

test_1 = 'https://api.companieshouse.gov.uk/company/04025989/filing-history?category=accounts&items_per_page=100&start_index=0'
test_2 = 'https://document-api.companieshouse.gov.uk/document/w3HCtbsw1OWNup0R3wbBxtbE3nixKhNwVYACspslbNA/content'

print('username: {}'.format(KEY))
print('password{}'.format(PASS))

# ----------------------------------------------------------------------------------------------------------------
# test with requests

# get the document metadata
print('\nrequests with test_1')
resp = requests_get(test_1)
print(resp.json())

# get one documents content
print('\nrequests with test_2')
resp = requests_get(test_2)
print(resp.text)  # pdf file contents

# ----------------------------------------------------------------------------------------------------------------
# tests with aiohttp
BAuth = aiohttp.BasicAuth(KEY, PASS)
print('\n', BAuth)

loop = asyncio.get_event_loop()

fetch_1 = asyncio.ensure_future(aiohttp_get(loop=loop, url=test_1, auth=BAuth))
fetch_2 = asyncio.ensure_future(aiohttp_get(loop=loop, url=test_2, auth=BAuth))

group = asyncio.gather(fetch_1, fetch_2)

loop.run_until_complete(group)

[Chanonry]

Getting anywhere with this ???

[asvetlov]

The issue is on my radar. I'll try to find a time for fixing before aiogttp 3.0

[Chanonry]

This get fixed in 3.0 ??

[zied2]

I got the same problem.. Same fix to be done as curl did ? https://github.com/request/request/pull/1184

[Dreamsorcerer]

We drop auth headers when origin changes, so I assume this is fixed.