Chunked artifact storage rebased

[nutbunnies]

@EnigmaCurry @aboudreault

This one is a biggie. It's the merger of 2 branches + some flamegraph for good measure.

See branches before looking here: https://github.com/datastax/cstar_perf/tree/progress_msg https://github.com/datastax/cstar_perf/tree/chunked_artifact_storage

Warnings

• There are 2 migrations that need to occur with this being merged
• A new python dependency on watchdog has been added

  Things to check when in staging environment

• old test artifacts are still viewable
• a fresh test with chunked artifacts are available
• progress message appears on task list page, task detail, and top of console log page

  Progress message

  note: only present when test is in progress -- hidden otherwise

[aboudreault]

Just pushed a fix for the unavailable artifact issue. Basicaly, we need to generate the the chunk object_id using all the primary keys of the table: test_id, type and name. It was only test_id and name, so the chunk upload was failing.

[aboudreault]

I'm going to test the progress_msg stuff, merge and deploy on staging.

[aboudreault]

Currently trying to get the progress_msg without luck.... I always have "None" in the UI, null in the json and the table 'progress_msg' attribute is effectively null also. Investigating ..

[nutbunnies]

Check the API auth settings. I had issues with it blocking due to auth failure locally until I fixed the keys. Turning on debug in the http stuff should give clues. On Nov 23, 2015 8:06 PM, "Alan Boudreault" notifications@github.com wrote:

Currently trying to get the progress_msg without luck.... I always have "None" in the UI, null in the json and the datastax progress_msg is effectively null also. Investigating ..

— Reply to this email directly or view it on GitHub https://github.com/datastax/cstar_perf/pull/137#issuecomment-159118721.

[aboudreault]

Thanks Andrew. I'll check that! On Nov 23, 2015 10:06 PM, "Andrew Hust" notifications@github.com wrote:

Check the API auth settings. I had issues with it blocking due to auth failure locally until I fixed the keys. Turning on debug in the http stuff should give clues. On Nov 23, 2015 8:06 PM, "Alan Boudreault" notifications@github.com wrote:

Currently trying to get the progress_msg without luck.... I always have "None" in the UI, null in the json and the datastax progress_msg is effectively null also. Investigating ..

— Reply to this email directly or view it on GitHub <https://github.com/datastax/cstar_perf/pull/137#issuecomment-159118721 .

— Reply to this email directly or view it on GitHub https://github.com/datastax/cstar_perf/pull/137#issuecomment-159138100.

[aboudreault]

Well, the problem is effectively the auth. The login succeed but we got a 401 on the progress update request.

[EnigmaCurry]

https://github.com/datastax/cstar_perf/blob/chunked_artifact_storage_rebased/frontend/cstar_perf/frontend/client/api_client.py#L77 is the login code, it uses the same key as the client does for connecting. Is the client able to connect otherwise?

[aboudreault]

this is where I'm looking.... client and the APIClient can connect successfully... so trying to understand that 401 on the other post request

[EnigmaCurry]

You will see a 401 before it logs in, if it's not logged in already, and then it will authenticate.

On Tue, Nov 24, 2015 at 9:40 AM, Alan Boudreault notifications@github.com wrote:

this is where I'm looking.... client and the APIClient can connect successfully... so trying to understand that 401 on the other post request

— Reply to this email directly or view it on GitHub https://github.com/datastax/cstar_perf/pull/137#issuecomment-159288033.

[aboudreault]

theoretically, yes.

DEBUG:requests.packages.urllib3.connectionpool:"POST /api/tests/progress/id/f8b1b5b6-92b5-11e5-9e71-0242ac110001 HTTP/1.1" 401 6400
send: 'GET /api/login HTTP/1.1\r\nHost: test_frontend_00:8000\r\nConnection: keep-alive\r\nCookie: session=.eJxVjU8LgjAchr9K_M5eAoMQupQu6M9k6lZ2GTqXmUPDbQcVv3sidOj68L7PMwIXunty09ayAW-EVQ4ekEQd8YBwvGaXiOKMDqmb-Kc4DlAYBsUhqomL6-3mptIdTA7k_SfTejGBZzorHVBtWcqCV80P2EZXZTOjTAg5j_-L8bAnWOE7VewcooJECekpwjhkjyvzEaPvV0BZ6dJBLEWrZcerYn4aqQ0XymojO5i-ZgZGiQ.CTYD-A.Si2r90uUl578AYhG0TVELBakc0k\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: python-requests/2.8.1\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Content-Length: 132
header: Set-Cookie: session=.eJxVjU8LgjAchr9K_M4dDApC6BC1BZUbuk3xNKbOPzg03CQy-u6Nbl0f3vd53iBLO9XSjb0eIHzDqoAQYm4uZMGEbdJ7IogSS77l5ytjCFOKqlPSx1vS73eZyQ_wWUPxeihrfyYI3TTrNZixaXQlOy-tlbGezIPtmsEzVZbar_-TEW_zTATPiMcBRRuS9ARnuD0zcwxEKhaKHzeRtooi8UvOVk-yq_zTaetkaWbr9ASfL4RQRjY.CTYD-A.zpI2IS1Qngb-R6VCp0ysZmG5eKA; HttpOnly; Path=/
header: Date: Tue, 24 Nov 2015 14:21:44 GMT
DEBUG:requests.packages.urllib3.connectionpool:"GET /api/login HTTP/1.1" 200 132
send: 'POST /api/login HTTP/1.1\r\nHost: test_frontend_00:8000\r\nContent-Length: 106\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: python-requests/2.8.1\r\nConnection: keep-alive\r\nCookie: session=.eJxVjU8LgjAchr9K_M4dDApC6BC1BZUbuk3xNKbOPzg03CQy-u6Nbl0f3vd53iBLO9XSjb0eIHzDqoAQYm4uZMGEbdJ7IogSS77l5ytjCFOKqlPSx1vS73eZyQ_wWUPxeihrfyYI3TTrNZixaXQlOy-tlbGezIPtmsEzVZbar_-TEW_zTATPiMcBRRuS9ARnuD0zcwxEKhaKHzeRtooi8UvOVk-yq_zTaetkaWbr9ASfL4RQRjY.CTYD-A.zpI2IS1Qngb-R6VCp0ysZmG5eKA\r\ncontent-type: application/json\r\n\r\n{"login": "test_cluster", "signature": "KFgJByDLlAtKhBvSjqMgj+O0ipBtQ4AsTauHChDo25myNuF4PrtvN9Hk85xJn5QM"}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Content-Length: 28
header: Set-Cookie: session=.eJxVjU8LgjAchr9K_M4eFAxC6BDpgsoN_0zxNKYuFYeKm0SK3z0ROnR9eN_nWYAVanwx3beiA2eBQw4OBLG84RnhyEqeIcWczpkdu_co8hAhXnkN28DG7emYyuwMqwH5Z-BK7SZw9DgJA2RfVaJkTfcDU6eaqtsQLwqxjf-LflxnKTXffhyYxLNw2GKUotqN5MWkCZ0JGh40qTnx6F6clBhZU25PLZRmhZyUFiOsX0M0Res.CTYD-A.7U_2JSYgUfjYFOV3edqH7Nj_5Xk; HttpOnly; Path=/
header: Date: Tue, 24 Nov 2015 14:21:44 GMT
DEBUG:requests.packages.urllib3.connectionpool:"POST /api/login HTTP/1.1" 200 28
send: 'POST /api/tests/progress/id/f8b1b5b6-92b5-11e5-9e71-0242ac110001 HTTP/1.1\r\nHost: test_frontend_00:8000\r\nContent-Length: 122\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: python-requests/2.8.1\r\nConnection: keep-alive\r\nCookie: session=.eJxVjU8LgjAchr9K_M4eFAxC6BDpgsoN_0zxNKYuFYeKm0SK3z0ROnR9eN_nWYAVanwx3beiA2eBQw4OBLG84RnhyEqeIcWczpkdu_co8hAhXnkN28DG7emYyuwMqwH5Z-BK7SZw9DgJA2RfVaJkTfcDU6eaqtsQLwqxjf-LflxnKTXffhyYxLNw2GKUotqN5MWkCZ0JGh40qTnx6F6clBhZU25PLZRmhZyUFiOsX0M0Res.CTYD-A.7U_2JSYgUfjYFOV3edqH7Nj_5Xk\r\ncontent-type: application/json\r\n\r\n{"progress_msg": "Last Op Completed: apache/cassandra-3.0:stress, finished 1 of 1 total ops (2015-11-24 09:21:44.338378)"}'
reply: 'HTTP/1.1 401 UNAUTHORIZED\r\n'
header: Content-Type: text/html; charset=utf-8
header: Content-Length: 6400
header: Set-Cookie: session=.eJxVjU8LgjAchr9K_M4eFAxC6BDpgsoN_0zxNKYuFYeKm0SK3z0ROnR9eN_nWYAVanwx3beiA2eBQw4OBLG84RnhyEqeIcWczpkdu_co8hAhXnkN28DG7emYyuwMqwH5Z-BK7SZw9DgJA2RfVaJkTfcDU6eaqtsQLwqxjf-LflxnKTXffhyYxLNw2GKUotqN5MWkCZ0JGh40qTnx6F6clBhZU25PLZRmhZyUFiOsX0M0Res.CTYD-A.7U_2JSYgUfjYFOV3edqH7Nj_5Xk; HttpOnly; Path=/
header: Date: Tue, 24 Nov 2015 14:21:44 GMT
DEBUG:requests.packages.urllib3.connectionpool:"POST /api/tests/progress/id/f8b1b5b6-92b5-11e5-9e71-0242ac110001 HTTP/1.1" 401 6400
Exception in thread Thread-6:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/home/aboudreault/git/cstar/cstar_perf/venv/local/lib/python2.7/site-packages/watchdog/observers/api.py", line 199, in run
    self.dispatch_events(self.event_queue, self.timeout)
  File "/home/aboudreault/git/cstar/cstar_perf/venv/local/lib/python2.7/site-packages/watchdog/observers/api.py", line 368, in dispatch_events
    handler.dispatch(event)
  File "/home/aboudreault/git/cstar/cstar_perf/venv/local/lib/python2.7/site-packages/watchdog/events.py", line 537, in dispatch
    _method_map[event_type](event)
  File "/home/aboudreault/git/cstar/cstar_perf/frontend/cstar_perf/frontend/client/client.py", line 664, in on_created
    self.__tell_server(event)
  File "/home/aboudreault/git/cstar/cstar_perf/frontend/cstar_perf/frontend/client/client.py", line 676, in __tell_server
    api_client.post('/tests/progress/id/{}'.format(self._job['test_id']), data=json.dumps({'progress_msg': msg}))
  File "/home/aboudreault/git/cstar/cstar_perf/frontend/cstar_perf/frontend/client/api_client.py", line 51, in post
    raise RuntimeError(error)
RuntimeError: Request failed to /tests/progress/id/f8b1b5b6-92b5-11e5-9e71-0242ac110001 - <Response [401]> <!DOCTYPE html>

[aboudreault]

more convenient to read: https://gist.githubusercontent.com/aboudreault/c79104bdb5411b7dffe6/raw/21b06b0d984395f35fc4ad2fa3b8c0ccd3df606f/gistfile1.txt

[aboudreault]

I think we'll merge and test on staging if my local setup is the problem. (although everything else works)

[aboudreault]

Ok, we found the issue. 'test_cluster' was missing in the users table, so the role check was failing. I will have to check on cstar if the users table contains all clusters. I will merge this PR. Thanks Andrew!!