search

cybertk / abao

REST API automated testing tool based on RAML
MIT License
353 stars 59 forks source link

abao should understand security schemes #184

Open bpytlik opened 7 years ago

bpytlik commented 7 years ago

As part of our API, we have a security scheme we use for some endpoints. Since abao doesn't know about those security schemes, it doesn't have a way of testing them.

I've got a solution I'll make a pull request for shortly which basically appends the name of the security scheme to the route to differentiate it.

plroebuck commented 7 years ago

This basically reused the same methodology @ferrylauda used for testing the same endpoint several times. Potential conflicts?

Oh, wow. Now I remember where I had seen your name before...

Does this actually work? No OAuth expert or anything, but seems like there's a lot more to faking that type authentication than the PR demonstrates.

bpytlik commented 7 years ago

From my perspective, it's the test writers' job to do the job setting up the authentication. The purpose of this changeset is to allow the test writers' to write security scheme specific tests. At least in our case, we already have to do the security setup to test all of our routes anyway. What this lets us do is specifically test those response codes which are unique because of the security scheme in place. (In our case the response codes of 401 and 403.)

I don't know about potential conflicts with other work that's not in the gate yet. I've been happily using these changes in my own gate for many months now. Since we were resyncing with the gate tip, I thought I'd offer up this commit. If it's not welcome, that's fine, I'll continue to carry it along in my fork.

galkin commented 7 years ago

@bpytlik, I'm making code-review now. We should add examples to readme file, how set the auth credentials.

plroebuck commented 7 years ago

Potential conflicts are due to reuse of same method that @ferrylauda used to implement calling the same endpoint multiple times (appending string to the test name); aware this is in general usage in several branches. So how to determine whether security or just use-provided tag?

Assuming these issues can be addressed, would need target 0.5.0 branch (a.k.a., next release), not master.

bpytlik commented 7 years ago

(In reply to @galk-in)

As I mentioned previously, I think examples of how to set the auth credentials are out of scope for this work. That's for a couple of reasons: 1) If you're working with abao against a secured API, then you've already had to figure out how to secure your endpoints before now. 2) How to set the credentials is entirely dependent upon what auth scheme you're using. If you're using Basic Auth, you'd have one approach. If you're using OAuth, a different approach. I'm guessing OAuth2 would be third approach. The one we're using internally is different than all of those.

bpytlik commented 7 years ago

@plroebuck I don't know. This is all a hack to shoehorn things in for which abao is not really designed for. The right answer, imo, is to implement mocha-like suites. That gets you out of all of this name mangling. In absence of that, you could choose:

  1. position based approach - always put test count first, then the security scheme (or vis versa, it doesn't really matter as long as you're consistent)
  2. name based approach - rather than append (security_scheme_name) or (test_count), append test_id:3 security_scheme:BasicAuth. That makes them order independent, but means more work to parse the test names correctly.

You could imagine other crazy approaches like writing a grammar to express these and other additions to the language of test names. That said, I suspect these are the two simplest and easiest.