search

matryer / silk

Markdown based document-driven RESTful API testing.
http://silktest.org/
GNU General Public License v2.0
942 stars 69 forks source link

make previous data available with special format #2

Closed matryer closed 8 years ago

matryer commented 8 years ago

If I create a thing:

   ## POST /item

and it creates an ID for me:

     * Data.id = /.?/

I might need to be able to use that ID in another request later:

   ## GET /item/{id}
matryer commented 8 years ago

Is there a way to name groups in Regex?

matryer commented 8 years ago

or else, it could be done with comments:

* Data.id = /.?/ // this is the {id}
neocortical commented 8 years ago

The only open issue is the issue I want. That makes me happy. :)

matryer's use case is a good one. Mine is slightly different -- I'd like to bind variables from an authentication request (i.e. an auth token) to use in subsequent authed-only requests.

I like the last idea of using comments to bind variables.

butsjoh commented 8 years ago

Maybe consider using some sort of environment file as well (like postman does) which allows you to define global variables which you could use. I think there is a big difference in making a tool that checks data on an existing API (mostly GET and format testing) and one that is able to create an verify that created information (POST, PUT). Somehow you always need to setup an environment where you can run your tests against. I don't think this tool needs to be made so complex to support all of it, but that is up to the maintainer of course.

matryer commented 8 years ago

Yes, environment vars etc. as well as some way to capture data within tests is definitely needed. The hard part is figuring out how to make that look nice for documentation, as well as functional for the test itself.

mattdotmatt commented 8 years ago

Does this mean that the ordering of the tests become important? If test 3 relies on a value returned by test 1 then test 1 would need to be guaranteed to execute before test 3.

If so then the individual tests have become part of a bigger workflow which might need to be managed.

grounded042 commented 8 years ago

I also have use cases where this is needed. My initial thought is declaring the variable using something along the lines of response.jwttoken from GET /login or something of that sort. The first part is like nested json and could potentially mention more data such as header, request, status, etc. However, that is just the initial thought. There could be some problems with that such as what if you declare more than one GET /login?

timchunght commented 8 years ago

Same, my use case is pretty much what @matryer Mat described, the POST creates an object and returns the id and I want to use that id to test another GET endpoint. In addition, I think this feature will be useful to test an API with OAuth features or any other means of authentication like JWT. Hope this will be added soon. Thanks and good work!

matryer commented 8 years ago

Ok this is the next thing we should solve.

Let's all brainstorm implementation ideas.

The only rule of Silk club: It must look good as documentation and be readable to human.

I've definitely used {id} in docs before.

Mat

On 1 Mar 2016, at 12:58, timchunght notifications@github.com wrote:

Same, my use case is pretty much what @matryer Mat described, the POST creates an object and returns the id and I want to use that id to test another GET endpoint. In addition, I think this feature will be useful to test an API with OAuth features or any other means of authentication like JWT. Hope this will be added soon. Thanks and good work!

— Reply to this email directly or view it on GitHub.

timchunght commented 8 years ago

@matryer I think it will also be great to have the ability to group tests into a context since you are testing related features. In regards to markdown syntax, I propose the following three options:

Option 1:

* id = Data.id: /.?/

Option 2:

* var id = Data.id: /.?/

Option 3:

* {id} = Data.id: /.?/

and the variable name must match the variable name used in the following test

GET /objects/{id}

These are some of the most ideal syntax I am thinking about. Of course, the context is not completely necessary but I think it might be a good feature for v2. For v1, it will be nice to achieve just the variable reuse. Please feel free to tell me what you think. Thank you.

PS. I really hope we can get this feature done soon since I have some API tests to write and they all rely on each other. I am quite reluctant to write them in ginkgo but would really love to use Silk as both tests and documentation for the API.

Tim

Dashue commented 8 years ago

Wanted to add my thoughts to this as I would like to use this in the future, but without auth and variables, it's probably won't suffice for any real world apis.

I've used other api testing solutions, and most of them seem to not make a distinction between auth and using returned data. I think this is a mistake, indeed you can solve two features with the same implementation but from a consumer standpoint it leaves the user usually having to duplicate and convolute each test with auth setup.

I think a good approach would be to treat them as two different things. Seeing as auth is pretty given functionality for an api, tackling it in isolation would allow a more appropriate solution I believe.

What about making auth a first class concept, perhaps a new section, tag or separate markdown file entirely, with configuration to be either per test or per "test group" as @timchunght suggested. That way in the generated documentation it would be possible to output auth info once for all tests, at the start of a "test group" or per test. That way, providing a means to prevent duplication.

Looking forward to seeing Silk gain some traction!

matryer commented 8 years ago

Starting work on the data capture piece of this, I'm going with this:

* Data.id: /.?/ // The unique {id}

Mention of the {id} in the comment on the line will 'capture' the value as a variable. Then future mentions (outside of comments) will insert the value in place of the placeholder. Mentions of non-set variables will be an error.

Started in issue-2 branch.

matryer commented 8 years ago

Guys, please test the new variables feature: https://github.com/matryer/silk#capturing-data-beta

matryer commented 8 years ago

I wrote about this feature: https://medium.com/@matryer/new-variables-feature-for-silk-4c84ac6d3d83#.icqbqdqts

matryer commented 8 years ago

Did you guys get chance to test this feature? @Dashue @timchunght @grounded042 @mattdotmatt @butsjoh @neocortical

mattdotmatt commented 8 years ago

I'm using it to document/test an api I have written for tech tests. It's working great so far - only used {id} Will carry on tonight and push up my specs and share the repo.

mattdotmatt commented 8 years ago

Like it.

Works well for me in https://github.com/mattdotmatt/moodicle/tree/master/specs.

2-planets-post.md gets the id and checks the POST succeeded with a GET. 3-planets-delete.md then uses the same id and deletes it.

(NB: This implies a workflow e.g. POST needs to occur before DELETE. To ensure this ordering I've numbered the filenames so they start 1-, 2-, 3-)

timchunght commented 8 years ago

@matryer I REALLY REALLY LOVE IT SO FAR! I recently implemented the Parse.com API in Golang (I didn't like the nodejs version they open sourced as a result of their shutdown in 2017). https://parse.com/docs/rest/guide#objects-creating-objects.

I was actually going to go through the pain of writing all the code in ginkgo again but am really happy you updated the package.

Create Objects /classes/:objectName

POST /classes/HelloWorld

{"name":"Silk","status":"awesome","a_bool":true,"nothing":null,"release_year":2016}

Response

GET /users

GET /classes/HelloWorld/{objectId}

Response

Sample test md file. It is incredibly easy to write and read. Thanks for the good work!

matryer commented 8 years ago

Excellent. Thanks for the feedback. Let me know anything else you run into.

Sent from my iPhone

On 23 Apr 2016, at 08:18, Timothy Chung notifications@github.com wrote:

@matryer I REALLY REALLY LOVE IT SO FAR! I recently implemented the Parse.com API in Golang (I didn't like the nodejs version they open sourced as a result of their shutdown in 2017). https://parse.com/docs/rest/guide#objects-creating-objects.

I was actually going to go through the pain of writing all the code in ginkgo again but am really happy you updated the package.

Create Objects /classes/:objectName

POST /classes/HelloWorld

Content-Type: "application/json" X-Another-Header: "value" {"name":"Silk","status":"awesome","a_bool":true,"nothing":null,"release_year":2016} Response

Status: 201 Data.objectId: /(\S){10}/ // The object's unique {objectId}. GET /users

GET /classes/HelloWorld/{objectId}

Content-Type: "application/json" X-Another-Header: "value" Response

Status: 200 Data.objectId: {objectId} Sample test md file. It is incredibly easy to write and read.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

rafavisa commented 8 years ago

@matryer I'm trying to use the environment variables feature Does it work only from code (including it as os.Setenv("$AppNameFromEnv", "Silk")) or can we use it with the exec silk?

matryer commented 8 years ago

It should work with normal environment variables too. Haven’t test it, but let me know.

On 9 May 2016, at 14:48, rafavisa notifications@github.com wrote:

@matryer https://github.com/matryer I'm trying to use the environment variables feature Does it work only from code (including it as os.Setenv("$AppNameFromEnv", "Silk")) or can we use it with the exec silk?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/matryer/silk/issues/2#issuecomment-217868878

rafavisa commented 8 years ago

@matryer I’ve might done something wrong then

GET http://outlearn-hello.appspot.com/hello
  name: "{$NAME}"
Hello {$NAME}.

body expected:  
Hello Mat.  

actual:  
Hello {$NAME}.  
 hello.silk.md:9 - body doesn't match  
FAIL

Hello API

GET /hello

Gets a personalised greeting.

* ?name={$NAME} // The name of the person to greet

Hello Mat.
matryer commented 8 years ago

@rafavisa can you open a new issue as a bug with that exact issue? It seems like you've found a place where the {...} format isn't implemented.

Or better still, can you add a PR with that as a failing test.

rafavisa commented 8 years ago

Could the problem be running the test using the silk exe file instead of running it with run_test.go ?

It was my first guess as I've also tried using the environment variable within the request body instead of the request parameters (as you are doing in captured-vars.silk.md) and neither that worked for me.

matryer commented 8 years ago

Shouldn't matter re run_test.go vs the command. The {...} thing was implemented 'manually', so some places may have been missed.

rafavisa commented 8 years ago

@matryer I created a PR using environment vars in path, body and parameters https://github.com/matryer/silk/pull/42

PR passed checks so I don't really know if the issue is where the env var is used.