search

r-lib / gh

Minimalistic GitHub API client in R
https://gh.r-lib.org
Other
223 stars 51 forks source link

Test cases #8

Closed gaborcsardi closed 4 years ago

gaborcsardi commented 9 years ago

Kinda tricky, as I don't want to make a hundred requests for each test run..... but maybe a mixture of mocking and real test cases.

jennybc commented 9 years ago

What are you thinking re: tests that require a token and tests that will actually change things on GitHub?

realAkhmed commented 9 years ago

How about "personalized" tests?

I can see (at least) three progressive levels of these tests:

  1. Requires no tokens
  2. Requires a token but can be done with a free Github account (say, with a throwaway account)
  3. Requires a token and requires a paid account (working with organizations, private repos etc)
gaborcsardi commented 9 years ago

I think the throwaway account is a good way, if the GH policy allows it. It will still be somewhat tricky to perform the tests that DELETE, PUT, etc. in a safe way.

It would make sense to look at how other similar projects are doing the testing. E.g. the Python GH API, the JS one, etc.

jennybc commented 9 years ago

I'm not sure if this is the main Python wrapper (?) but it appears PyGithub's tests run as current user:

https://github.com/PyGithub/PyGithub/blob/master/Contributing.rst

According to the README, the library covers almost the full API and "all methods are tested against the real Github site".

It seems like there's two problems: having fixed entities to test against and having permission to do interesting things. They're related but not exactly the same.

Maybe it would make sense to have an organization that makes fixed repos available for testing. And to do full testing, someone would have to either have a paid account w/ sufficient privileges (can create a private repo, for example) or have a role within that organization that carries the necessary permissions.

gaborcsardi commented 9 years ago

@jennybc I would not worry about the private repos much at this point.

I don't mind creating a robot user or org to do the testing. I still have two issues:

gaborcsardi commented 9 years ago

I guess the credentials might be just encrypted variables on the CI, so problem solved....

jennybc commented 9 years ago

When you say "How not to expose the credentials to the robot org", you mean, like, encrypting the personal access token or OAuth token?

Yes I see what you mean about the slowness. Maybe you set it up so it's fairly easy to make the testing more or less thorough?

gaborcsardi commented 9 years ago

So, yes, @akhmed1 's "personal testing" is good, I think.

People can set up an org/repo to test on, and then they need to supply the proper tokens.

And we can just do the same on the CI, and our dev machines.

Pull requests won't be tested properly, but I don't think we can do much about that.

realAkhmed commented 9 years ago

Yep. This is what I meant by "personal testing": each user needs to supply their own test token for full-scale testing. The test suite may then build a test repo (under that user's account) and put it under different test conditions.

By the way, hundreds of API requests does not sound that bad. Github limits API requests to 5,000 per hour for authenticated requests.

gaborcsardi commented 9 years ago

Also see the recent discussion in #21.

gaborcsardi commented 8 years ago

FWIW, for another package, I used a declarative approach to "define" the API: https://github.com/gaborcsardi/webdriver/blob/master/R/endpoints.R https://github.com/mangothecat/webdriver/blob/master/R/endpoints.R

Obviously, the GH API is a lot bigger and more complex, but we could do something similar here.

gaborcsardi commented 8 years ago

Here is a good-looking GH API description: https://github.com/mikedeboer/node-github/blob/master/lib/routes.json

gaborcsardi commented 8 years ago

OK, some progress on this. Assuming this PR: https://github.com/hadley/httr/pull/409 will be merged, httr will support recording and replaying HTTP requests.

The httrmock package uses this: https://github.com/gaborcsardi/httrmock

There are already some test cases in gh.

Generating the tests from the node-github project's description does not look straightforward, but I'll have another look...

jennybc commented 8 years ago

I am happy to be enlisted in this @gaborcsardi if you want to give some direction. I am getting more and more time for such stuff as my teaching winds down. I have a lot of gh usage ahead of me as well in the coming weeks. But I realize it may be easier to do it yourself sometimes.

gaborcsardi commented 8 years ago

It would be great to work out a good workflow for API testing using httrmock (or one of Scott's packages later, when they are ready).

So if you have any comments on that, that would be great!

gaborcsardi commented 4 years ago

We can come back to this later, but there is nothing that we'll do here in the short term, so I'll just close this now.