search

eclipsesource / restfuse

A JUnit extension for automated HTTP/REST Tests
http://developer.eclipsesource.com/restfuse/
91 stars 30 forks source link

Is restfuse dead? #63

Open mrkdev opened 9 years ago

mrkdev commented 9 years ago

Please tell us what you plan to do with restfuse and where it will go. As of now, there seams to be no current commit nighter some other work happening. Also there are multiple long pending pull requests in this repo. Is restfuse dead or still alive? It is an interesting project and if it's still alive, I'm interested to contribute and add some new features.

kamal2222ahmed commented 9 years ago

i think we (the contributors) should pickup this where it left off. This is ONE of a KIND project, and if you try to look for a REST testing framework, you would not find ANY that is reasonably good and Free

bbwinter commented 9 years ago

HTML Unit.  I lost patience with Restfuse as it was buggy and didn't do everything it said it should.  html unit has been very nice so far.  

 On Friday, December 19, 2014 7:28 AM, kamal2222ahmed <notifications@github.com> wrote:

i think we (the contributors) should pickup this where it left off. This is ONE of a KIND project, and if you try to look for a REST testing framework, you would not find ANY that is reasonably good and Free— Reply to this email directly or view it on GitHub.

smoyer64 commented 9 years ago

I looked at Restfuse, saw comments similar to those in this thread and decided to use RestAssured instead. I like the programming model of Restfuse ... it's so much better than the (Ruby like?) DSL you use in RestAssured. I do like that the DSL includes methods to test the response contents including parsing JSON structures (I can add JSONUnit to Restfuse and get the same functionality).

In any case, I'd be happy to pitch in and help with the Restfuse project IF the current maintainer will let someone take it over.

kamal2222ahmed commented 9 years ago

Sent an email to Holger Staudacher , asking his permission, but i personally would think it would be ok to start at least fixing the build and work on the bugs. As for RestAssured, i did look at it while evaluating a REST testing framework, and it has the disadvantage of Groovy being used, which is yet another language to learn and increases the pre-requisite knowledge in order to use the framework

kamal2222ahmed commented 9 years ago

I have started working on the bugs , and once i get a bit deeper understanding of restfuse, i will perhaps contribute to enhancements. My question is: what is the best way to maintain restfuse.

  1. Should we fork it, and THEN Enhance it
  2. Should we maintain THIS repo also, what is the process of fixing bugs, enhancements , etc since the owner is not available. Here is what i found as an example:

how to use GitHub effectively for your project

GitHub provide an awesome set of tools for opensource developers. For Dancer, we use them as much as possible. I'll show and explain how we do our development.

code review

Dancer's development goes fast. We do our best to ship often, we do a lot of refactoring, and we listen our users. This means processing pull request and issues as fast as possible. pull request

There is five developers with write access to the main repository. Each one is asked to do a review of pending pull request every morning (this is not required, neither enforced, it's a "if you have ten minutes in the morning while drinking your cofee, please, review the PR").

When we're reviewing something, most of the time we pull the submited patches into a branch named review/username. If the reviewer is happy with the modifications, he will add a comment to the pull request: "approved" (and some other comments if it's required). In case he's not happy, he will comment disaproved and give his reasons (tests doesn't passes, or there is no tests for the patch provided, or this is not something we want, etc). If the PR is about something the developer doesn't really knows, or has a doubt, he skip this one, and ask for someone else to take a look.

In order to merge the branch review/username, we need two developers to comment approved. Of course, for something simple, like a documentation fix, or changing a few line somewhere, we can merge this without applying this process.

As the work consists to read the code and comment, it's quiete easy to handle two/three pull request each day for a developer. When one of the developper with access to the repository find some time, he can go through the pull request, and merge the one marqued as approved, since he knows that the people who had already approved them understand the code. issues

We don't use RT to manage our issues. It's not that RT is bad or that we don't like it, it's just that GitHub's issues are more integrated with our workflow. GitHub's issues are really simple (one can even say naive), but most of the time it's ok for our usage. We don't need advanced features like "track how much time you've spend on this ticket" (even if I do track my time spent on Dancer, using orgmode).

One of the nice feature of GitHub's issues, is that you can close them with a commit. If the commit's message looks like 'closes GH-123', the issue 123 will be closed, with a link to the commit in the comment (take a look). I find this feature really useful, since when refering to a closed issue, you can find the commit inside the ticket.

Once or twice a week, we try to proceed a triage, where we go through the issues, and tag them.

When someone report something on the mailing list or on irc, we ask them if they can open an issue, since it's easier for us to track the request.

An issue doesn't need to be about a bug, it could be:

a feature request (I want to do x or y with dancer)
something that need to be refactored
an issue reported by an user that need feedback from the developers

commenting on code

Another nice feature is the possibility to comment the code. Most of the time you'll do it while reviewing a pull request. But a user can also comment on something.

Sometimes we push a branch that need some feedback, and a discussion will be started. comparing branches

You can easily diff two branches. With this url you can do a quick diff of the changes between master and devel, see the list of commits, and which files have changed.

This is usefull when you want to have an overview of the work. gitflow

We're using gitflow to work. Gitflow is a nice tool that help you creating and merging branches. If you don't know gitflow, there is a good article that explain the reasons behind it.

Ok, this has nothing to do with GitHub, but I'll explain quickly what we do with it.

an article on gitflow: http://scottchacon.com/2011/08/31/github-flow.html

We use the following conventions:

master: only for release
devel: this is the development branch. This one should always work (tests can't be broken)
topic/$name: when we start to develop a new feature, we create a topic branch
hotfix/$name: when we need to release a new version right now, without merging devel before, we create a hofrix branch
release/$version: when we're ready to ship, we create a new branch

It's very rare that we need to push a hotfix or release branch to GitHub: thoses branches had a really short life span. But topic branches can be pushed, when we want feedback from users or other developers. future

We're already using jitterbug to do some continuous integration. When we push to GitHub, a payload is posted to jitterbug, and a build is triggered. In a near future, we will use git's notes with jitterbug. The idea is to store inside a note, for each build, the duration of the build, and the result of the build (failure / success) and maybe the TAP output in case of a failure. This will allow developers to see directly in the logs the status. Of course, GiHub already display them, so this will be very useful for all of us.

lumberjaph.net is © 2008 – 2014 Franck Cuny // @franckcuny

references:

  1. http://code.tutsplus.com/articles/team-collaboration-with-github--net-29876
  2. https://gun.io/blog/how-to-github-fork-branch-and-pull-request/
GGBeer commented 9 years ago

+1 for option "1. Fork it, and THEN Enhance it." Optional creating a new repo "RestFuse-NG" would give us a clean start.

smoyer64 commented 9 years ago

@GGBeer I came here to play devil's advocate to your comment (+1 for option "1. Fork it, and THEN Enhance it."). Now that I've pulled the code and attempted to build and test it in a few different configurations, my opinion has changed dramatically. To be fair to the discussion, I'll include the pros I was originally going to promote:

I'll admit I'm a fan of more minimalist projects and therefore not a fan of some of the (what I consider) Maven bastardizations. The main point of Maven is "convention over configuration" and I think this project is a great example of what NOT to do:

I haven't tried analyzing the project with Sonar yet, but I'll give my impressions ... I think the code quality is actually pretty good, but there's an almost complete lack of comments in the classes I looked at.

I spent a couple more hours carefully merging the existing pull requests into restfuse and while I'm not sure I understand all of the Tycho related cruft, it does seem to build the library. One integration test is failing as noted above. You can find the (hopefully temporary) project here:

https://github.com/selesy/restfuse

and a JAR file containing the results of my work here:

https://github.com/selesy/restfuse/releases/tag/v1.2.1-SNAPSHOT

For the most part, the example code works ... there are two TestCases with two tests each that fail:

If the group is so inclined, I'd be more than happy to delete the unnecessary build tools and set the project up so that it's built in a conventional Maven fashion. The only issues I have with doing so is whether we'd conform to the EPL license with a dramatically restructured project, and whether it's fair to the original developer(s). It's entirely possible either don't understand what they were trying to accomplish with OSGI/Tycho or they were working towards a bigger goal that isn't visible to the group.

For posterity, here's the process I used to create v1.2.1-SNAPSHOT:

  1. Forked the restfuse project.
  2. Tagged the fork to preserve a starting point.
  3. Pulled and merged a pull request into the project.
  4. Built the library from the root of the project with: mvn install -f com.eclipsesource.restfuse/pom.xml
  5. Ran the tests from the root of the project with: mvn test -f com.eclipsesource.restfuse.test/pom.xml
  6. Tagged the project with a checkpoint.
  7. Repeat from step three for each pull request.
  8. Ran the example tests (in Eclipse since there's no pom.xml in that project).
  9. Updated the project version to match the new OSGI bundle version.
  10. Pushed the project back up to Github.
  11. Created a manual release (uploading the created JAR.

Sorry for posting such a long comment ... where should we go from here?

GGBeer commented 9 years ago

Had the same problems with the directory structure and build tools. As I am currently not using Eclipse and are not that familiar with OSGi/Tycho/etc. I just created a new local maven pom.xml with a standard directory structure to get going, but copied my changes back into the original dir structure before commiting to my fork.

Your pros are definitely true and we should respect the original project developers work and efforts. Also don't know if the EPL license model is still OK or should be changed. Anyway the fork you made is certainly a good way to get things going and bring restfuse back to live again.

smoyer64 commented 9 years ago

I have confirmed that the DynamicPathTest.java tests failed due to changes in search.maven.org and have modified the tests to expect a 302 HTTP status code (the latest is at: https://github.com/selesy/restfuse). I'm tempted to make a branch with the project moved into a conventional Maven project ... I think a parent project with one child module for the library and another for the examples would be enough.

The test module could be moved into the library project under src/it/java and run with the maven-failsafe-plugin. I can provide CI and analysis via Sonar but I think before we get too far along we should make some decisions about project name (if it needs to change), how to update the documentation (especially developer documentation), etc.

yurgisbaykshtis commented 9 years ago

Thanks smoyer64 for taking over this great project. Any plans to update an artifact in maven repo?

smoyer64 commented 9 years ago

I started working on simplifying the project and believe it's back to working the way it did before. I was going to convert the issues in the original repository and start fixing them to (I've already applied the pull requests).

I was waiting to hear what the group wanted to do long-term ... I feel as though I've stolen the project in a way but my intent was just to move it forward a bit. What does the community want?

In any case, I'll be generating releases as I go but I'm not sure I can get them into Central under the original GAV. Does anyone have a contact at EclipseSource? How can we get an official blessing and take over the web-site, etc?

On Fri, Feb 6, 2015 at 5:55 PM, yurgisbaykshtis notifications@github.com wrote:

Thanks smoyer64 for taking over this great project. Any plans to update an artifact in maven repo?

— Reply to this email directly or view it on GitHub https://github.com/eclipsesource/restfuse/issues/63#issuecomment-73328576 .

Sent from my iBerry MacPhone

kamal2222ahmed commented 9 years ago

smoyer64, The name would depend on what are we going with restfuse . Here are some which we can ammend, or augment.

  1. Generates test run reports in JUnit XML format. Which is compaitlble with most CI servers
  2. Provides benchmarking
  3. Orchestration
  4. Extensible
  5. supports POST, GET, PUT, DELETE, OPTIONS, PATCH and HEAD
  6. Support Response types, JSON, XML , HTML etc
  7. Coverage Reporting
drejc commented 9 years ago

Nice to hear that RestFuse is moving again. What I personally miss the most is a way to perform scenario tests. I have submitted a patch to add an Order property to the test, forcing RestFuse to execute tests in a certain order. This solves some of the scenario issues but not all of them.

An addition pain is the use of dynamically generated paths. It can be achieved but is very cumbersome. Especially if you want the next REST call to use the return value of a previous one (scenarios again).

Just to illustrate I will give you an example what I'm currently testing with REST fuse:

In this scenario the login call creates a cookie (session), which must be than provided on the next calls an so on.

Currently the code is way to complicated for what it does. Just a simple history stack with return values of all calls in a scenario would reduce my code in half.

smoyer64 commented 9 years ago

Unit tests are supposed to be idempotent ... the changes you're describing sound more like a Selenium test (which we've used to do UI testing) and I can see a place for them testing RESTful APIs as well.

REST isn't supposed to rely on state being managed on the server and in the case you've shown above, I would tend to make a parameterized test that tested without any credentials, with the wrong credentials and with the correct credentials. Using annotations to define these are, as you've noted, somewhere between impossible and cumbersome.

I think testing security is a very common use case and can be broadly applied (having a Cookie is roughly equivalent to having a Bearer-Token in a Header). So the real question is what's the best way to allow programmatic configuration of the call. At some point, what if the @HttpTest annotation was empty and you had to trigger the call within the test?

I've used RestAssured and it's a lot more like what you describe ... the downside is that it's all programmatic (and based on a Groovy DSL).

This should probably be a separate conversation ... I think there's a lot to discuss!

On Sun, Feb 8, 2015 at 6:42 AM, drejc notifications@github.com wrote:

Nice to hear that RestFuse is moving again. What I personally miss the most is a way to perform scenario tests. I have submitted a patch to add an Order property to the test, forcing RestFuse to execute tests in a certain order. This solves some of the scenario issues but not all of them.

An addition pain is the use of dynamically generated paths. It can be achieved but is very cumbersome. Especially if you want the next REST call to use the return value of a previous one (scenarios again).

Just to illustrate I will give you an example what I'm currently testing with REST fuse:

  • create account
  • login with wrong credentials
  • login with correct credentials
  • make call to a restricted REST interface
  • log out

In this scenario the login call creates a cookie (session), which must be than provided on the next calls an so on.

Currently the code is way to complicated for what it does. Just a simple history stack with return values of all calls in a scenario would reduce my code in half.

— Reply to this email directly or view it on GitHub https://github.com/eclipsesource/restfuse/issues/63#issuecomment-73407392 .

Sent from my iBerry MacPhone

yurgisbaykshtis commented 9 years ago

Starting from junit 11 you can execute tests in name order. What we do is just prefixing names with test###_ (like test002_userPut). That works for us with restfuse. The only (minor) problem is that you can't mix @HttpTest and @Test with the latter ones always being executed after the former.

On Feb 8, 2015, at 3:42 AM, drejc notifications@github.com<mailto:notifications@github.com> wrote:

Nice to hear that RestFuse is moving again. What I personally miss the most is a way to perform scenario tests. I have submitted a patch to add an Order property to the test, forcing RestFuse to execute tests in a certain order. This solves some of the scenario issues but not all of them.

An addition pain is the use of dynamically generated paths. It can be achieved but is very cumbersome. Especially if you want the next REST call to use the return value of a previous one (scenarios again).

Just to illustrate I will give you an example what I'm currently testing with REST fuse:

In this scenario the login call creates a cookie (session), which must be than provided on the next calls an so on.

Currently the code is way to complicated for what it does. Just a simple history stack with return values of all calls in a scenario would reduce my code in half.

— Reply to this email directly or view it on GitHubhttps://github.com/eclipsesource/restfuse/issues/63#issuecomment-73407392.

smoyer64 commented 9 years ago

Maybe this is one of those cases where it's best to work backwards ... could you provide a complete test class that completes the test you describe above and is structured the way you envision it? I realize it won't run on the current Restfuse but it's a great way to discuss how we'd accomplish such a test.

On Sun, Feb 8, 2015 at 12:48 PM, yurgisbaykshtis notifications@github.com wrote:

Starting from junit 11 you can execute tests in name order. What we do is just prefixing names with test###_ (like test002_userPut). That works for us with restfuse. The only (minor) problem is that you can't mix @HttpTest and @Test with the latter ones always being executed after the former.

On Feb 8, 2015, at 3:42 AM, drejc <notifications@github.com<mailto: notifications@github.com>> wrote:

Nice to hear that RestFuse is moving again. What I personally miss the most is a way to perform scenario tests. I have submitted a patch to add an Order property to the test, forcing RestFuse to execute tests in a certain order. This solves some of the scenario issues but not all of them.

An addition pain is the use of dynamically generated paths. It can be achieved but is very cumbersome. Especially if you want the next REST call to use the return value of a previous one (scenarios again).

Just to illustrate I will give you an example what I'm currently testing with REST fuse:

  • create account
  • login with wrong credentials
  • login with correct credentials
  • make call to a restricted REST interface
  • log out

In this scenario the login call creates a cookie (session), which must be than provided on the next calls an so on.

Currently the code is way to complicated for what it does. Just a simple history stack with return values of all calls in a scenario would reduce my code in half.

— Reply to this email directly or view it on GitHub< https://github.com/eclipsesource/restfuse/issues/63#issuecomment-73407392

.

— Reply to this email directly or view it on GitHub https://github.com/eclipsesource/restfuse/issues/63#issuecomment-73422010 .

Sent from my iBerry MacPhone

GGBeer commented 9 years ago

Currently I am using RestFuse to test my JSON-based Webservices. For testing a specific WS-Function I have to make sure that all objects which are required for executing that function have been generated/added in advance and the TestCase must use/reference the specific UUIDs associated to these (temporary) objects. Also some parameters of the testcases must be generated on TC-execution e.g. include timestamps, provide unique object names, etc. The ordered testcase execution and DynamicBody features can be used for this but are difficult to use in large test scenarios.

For that reason I extended RestFuse to have full control over the HTTP headers and body just before the message is sent out via HttpRequest. At that point also access to the HttpTest-Annotations was important to me for defining further test specifc parameters/details (e.g. the fields containing the UUIDs to be used). Via reflection an instance of a ProcessRequest-class (definable by an HttpTest annotation parameter) is created in the HttpStatement.evaluate() phase. This ProcessRequest class can do just anything to/with the HTTP contents at that point. Also via reflection the test specifc parameters are retrieved by their field getters. Of course this kind of RestFuse usage is not an idempotent JUnit test anymore (but solves my testcase writing problems).

My restfuse changes are far from perfect and have lots of room for improvement, but maybe they are usefull to the community.

Thx, Guus

drejc commented 9 years ago

Ok ... here is an example ... of a simple test of mine ... just for illustration purposes. The destination is dynamically created on each call ... once the cookie is filled (with previous call) the cookie is used:

@Rule
public Destination destination = getDestination();

// provide dynamic destination according to stored cookie
private static String authCookie;
private Destination getDestination() {
        Destination destination = new Destination( this, TestSettings.HOST);

        if (authCookie != null)
            destination.getRequestContext().addHeader("Cookie", authCookie);
        return destination;
    }

@HttpTest(method = Method.POST,
              path = "/rest/public/login",
              content = LOGIN_JSON,
              type = MediaType.APPLICATION_JSON,
              order = 20)
    public void login() {

        // login user
        assertEquals(HttpStatus.SC_OK, response.getStatus());
            List<String> cookie = response.getHeaders().get("Set-Cookie");
                // cookie is store in static variable ...
        authCookie = cookie.get(0);
    }

// not public ...
@HttpTest(method = Method.GET,
              path = "/rest/user/info",
              order = 30)
    public void getInfo() {
        assertEquals(HttpStatus.SC_OK, response.getStatus());
        String body = response.getBody();
        ... assert response ...
    }

Current disadvantage is that @path, @content and @authentication can not be dynamically generated so you are kind of limited with the current implementation.

While the static annotation principle is nice for simple tests ... the dynamic aspect should also be considered ... for instance a call to GET /test/user/info/{userId} is not possible at the moment.