E2E Tests

[kahboom]

Create E2E tests for majority of components

[jludvice]

@kahboom It looks like I have working configuration of cucumber. Commits and some notes are related to issue #190.

Current layout looks like this and it's open to discussion :)

ipaas-ui/features
├── data
│   ├── users.example.json
│   ├── users.json
│   └── user.ts
├── env.js
├── login.feature
├── pages
│   ├── app.ts
│   ├── connections.ts
│   └── login.ts
├── Readme.md
└── step_definitions
    └── login.steps.ts

users.json is "mapping" between user alias used in *.feature specs and actual login credentials (eg. my github login/password)

Current state is in my fork/branch cucumber https://github.com/jludvice/ipaas-ui/tree/cucumber/features

[kahboom]

@jludvice I like it. We can also use something like config to determine the environment and user credentials, if you prefer. But the current setup for that is perfectly fine.

As for the directory layout, I think it could be improved a bit. The biggest thing that stands out to me is that all common components are not bundled into a single directory the way that the application is structured. I think something like this would be more intuitive for new collaborators, including renaming the directory to e2e or tests:

ipaas-ui/e2e
├── fixtures (or data, as you have it, is fine)
│   ├── users.example.json
│   ├── users.json
│   └── user.ts
├── common
│   ├── modal.ts
│   ├── toast.notification.ts
│   ├── login.steps.ts
│   └── logout.steps.ts
├── login
│   ├── login.ts // Page, which calls the common steps
│   └── logout.ts
├── dashboard
│   ├── dashboard.ts
│   └── dashboard.empty.ts
├── connections
│   ├── list.ts
│   ├── list.empty.ts
│   └── create.ts
├── env.js
└── README.md

Are you logging in before each test? We should also be testing that users must be logged in for certain restricted routes (when fully implemented in the UI).

[kahboom]

And thank you @jludvice !

[jludvice]

Regarding layout - sure it can be moved to e2e. I started with features because cucumber.js looked there by default. Can new layout look like this?

ipaas-ui/e2e
├── fixtures (or data, as you have it, is fine)
│   ├── users.example.json
│   ├── users.json
│   └── user.ts
├── connections
│   ├── connections.feature     # gherkin feature file
│   ├── connections.steps.ts    # mapping from feature -> protractor
│   ├── connections.specs.ts   # in case we wan't to mix also jasmine
│   └── connections.po.ts        # page objects

Regarting *.feature files and their "coverage". I thought that main benefit of using cucumber / gherking syntax would be to have 1 to 1 mapping between feature spec and our sprint narative.

Assuming we have narative Camilla attempts to do real integration with the iPaaS we will have feature CamilaRealIntegration.feature

Rough draft might look something like this:

Feature: Camilla attempts to do real integration with the iPaaS

#  inspired by zoran regvart's demo project
#  https://github.com/zregvart/bdd-example/blob/master/src/test/resources/com/github/zregvart/bdd/example/Cameleers.feature

  Background:
    Given integration "push" resource properties:
      | url             | name            | username | password  |
      | http://asdfasdf | my awesome push | user1    | password1 |

    And integration "pull" resource properties:
      | url            | name            | username | password  |
      | http://eeeffaa | my awesome pull | user2    | password2 |

  Scenario: create http connections
    When I navigate to "Connections" page
    And I click on "Create connection" button
    And I fill "pull" connection properties
    And I click on "Save connection" button
    Then I'm presented with "pull" connection on dashboard

  Scenario create integration
    When I navigate to "Integrations" page
    And I click "Create integration" button
    Then I'm presented with available connections:
      | name            |
      | my awesome push |
      | my awesome pull |

• if we first write this feature file, then developers know how the user flow is defined and as a bonus I have test scenario defined, right ? But where does this user flow belong to, which component directory?

[jludvice]

Regarding logging in - now it's really tricky and challenging task to me, because Jimmi keeps changing stuff around and I'm not able to run keycloak locally with their development single tenant template due to some bug in openshfit (https://github.com/openshift/origin/issues/13197).

Future wise - I'm counting on having some test to check "public parts" of ipaas and test login, and I'd like to be able to inject refresh_token or whatever into browser session storage instead of filling login form again in every feature.

But injecting this stuff will be highly dependent on environment tests are running in imho...

[jimmidyson]

because Jimmi keeps changing stuff around

Apologies for adding necessary functionality...

[kahboom]

@jludvice Yes! That's even better, actually. Sorry, I didn't include an example of a *.feature file, but I would have put it in the same place you did in your example. I agree they are very useful when mapping to the sprint narrative, so that's a very good mindset to have.

Also, I don't mind if we leave it as features instead of e2e, but at first glance I didn't know what it was, as I'm new to Cucumber. It's up to you.

Regarding the user flow, we can have a separate users directory (eventually I believe we will have settings and things like that). I think we will need one eventually in the app. The purpose of the common dir is that there are some things you'll want to re-use, just as we are doing within the app itself, like toast notifications, modals, etc. but there may not be enough to warrant a dedicated common directory. I'll leave that up to you to decide. You can always do this later as well if you find it's useful or that you are re-using a lot of code. It depends on how often you find yourself writing the same code over and over again. For instance, is it easy for you to call the 'login' or 'logout' steps before certain tests to see if the authorization feature (we will eventually implement, not now) is working properly?

It may be a good idea for you to see what the long-term plans for iPaaS were, so you can plan in advance. Here are several good resources (notably the 'Planning' and 'Workflows and Requirements' sections).

[kahboom]

Apologies for adding necessary functionality...

ROFL

[kahboom]

Regarding logging in - now it's really tricky and challenging task to me, because Jimmi keeps changing stuff around and I'm not able to run keycloak locally with their development single tenant template due to some bug in openshfit (openshift/origin#13197).

@jimmidyson is always breaking things and subsequently fixing them. We just yell at him on IRC. ;D

Yeah things are still going to be changing around quite a bit, as Jimmi just started architecting the project in January and had a huge undertaking of re-structuring a lot of moving parts so that they actually function properly. I'm honestly surprised things have worked out as well as they have, considering the task at hand. The constant changes are actually major improvements disguised as pains in the ass.

Unfortunately, the user session handling and staging setup get affected quite frequently especially in the beginning stages, but as long as we communicate we can all adjust accordingly. If it makes you feel better, we had the same problem with the UI (refer to 'yelling' comment above for advice on how to deal with it).

Future wise - I'm counting on having some test to check "public parts" of ipaas and test login, and I'd like to be able to inject refresh_token or whatever into browser session storage instead of filling login form again in every feature.

That would probably be the most efficient, but I'm going to assume that much of it won't be public eventually. We'll have to see, but we can cross that bridge when we get there.

But injecting this stuff will be highly dependent on environment tests are running in imho...

Yep, E2E can be fun until you start having to work with multiple environments, as is usually the case for the most accurate E2E testing. Unfortunately, it comes with the territory.

[jludvice]

Hi @kahboom. Could you please suggest any alternative for console.log in e2e tests ? I'm looking for alternative of log4j in java

private static final Logger log = LoggerFactory.getLogger(ISDSComponentTest.class);
// ...
log.info("whatever i want to from obj {}", this.obj);
log.warn("whatever i want to from obj {}", this.obj);

I'd like to make yarn lint happy :)

e2e/login/login.po.ts[69, 7]: Calls to 'console.log' are not allowed.

Looks like there are some logging libraries for angular, but does it make sense for e2e tests? https://libraries.io/search?keywords=log4j&languages=TypeScript

Thanks :)

[jludvice]

just for the reference, this is e2e/cucumber-reports/cucumber-report.html

The feature for sprint 8 narrative is just draft. I'd like to start implementing it tomorrow

[jludvice]

And I'm sorry for this commit mess. I was rebasing my cucumber branch on master with #209 resolved.

[kahboom]

@jludvice - Looks great. We are using an internal logger that @gashcrumb worked on here, which you can use like this. If you decide to use another logger, that's fine too. There are a few around, like angular2-logger or any really, but yeah we already have one we are using.

Mr. Lint is difficult to keep happy. :)

[jludvice]

found #245 along the way. We need to align that to avoid complications in searching elements in tests.

[jludvice]

For the reference, QE would like to investigate this rest resource https://github.com/redhat-ipaas/ipaas-rest/blob/master/jsondb/src/main/java/com/redhat/ipaas/jsondb/rest/JsonDBResource.java

reason ? We need a way to restore IPaaS state between scenarious. Ideally we'd pass state data in Given section in *.feature and use this JsonDB resource to "inject" state into IPaaS REST. Great question is whether use this resource to populate/drop-create for instance this

• connections list
• integrations list

@jimmidyson Can we use json db this way?

(CC @dsimansk @lvydra )

EDIT JsonDBResource might go away any time. We might wanna implement test resouce in ipaas-rest for resetting the state via rest. https://github.com/redhat-ipaas/ipaas-rest/issues/177

[jludvice]

status update: current state of narrative 8 http://htmlpreview.github.io/?https://gist.githubusercontent.com/jludvice/95f07304decacc0406742bb5c0377d93/raw/6769811b5c347123cbfc42cdd459c13465bf2bc6/cucumber-report.html

[jludvice]

Tests were failing because there are multiple connection with same name on staging. Tried to start ipaas with oc cluster up and found this issue with ipaas template https://github.com/redhat-ipaas/openshift-templates/issues/19 edit - issue resolved

[jludvice]

Opened issue & pr in openshift templates to workaround issues with ssl certificates in dev mode https://github.com/redhat-ipaas/openshift-templates/issues/20

[jludvice]

Hello @gashcrumb @kahboom. I just opened PR #308 to merge current state of e2e tests. Here is definitelly lot to fix and improve. You can expect more pull requests on regular basis :) It's failing on #295 currently. Resulting test report can be found here and in e2e/cucumber-reports/cucumber-report.html after every test run.

[jludvice]

Btw this issue states also Test coverage, but isn't it out of e2e scope?

[kahboom]

@jludvice - Yes, I need to add that for unit tests. I'll make a separate issue. Sorry about that. :) I'll take a look at the failing classes again today.

[iocanel]

@jludvice: A little late to the party (as I just saw that we were having similar issues with openshift). FWIW, what works fine for me is to use oc new-app instead of oc process. I even updated arquillian to support initializing using shell scripts.

[kahboom]

Closing this w/ the merge of https://github.com/redhat-ipaas/ipaas-ui/pull/308. Let's create new issues for specific E2E tasks. Thanks for the hard work @jludvice ! 👍