search

cypress-io / cypress

Fast, easy and reliable testing for anything that runs in a browser.
https://cypress.io
MIT License
47.04k stars 3.19k forks source link

Proposal: Integrate Dashboard test results with outside services. #981

Closed jennifer-shehane closed 5 years ago

jennifer-shehane commented 6 years ago

We'd like to have test results integrate with third party services like:

Our Dashboard Service knows when tests fail, it has details about a user's recorded CI run. When certain conditions are met, it can notify other integrated services and do things like:

The Dashboard should likely have a UI interface allowing the user to set what integrations they would like and what data they would like sent (on project? on organization?)

This proposal is a WIP currently, so we'd love to hear from the community on what you'd like to see.

You can express interest and see progress for this feature on our Roadmap from our Dashboard's product board here: https://portal.productboard.com/cypress-io/1-cypress-dashboard (Look for the card with your desired integration or Submit idea if you don't see it)

cameronc56 commented 6 years ago

Hipchat would be great

jennifer-shehane commented 6 years ago

Conversations in Gitter surrounding integrations:

My main goal is to have some sort of notification that a test failed on a git commit Lots of ways to slice this, a simple solution I found was using a custom reporter but I needed to use multiples to accomplish the goal Built in slack integration would be best https://gitter.im/cypress-io/cypress?at=597a2d90c101bc4e3ac74704

yes that solution for slack notification used to work with my protractor tests. I changed because I have not enough information about the failure (only exit code 1). Using the afterLaunch function I have more information about which test is failing... Thats right with cypress I can send the notification to slack using the link to the dashboard to check the failure report. I love the dashboard. Have you got any plans to integrate with slack o email failure notifications? Have you got any idea/workarround how can I develop the notifications? https://gitter.im/cypress-io/cypress?at=58bee213e961e53c7fb0f7b7

Is there currently a way to hook Cypress up to a github pull request? If not, will github integration be a feature at some point? https://gitter.im/cypress-io/cypress?at=5a1ec30571ad3f873644c9f1

frankiethekneeman commented 6 years ago

Would really love it if these could be implemented as a post-run/post-upload event so we can write our own service integration plugins.

jennifer-shehane commented 6 years ago

@frankiethekneeman We are currently creating an integration with Zapier, I know that isn't exactly what you are asking, but would this help you integrate with the services you want?

frankiethekneeman commented 6 years ago

I mean, most of the major ones, probably, but it'd be nice to be able to write my own hooks for services outside that ecosystem, or even totally custom automation, around the results of these tests.

amirrustam commented 6 years ago

@jennifer-shehane I would be happy to contribute GitHub and Jira integrations. And due Atlassian Connect, the Jira work would also cover Bitbucket integration. I think this would help with #1173. I'm familiar with the APIs, and would love to help. 🙂

acthilina2020 commented 6 years ago

Cucumber integration?

jennifer-shehane commented 6 years ago

@acthilina There is a cucumber plugin that integrates with Cypress you may want to check out. https://docs.cypress.io/plugins/

acthilina2020 commented 6 years ago

Thank you @jennifer-shehane , If you have more info, please send my way. :-)

soccer99 commented 6 years ago

Gitlab please!

amirrustam commented 6 years ago

@soccer99 Thanks for the suggestion. Gitlab currently isn't on our immediate integrations roadmap. But it might be in the future, specially if there is user demand for it. We'd be glad to hear what level of integration you're looking for.

gscott commented 6 years ago

Before doing too many custom integrations I’d recommend implementing outbound webhooks; they can be used to integrate just about anything.

salty-blue-mango commented 6 years ago

We're looking for a possible integration with confluence that prints out bar charts, pass/fail, etc.

amirrustam commented 6 years ago

@Nate-Watts feel free to elaborate further on the Confluence integration for the sake of documentation. We might integrate with the Atlassian stack down the line, and it would be nice to have your feedback in our considerations. Thank you.

prescottprue commented 6 years ago

Would love to see this.

I agree with @gscott on outbound webhooks. If those exist, almost anything should be able to be wired up, then the more specific integrations can come in the future (potentially built on that same webhook system).

Shabbir124 commented 6 years ago

How we integrate Test-link to cypress? :(

Knaledge commented 6 years ago

@jennifer-shehane - is the idea with this proposal that Cypress itself will be able to integrate with all (or at least some) of the listed services? Or is this the Cypress Dashboard service integrating with the listed services?

I'm hoping for the former, though I'd understand the latter.

jennifer-shehane commented 6 years ago

@Knaledge The scope of this issue is for the Dashboard Service (any test using --recording), otherwise we don't have the data to integrate or know when to trigger notifications.

There is a separate issue regarding native notifications that could be triggered when running the Cypress test runner by itself here: https://github.com/cypress-io/cypress/issues/2145

haleagar commented 6 years ago

This may sound silly but I'd like the dashboard to integrate with the test runner. If I open the test runner and I'm logged in and it finds a projectId then show the last pass/fail state of the test on each row. And have a button in that row link to the report record on the dashboard.

It would be super convenient for digging in to explore the details of a fail, or to just highlight the rows you're interested in when fixing a bug and you need to re-run the failed test.

This then becomes a more of a request for the test runner, but boy I wish it kept the last run test highlighted even when I close the test window. (I know it has the greed dot but that disappears when I close the test window. It would be nice to indicate a very brief note about it's last pass fail count from local tests as well.

haleagar commented 6 years ago

Now less silly, Jira would be my number 1 request for the dashboard integration. Level 1) I give the Cypress dashboard access to my jira project and it scans test names for jira ticket ids. PROJ-#### that exist in my project and make them links to the tickets.

Level 2) First each test in my Suite get's it's own persistent record. (which your going to need for Analytics and api integrations) There is a URL for that record I can pop into Jira to link to the test results.

Level 3a) There is a API I can hit with ID that I can query for some brief info like last run time and pass fail status, and the URL to link to. I then build a custom integration to show that in Jira

or

Level 3b) The test record gets shown in the Issue Links in the Links to section like GitLab: Mentioned on commit messages do via whatever magic does that.

haleagar commented 6 years ago

Number 2 integration request is Google Docs 1) When it is reported to the dashboard set the value of a cell in a google sheet with pass fail and cell with date of test That would be super useful for integrating with our manual regression test matrix which is run in a google sheet.

2) Add a row to a sheet with pass fail, date, test name, link to dashboard That would allow for some history and Analytics of tests which is starkly lacking from the dashboard now. With data in a fairly pedestrian format that project managers are familiar with and can generate their own reports.

nmchaves commented 6 years ago

Having a notification for test failures would enable a really cool use case for Cypress: running "health checks" to verify that the application is operating normally in production.

Every time you deploy a new version of your application, you could have Cypress automatically visit the site and run a few smoke tests. The vast majority of tests would still run in CI, but this small set of production tests would help give you confidence that the testing environment was consistent with the prod environment. Of course, you can already do this now...but it'll be more useful when Cypress has a first-class way to notify you of test failures. Looking forward to this! ⭐️

bdimitrovski commented 5 years ago

@nmchaves For such a purpose, I have made a simple framework which integrates Cypress with Slack, you might want to check it out: https://github.com/bdimitrovski/cypress-healthcheck.

Truth be told, it is not automatic in a way you described, but if you apply a cronjob on your server to run it at specific intervals, you can have greater confidence that your production app works as intended.

jennifer-shehane commented 5 years ago

@bdim Could you open a pull request to add this plugin to our documentation? Thanks! Instructions here.

bdimitrovski commented 5 years ago

@jennifer-shehane Here it is https://github.com/cypress-io/cypress-documentation/pull/1444. Thanks! :)

nir0s commented 5 years ago

PagerDuty please :)

pareshkrc commented 5 years ago

integration with qTest https://www.qasymphony.com/

trydionel commented 5 years ago

Hi all! We're continuing to make progress on adding GitHub integration to the Cypress Dashboard and would love to get some hands-on user feedback at this point. Does anyone have a few minutes to talk through the designs with me? You'd get a chance to try out this new integration in your own project, and you'll get to keep access to the feature afterwards!

Please leave a comment here, or email me directly if you're interested. Thank you!

salty-blue-mango commented 5 years ago

I'd like to try it out!

On Fri, Mar 29, 2019, 11:25 AM Jeff Tucker notifications@github.com wrote:

Hi all! We're continuing to make progress on adding GitHub integration to the Cypress Dashboard and would love to get some hands-on user feedback at this point. Does anyone have a few minutes to talk through the designs with me? You'd get a chance to try out this new integration in your own project, and you'll get to keep access to the feature afterwards!

Please leave a comment here, or email me directly if you're interested. Thank you!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cypress-io/cypress/issues/981#issuecomment-478082501, or mute the thread https://github.com/notifications/unsubscribe-auth/AkqCPo7_fZA5S59dXOAWKX-6hEUpFrnSks5vbkyTgaJpZM4QvSL- .

nir0s commented 5 years ago

Same here!

On Fri, Mar 29, 2019, 20:45 MangoVerde-con-Sal notifications@github.com wrote:

I'd like to try it out!

On Fri, Mar 29, 2019, 11:25 AM Jeff Tucker notifications@github.com wrote:

Hi all! We're continuing to make progress on adding GitHub integration to the Cypress Dashboard and would love to get some hands-on user feedback at this point. Does anyone have a few minutes to talk through the designs with me? You'd get a chance to try out this new integration in your own project, and you'll get to keep access to the feature afterwards!

Please leave a comment here, or email me directly if you're interested. Thank you!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/cypress-io/cypress/issues/981#issuecomment-478082501 , or mute the thread < https://github.com/notifications/unsubscribe-auth/AkqCPo7_fZA5S59dXOAWKX-6hEUpFrnSks5vbkyTgaJpZM4QvSL-

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cypress-io/cypress/issues/981#issuecomment-478089138, or mute the thread https://github.com/notifications/unsubscribe-auth/AB6QhL1qXO_8Jzc09RJ120AyP0YySRDKks5vblE9gaJpZM4QvSL- .

prescottprue commented 5 years ago

@trydionel Same here!

trydionel commented 5 years ago

Awesome, thank you so much @Nate-Watts, @nir0s, @prescottprue! Just to confirm, you'll need to have 1) a project recording to the Cypress Dashboard, and 2) the repo for this project hosted on github.com, either publicly or privately. If you meet those criteria, please grab a time slot on my calendar:

https://calendly.com/trydionel/github-integration-research?month=2019-04

Looking forward to talking soon!

freedomflyer commented 5 years ago

@trydionel would also love to hop on this beta, we meet the criteria

trydionel commented 5 years ago

@freedomflyer Sure thing! Can you get in touch with me at the email listed on my GitHub profile?

sravan-s commented 5 years ago

+1 to the webhooks, it would be great if we can have webhooks with information such as -

timiyay commented 5 years ago

We use Flowdock for our company chat, so this integration would be useful for us.

amirrustam commented 5 years ago

👋 Hi everyone, we're excited to release GitHub Integration for the Cypress Dashboard today. The Cypress GitHub App tightly integrates with your workflow via commit status checks and pull-request comments. Read all about it below, and let us know what you think (@ me).

Blog post: https://cypress.io/blog/2019/07/29/github-integration-for-the-cypress-dashboard/ Docs: https://on.cypress.io/github-integration.

mitar commented 5 years ago

Awesome! :-) Please do GitLab next. :-)

cryptiklemur commented 5 years ago

Love the integration, but the experience of using it with Slack needs some love.

Right now, there are too many images being used in the commit reply, and it Slack, the message is basically a big block of image links. I like to suggest the use of emoji over the icons that are shown there and the removal of the header image.

image image

deluxetom commented 5 years ago

Love the integration, but the experience of using it with Slack needs some love.

Right now, there are too many images being used in the commit reply, and it Slack, the message is basically a big block of image links. I like to suggest the use of emoji over the icons that are shown there and the removal of the header image.

It's great to be able to see the Cypress test results on the PR page but as @aequasi mentioned, it makes the integration very noisy in slack and pretty much unusable.

trydionel commented 5 years ago

Hey @aequasi, @thomaspicquet,

Yea, that's a bit rough :grimacing: Would you mind to answer some questions about your workflow?

Thanks for the feedback!

PS: If you have any other feedback for the Cypress Dashboard, or are just curious about what we're working on, please check out our updated roadmap at https://portal.productboard.com/cypress-io/1-cypress-dashboard/tabs/1-under-consideration.

deluxetom commented 5 years ago

Hey @trydionel,

We use Slack for a lot of things and we have a channel for our CI with:

It's nice to see the Cypress PR message and I'd like to keep that over the Checks tab on github but the message is really big and not really useful when it gets to Slack, unfortunately. A slack integration sounds good or maybe an option for the Github integration to turn off the message?

trydionel commented 5 years ago

Cool, I've added your feedback to our product backlog. Thanks for the info!

Also, if you find the PR comments too noisy, you can turn them off independently of the commit status checks. See our documentation.

arifmasood commented 5 years ago

@jennifer-shehane @amirrustam @trydionel +1 for sending cypress run reports to designated emails.

It will be great if we can have summarized results of cypress run (from dashboard) being delivered to specific email addresses. Raising a red alarm in case of any failures or sending a green report in case tests passed.

jennifer-shehane commented 5 years ago

You can express interest and see progress for this feature on our Roadmap from our Dashboard's product board here: https://portal.productboard.com/cypress-io/1-cypress-dashboard (Look for the card with your desired integration or Submit idea if you don't see it)

I will be closing this issue and closing it to all comments since this is related to our Dashboard.

All related work for Dashboard features is handled in that ProductBoard and will be addressed by the Dashboard team directly when you comment there. We have already noted the initial comments from this issue there. Please refer all future comments there. Thank you.

jennifer-shehane commented 4 years ago

Slack integration is now available within our Cypress Dashboard. It allows you to see your Cypress test results directly in your teams’ Slack channels. Please read our Slack Integration docs to get started.