search

redhat-developer / odo

odo - Developer-focused CLI for fast & iterative container-based application development on Podman and Kubernetes. Implementation of the open Devfile standard.
https://odo.dev
Apache License 2.0
792 stars 244 forks source link

Move e2e tests to ci.centos.org #952

Closed kadel closed 5 years ago

kadel commented 5 years ago

We are starting to have quite a few issues with our e2e tests. They are slow and there is a lot of timeouts.

As a first step, let's move all our existing e2e tests from travis-ci to ci.centos.org.

syamgk commented 5 years ago

As the step 1. https://wiki.centos.org/QaWiki/CI/GettingStarted#head-e22a33cdce4ad83057616201c437c0bf9097c05e

created request add odo to ci.centos.org here https://bugs.centos.org/view.php?id=15460

geoand commented 5 years ago

@syamgk Do you have any idea how long it takes to get approved for ci.centos?

I am asking because if this is going to take a long time, perhaps it makes sense to also explore other options (like Semaphore or CircleCI)

geoand commented 5 years ago

@syamgk Do you have any idea how long it takes to get approved for ci.centos?

I am asking because if this is going to take a long time, perhaps it makes sense to also explore other options (like Semaphore or CircleCI)

syamgk commented 5 years ago

we got ci.centos.org account as of today! am working on configuring it

syamgk commented 5 years ago

did a mocking by creating a test branch and the results can be found here : https://ci.centos.org/job/odo/job/cico-test/16/

amitkrout commented 5 years ago

did a mocking by creating a test branch and the results can be found here : https://ci.centos.org/job/odo/job/cico-test/16/

Cool!!! @syamgk Only UT are running in the ci. Can you please start adding the integration test to it. Please let me know if you need any help to address those test in CI.

syamgk commented 5 years ago

the remaining part is adding webhoooks pointing to https://ci.centos.org/github-webhook/ for PR events and configuring e2e tests

amitkrout commented 5 years ago

Thanks for the update!!! I can see a big exception of java.lang.Exception: Method POST required thrown in [1]. Any reason behind it ?

[1] https://ci.centos.org/github-webhook/

java.lang.Exception: Method POST required
    at org.kohsuke.stapler.HttpResponses.error(HttpResponses.java:83)
    at org.jenkinsci.plugins.github.webhook.RequirePostWithGHHookPayload$Processor.shouldBePostMethod(RequirePostWithGHHookPayload.java:88)
    at org.jenkinsci.plugins.github.webhook.RequirePostWithGHHookPayload$Processor.invoke(RequirePostWithGHHookPayload.java:71)
    at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
    at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184)
    at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117)
    at org.kohsuke.stapler.IndexDispatcher.dispatch(IndexDispatcher.java:27)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
    at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:384)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
    at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:105)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
    at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.Server.handle(Server.java:531)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
    at java.lang.Thread.run(Thread.java:748)
syamgk commented 5 years ago

Thanks for the update!!! I can see a big exception of java.lang.Exception: Method POST required thrown in [1]. Any reason behind it ?

[1] https://ci.centos.org/github-webhook/

Its an endpoint which is expecting POST request and when you are accessing it from browser its doing a GET request, hence the error

$ curl -I https://ci.centos.org/github-webhook/
HTTP/1.1 405 Method Not Allowed
syamgk commented 5 years ago
syamgk commented 5 years ago

Hitting the same error as specified in this upstream issue https://github.com/openshift/origin/issues/21253 logs on ci.co https://ci.centos.org/job/odo/job/cico-test/45/console

kadel commented 5 years ago

Do you know where exactly it fails? Have you tried to run oc cluster up in verbose mode?

amitkrout commented 5 years ago

Do you know where exactly it fails? Have you tried to run oc cluster up in verbose mode?

@kadel I guess the same cluster up failure due to waiting for condition is observed long back https://github.com/openshift/origin/issues/20617 and during that time it was not that important as it was inconsistent on same setup. But after few days it grows like monster I mean in most of the platform the same error appeared consistently. Now you can see many similar kind of error waiting for condition are reported in OKD.

@syamgk Can you try out this workaround commented https://github.com/openshift/origin/issues/20617#issuecomment-452996578

syamgk commented 5 years ago

Had a discussion with dv and now I'm going to try Minishift against a remote machine

kadel commented 5 years ago

Had a discussion with dv and now I'm going to try Minishift against a remote machine

Isn't minishift doing essentially the same? It is still oc cluster up under the hood.

amitkrout commented 5 years ago

Had a discussion with dv and now I'm going to try Minishift against a remote machine

@syamgk if you are lucky enough then it works for you ;)

amitkrout commented 5 years ago

Isn't minishift doing essentially the same? It is still oc cluster up under the hood.

@kadel Yes, This is the same command with some extra flag arguments what minishuft uses to start the cluster. 

/var/lib/minishift/bin/oc cluster up --base-dir /var/lib/minishift/base --image 'openshift/origin-${component}:v3.11.0' --public-hostname 192.168.64.4 --routing-suffix 192.168.64.4.nip.io
syamgk commented 5 years ago

using minishift for bringing up cluster is working fine and was able to run end2end test on ci.centos.org logs can be found here: https://ci.centos.org/job/odo/job/cico-test/59/console

kadel commented 5 years ago

What needs to be done to complete first step:

syamgk commented 5 years ago

tested with minishift and ran all tests on ci.centos.org.

amitkrout commented 5 years ago

Currently the complete CI run takes around 1hr10min. Not a bad start i guess as minishift start itself takes almost 8min for each test in series. I am not sure if test can run parallel in centos ci, but this is a discussion material for later as @kadel mentioned.

Well done syam 💯

syamgk commented 5 years ago

Since the documents only talks about the older pattern ie, using jjb files waiting for the resoponse from brian on how to extract the branch name and pull request number for our case, also for accepting the collaborator request for ci-user account

kadel commented 5 years ago

Since the documents only talks about the older pattern ie, using jjb files waiting for the resoponse from brian on how to extract the branch name and pull request number for our case, also for accepting the collaborator request for ci-user account

how are other projects doing that?

syamgk commented 5 years ago

how are other projects doing that?

this the doc link for github integration https://wiki.centos.org/QaWiki/CI/GithubIntegration

kadel commented 5 years ago

closing in favor of https://github.com/redhat-developer/odo/issues/1431