Closed Zlopez closed 4 years ago
This implements #132
Pretty ping? :)
@pypingou @Zlopez
I am getting back to reviewing PRs and the CI environment is having issues. I am testing things locally and will test this soon.
On a side note, how does this differ from https://github.com/jenkinsci/jms-messaging-plugin/pull/145 ?
@scoheb This is implemented using RabbitMQ client and supports everything RabbitMQ has to offer.
I used AMQP and Fedmsg as inspiration, but RabbitMQ has more features than these two.
@Zlopez so can your provider be used to connect to an AMQP instance/broker?
@scoheb I don't think so, the connection is established by using RabbitMQ ConnectionFactory.
I would like to see some basic tests that exercise sending and receiving messages. Please base yourself on the existing FedMsg and ActiveMq tests that make use of a Docker image to run rabbitmq.
I will look at the existing tests, but I wasn't able to run them when I wanted to test them.
I will be merging some fixes to make the tests more robust soon...
@scoheb Do you plan to merge to rabbitmq_skeleton branch too?
And one important question, how should I run those tests?
@Zlopez do you want to re-base off of master now instead of the skeleton branch?
As for running the tests, you need:
then run:
mvn clean install -DskipTests
mvn -B test -DElasticTime.factor=2 -Djenkins.version=2.176.1 -DforkCount=1 -Dtest=FedMsgMessagingPluginIntegrationTest#testSimpleCIEventTriggerWithCheck
or remove the -Dtest=
to run whole suite.
Warning they do take a while to run.
I could rebase to master, but I will create a new PR, I'm not sure if you could change existing PR to merge to different branch.
@Zlopez if you edit the PR at the top, you can change the target branch :)
I already created a new branch in my fork for this, because the rebase to master branch didn't look well. So I will close this one, when I open a new one, which is correctly created from master branch.
@scoheb I'm trying to run tests now, I have geckodriver in PATH, firefox installed and podman installed. But almost every test is skipped:
[INFO] Running com.redhat.jenkins.plugins.ci.MigrationTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.272 s - in com.redhat.jenkins.plugins.ci.MigrationTest
[INFO] Running com.redhat.jenkins.plugins.ci.integration.AmqMessagingPluginLockdownIntegrationTest
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 12.38 s - in com.redhat.jenkins.plugins.ci.integration.AmqMessagingPluginLockdownIntegrationTest
[INFO] Running com.redhat.jenkins.plugins.ci.integration.FedMsgMessagingPluginIntegrationTest
[WARNING] Tests run: 30, Failures: 0, Errors: 0, Skipped: 30, Time elapsed: 252.076 s - in com.redhat.jenkins.plugins.ci.integration.FedMsgMessagingPluginIntegrationTest
[INFO] Running com.redhat.jenkins.plugins.ci.integration.AmqMessagingPluginWithFailoverIntegrationTest
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 50.945 s - in com.redhat.jenkins.plugins.ci.integration.AmqMessagingPluginWithFailoverIntegrationTest
It may be that warning message from podman screwing things up when it tries to detect if Docker is running or available...what does 'docker ps' give you? If it mentions to create a file to remove a message, I think that is the problem.
You were right, I didn't had docker installed and the docker ps
wasn't recognized. Now I have it, but I'm getting this:
Skipping testSimpleCIEventTriggerWithCheck(com.redhat.jenkins.plugins.ci.integration.FedMsgMessagingPluginIntegrationTest)
org.junit.internal.AssumptionViolatedException: Docker is needed for the test
at org.jenkinsci.test.acceptance.junit.WithDocker$RuleImpl$1.hasDocker(WithDocker.java:57)
at org.jenkinsci.test.acceptance.junit.WithDocker$RuleImpl$1.evaluate(WithDocker.java:49)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.jenkinsci.test.acceptance.junit.JenkinsAcceptanceTestRule$1.evaluate(JenkinsAcceptanceTestRule.java:58)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
1572611919126 Marionette INFO Stopped listening on port 35709
What I tried:
And still I'm getting the error above.
in my env, podman is installed...not docker. The check in the tests simply does:
docker ps
and expects a '0' exit code.
Here is the result from my local env:
$ which docker
/usr/bin/docker
$ file /usr/bin/docker
/usr/bin/docker: a /usr/bin/sh script, ASCII text executable
$ cat /usr/bin/docker
#!/usr/bin/sh
[ -f /etc/containers/nodocker ] || \
echo "Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg." >&2
exec /usr/bin/podman "$@"
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ echo $?
0
Thanks, the bash script did the trick.
Hi, we have a plan to deploy the plugin on https://jenkins-continuous-infra.apps.ci.centos.org/ The master already has jms-messaging-plugin. Question: can the updated plugin continue work with fedora-fedmsg?
@Andrei-Stepanov Fedmsg is different messaging system, so it is possible to use this beside RabbitMQ.
@scoheb I was able to run the tests without docker error, I'm now trying to run mvn -B test -DElasticTime.factor=2 -Djenkins.version=2.176.1 -DforkCount=1 -Dtest=FedMsgMessagingPluginIntegrationTest#testSimpleCIEventTriggerWithCheck
But it's failing with this:
Building Docker image `docker build -t jenkins/fedmsg-relay:e048987d8a07 /tmp/Dockerfile5088748677418623093dir`: logfile is at target/diagnostics/testSimpleCIEventTriggerWithCheck(com.redhat.jenkins.plugins.ci.integration.FedMsgMessagingPluginIntegrationTest)/docker-FedmsgRelayContainer.build.log
Launching Docker container `docker run -d -p 127.0.0.1::4001 -p 127.0.0.1::2003 -p 127.0.0.1::22 jenkins/fedmsg-relay:e048987d8a07`: logfile will be at target/diagnostics/testSimpleCIEventTriggerWithCheck(com.redhat.jenkins.plugins.ci.integration.FedMsgMessagingPluginIntegrationTest)/docker-FedmsgRelayContainer.run.log
Lis 01, 2019 4:25:03 ODP. org.jenkinsci.test.acceptance.update_center.MockUpdateCenter lambda$serverExceptionHandler$5
WARNING: Exception thrown while serving request
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
at org.apache.http.impl.io.DefaultHttpRequestParser.parseHead(DefaultHttpRequestParser.java:129)
at org.apache.http.impl.io.DefaultHttpRequestParser.parseHead(DefaultHttpRequestParser.java:53)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpServerConnection.receiveRequestHeader(DefaultBHttpServerConnection.java:129)
at org.apache.http.protocol.HttpService.handleRequest(HttpService.java:308)
at org.apache.http.impl.bootstrap.Worker.run(Worker.java:66)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
127.0.0.1:36807
127.0.0.1:41561
master53872|Nov 01, 2019 4:25:38 PM com.redhat.jenkins.plugins.ci.CIBuildTrigger stopTriggerThreads
master53872|INFO: Removed thread lock for 'representative_glow'.
master53872|Nov 01, 2019 4:25:38 PM com.redhat.jenkins.plugins.ci.CIBuildTrigger startTriggerThreads
master53872|INFO: Starting thread (183) for 'representative_glow'.
master53872|Nov 01, 2019 4:25:38 PM com.redhat.jenkins.plugins.ci.ProjectChangeListener onUpdated
master53872|INFO: Getting trigger threads.
master53872|Nov 01, 2019 4:25:39 PM com.redhat.jenkins.plugins.ci.messaging.FedMsgMessagingWorker subscribe
master53872|INFO: Subscribing job 'representative_glow' to org.fedoraproject topic.
master53872|Nov 01, 2019 4:25:39 PM com.redhat.jenkins.plugins.ci.messaging.FedMsgMessagingWorker subscribe
master53872|INFO: Successfully subscribed job 'representative_glow' to topic 'org.fedoraproject'.
master53872|Nov 01, 2019 4:25:53 PM com.redhat.jenkins.plugins.ci.ProjectChangeListener onUpdated
master53872|INFO: No CIBuildTrigger found, forcing thread stop.
master53872|Nov 01, 2019 4:25:53 PM com.redhat.jenkins.plugins.ci.CIBuildTrigger stopTriggerThreads
master53872|INFO: Removed thread lock for 'trivial_liquid'.
master53872|Nov 01, 2019 4:25:56 PM hudson.model.Run execute
master53872|INFO: trivial_liquid #1 main build action completed: SUCCESS
master53872|Nov 01, 2019 4:25:56 PM com.redhat.utils.MessageUtils sendMessage
master53872|INFO: Sending message for job 'trivial_liquid'.
master53872|Nov 01, 2019 4:26:03 PM com.redhat.jenkins.plugins.ci.messaging.JMSMessagingWorker trigger
master53872|INFO: Scheduling job 'representative_glow' based on message:
master53872|{"content":"catch me"}
master53872|Nov 01, 2019 4:26:03 PM hudson.model.Run execute
master53872|INFO: representative_glow #1 main build action completed: SUCCESS
1572621965303 Marionette INFO Stopped listening on port 46111
###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
Closing this in favor of #154
This client is based on RabbitMQ Java Client.
Supports SSL and Username/Password authentication and could be configured to work with any RabbitMQ server.
Connection to each server is created only once and separate channel is created for each consumer and publisher.
When publishing generates message id using java.util.UUID and send this id as property of the message.
Consuming is done using by using Push API (subscription) even when listening for messages for a limited time.
Signed-off-by: Michal Konečný mkonecny@redhat.com