search

jenkinsci / jms-messaging-plugin

https://plugins.jenkins.io/jms-messaging
12 stars 38 forks source link

Add simple client for RabbitMQ #147

Closed Zlopez closed 4 years ago

Zlopez commented 4 years ago

This client is based on RabbitMQ Java Client.

Supports SSL and Username/Password authentication and could be configured to work with any RabbitMQ server.

Connection to each server is created only once and separate channel is created for each consumer and publisher.

When publishing generates message id using java.util.UUID and send this id as property of the message.

Consuming is done using by using Push API (subscription) even when listening for messages for a limited time.

Signed-off-by: Michal Konečný mkonecny@redhat.com

Zlopez commented 4 years ago

This implements #132

pypingou commented 4 years ago

Pretty ping? :)

scoheb commented 4 years ago

@pypingou @Zlopez

I am getting back to reviewing PRs and the CI environment is having issues. I am testing things locally and will test this soon.

On a side note, how does this differ from https://github.com/jenkinsci/jms-messaging-plugin/pull/145 ?

Zlopez commented 4 years ago

@scoheb This is implemented using RabbitMQ client and supports everything RabbitMQ has to offer.

I used AMQP and Fedmsg as inspiration, but RabbitMQ has more features than these two.

scoheb commented 4 years ago

@Zlopez so can your provider be used to connect to an AMQP instance/broker?

Zlopez commented 4 years ago

@scoheb I don't think so, the connection is established by using RabbitMQ ConnectionFactory.

scoheb commented 4 years ago

I would like to see some basic tests that exercise sending and receiving messages. Please base yourself on the existing FedMsg and ActiveMq tests that make use of a Docker image to run rabbitmq.

Zlopez commented 4 years ago

I will look at the existing tests, but I wasn't able to run them when I wanted to test them.

scoheb commented 4 years ago

I will be merging some fixes to make the tests more robust soon...

Zlopez commented 4 years ago

@scoheb Do you plan to merge to rabbitmq_skeleton branch too?

Zlopez commented 4 years ago

And one important question, how should I run those tests?

scoheb commented 4 years ago

@Zlopez do you want to re-base off of master now instead of the skeleton branch?

As for running the tests, you need:

then run:

or remove the -Dtest= to run whole suite.

Warning they do take a while to run.

Zlopez commented 4 years ago

I could rebase to master, but I will create a new PR, I'm not sure if you could change existing PR to merge to different branch.

pypingou commented 4 years ago

@Zlopez if you edit the PR at the top, you can change the target branch :)

Zlopez commented 4 years ago

I already created a new branch in my fork for this, because the rebase to master branch didn't look well. So I will close this one, when I open a new one, which is correctly created from master branch.

Zlopez commented 4 years ago

@scoheb I'm trying to run tests now, I have geckodriver in PATH, firefox installed and podman installed. But almost every test is skipped:

[INFO] Running com.redhat.jenkins.plugins.ci.MigrationTest                                                                                                                                                                                    
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.272 s - in com.redhat.jenkins.plugins.ci.MigrationTest                                                                                                              
[INFO] Running com.redhat.jenkins.plugins.ci.integration.AmqMessagingPluginLockdownIntegrationTest                                                                                                                                            
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 12.38 s - in com.redhat.jenkins.plugins.ci.integration.AmqMessagingPluginLockdownIntegrationTest                                                                    
[INFO] Running com.redhat.jenkins.plugins.ci.integration.FedMsgMessagingPluginIntegrationTest                                                                                                                                                 
[WARNING] Tests run: 30, Failures: 0, Errors: 0, Skipped: 30, Time elapsed: 252.076 s - in com.redhat.jenkins.plugins.ci.integration.FedMsgMessagingPluginIntegrationTest                                                                     
[INFO] Running com.redhat.jenkins.plugins.ci.integration.AmqMessagingPluginWithFailoverIntegrationTest                                                                                                                                        
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 50.945 s - in com.redhat.jenkins.plugins.ci.integration.AmqMessagingPluginWithFailoverIntegrationTest
scoheb commented 4 years ago

It may be that warning message from podman screwing things up when it tries to detect if Docker is running or available...what does 'docker ps' give you? If it mentions to create a file to remove a message, I think that is the problem.

Zlopez commented 4 years ago

You were right, I didn't had docker installed and the docker ps wasn't recognized. Now I have it, but I'm getting this:

Skipping testSimpleCIEventTriggerWithCheck(com.redhat.jenkins.plugins.ci.integration.FedMsgMessagingPluginIntegrationTest)
org.junit.internal.AssumptionViolatedException: Docker is needed for the test
    at org.jenkinsci.test.acceptance.junit.WithDocker$RuleImpl$1.hasDocker(WithDocker.java:57)
    at org.jenkinsci.test.acceptance.junit.WithDocker$RuleImpl$1.evaluate(WithDocker.java:49)
    at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
    at org.jenkinsci.test.acceptance.junit.JenkinsAcceptanceTestRule$1.evaluate(JenkinsAcceptanceTestRule.java:58)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
    at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
1572611919126   Marionette  INFO    Stopped listening on port 35709

What I tried:

And still I'm getting the error above.

scoheb commented 4 years ago

in my env, podman is installed...not docker. The check in the tests simply does:

docker ps and expects a '0' exit code.

Here is the result from my local env:

$ which docker
/usr/bin/docker

$ file /usr/bin/docker
/usr/bin/docker: a /usr/bin/sh script, ASCII text executable

$ cat  /usr/bin/docker
#!/usr/bin/sh
[ -f /etc/containers/nodocker ] || \
echo "Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg." >&2
exec /usr/bin/podman "$@"

$ docker ps
CONTAINER ID  IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES

$ echo $?
0
Zlopez commented 4 years ago

Thanks, the bash script did the trick.

Andrei-Stepanov commented 4 years ago

Hi, we have a plan to deploy the plugin on https://jenkins-continuous-infra.apps.ci.centos.org/ The master already has jms-messaging-plugin. Question: can the updated plugin continue work with fedora-fedmsg?

Zlopez commented 4 years ago

@Andrei-Stepanov Fedmsg is different messaging system, so it is possible to use this beside RabbitMQ.

Zlopez commented 4 years ago

@scoheb I was able to run the tests without docker error, I'm now trying to run mvn -B test -DElasticTime.factor=2 -Djenkins.version=2.176.1 -DforkCount=1 -Dtest=FedMsgMessagingPluginIntegrationTest#testSimpleCIEventTriggerWithCheck

But it's failing with this:

Building Docker image `docker build -t jenkins/fedmsg-relay:e048987d8a07 /tmp/Dockerfile5088748677418623093dir`: logfile is at target/diagnostics/testSimpleCIEventTriggerWithCheck(com.redhat.jenkins.plugins.ci.integration.FedMsgMessagingPluginIntegrationTest)/docker-FedmsgRelayContainer.build.log
Launching Docker container `docker run -d -p 127.0.0.1::4001 -p 127.0.0.1::2003 -p 127.0.0.1::22 jenkins/fedmsg-relay:e048987d8a07`: logfile will be at target/diagnostics/testSimpleCIEventTriggerWithCheck(com.redhat.jenkins.plugins.ci.integration.FedMsgMessagingPluginIntegrationTest)/docker-FedmsgRelayContainer.run.log
Lis 01, 2019 4:25:03 ODP. org.jenkinsci.test.acceptance.update_center.MockUpdateCenter lambda$serverExceptionHandler$5
WARNING: Exception thrown while serving request
java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:210)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
    at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
    at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
    at org.apache.http.impl.io.DefaultHttpRequestParser.parseHead(DefaultHttpRequestParser.java:129)
    at org.apache.http.impl.io.DefaultHttpRequestParser.parseHead(DefaultHttpRequestParser.java:53)
    at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
    at org.apache.http.impl.DefaultBHttpServerConnection.receiveRequestHeader(DefaultBHttpServerConnection.java:129)
    at org.apache.http.protocol.HttpService.handleRequest(HttpService.java:308)
    at org.apache.http.impl.bootstrap.Worker.run(Worker.java:66)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

127.0.0.1:36807
127.0.0.1:41561
master53872|Nov 01, 2019 4:25:38 PM com.redhat.jenkins.plugins.ci.CIBuildTrigger stopTriggerThreads
master53872|INFO: Removed thread lock for 'representative_glow'.
master53872|Nov 01, 2019 4:25:38 PM com.redhat.jenkins.plugins.ci.CIBuildTrigger startTriggerThreads
master53872|INFO: Starting thread (183) for 'representative_glow'.
master53872|Nov 01, 2019 4:25:38 PM com.redhat.jenkins.plugins.ci.ProjectChangeListener onUpdated
master53872|INFO: Getting trigger threads.
master53872|Nov 01, 2019 4:25:39 PM com.redhat.jenkins.plugins.ci.messaging.FedMsgMessagingWorker subscribe
master53872|INFO: Subscribing job 'representative_glow' to org.fedoraproject topic.
master53872|Nov 01, 2019 4:25:39 PM com.redhat.jenkins.plugins.ci.messaging.FedMsgMessagingWorker subscribe
master53872|INFO: Successfully subscribed job 'representative_glow' to topic 'org.fedoraproject'.
master53872|Nov 01, 2019 4:25:53 PM com.redhat.jenkins.plugins.ci.ProjectChangeListener onUpdated
master53872|INFO: No CIBuildTrigger found, forcing thread stop.
master53872|Nov 01, 2019 4:25:53 PM com.redhat.jenkins.plugins.ci.CIBuildTrigger stopTriggerThreads
master53872|INFO: Removed thread lock for 'trivial_liquid'.
master53872|Nov 01, 2019 4:25:56 PM hudson.model.Run execute
master53872|INFO: trivial_liquid #1 main build action completed: SUCCESS
master53872|Nov 01, 2019 4:25:56 PM com.redhat.utils.MessageUtils sendMessage
master53872|INFO: Sending message for job 'trivial_liquid'.
master53872|Nov 01, 2019 4:26:03 PM com.redhat.jenkins.plugins.ci.messaging.JMSMessagingWorker trigger
master53872|INFO: Scheduling job 'representative_glow' based on message:
master53872|{"content":"catch me"}
master53872|Nov 01, 2019 4:26:03 PM hudson.model.Run execute
master53872|INFO: representative_glow #1 main build action completed: SUCCESS
1572621965303   Marionette  INFO    Stopped listening on port 46111

###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
Zlopez commented 4 years ago

Closing this in favor of #154