Correct way to configure scicat to run with RabbitMQ

[LAShemilt]

Configure Scicat to run with Rabbit MQ

Summary

I have followed the documentation to change the configuration files for scicat so it will run with a local service of RabbitMQ running in a docker container. I have tested my local service using the pika python client so I know that I can connect to it independently of Scicat. For testing purposes I also have scicat running locally, I am trying to get this instance to talk to my RabbitMQ server. I have tried to follow the template of the component-config.local.js from this repos CI/PSI/component-config.local.js to set up my RabbitMQ server. I have also edited the config.local.js file to give the correct ports for my local version of RabbitMQ.

I can start the instance of Scicat with these two component files, however I cannot see any evidence that Scicat is sending messages on rabbitMQ. I have tried checking with rabbitmq cli to list the bindings and queues, but cannot see those that I define in my component config. I have also tried posting a Job through loopback explorer and seeing if a message appears in the rabbitmq logs but do not see anything. Should I set up the queue and the exchange before connecting with Scicat? Or does Scicat create these through the config?

The part of my config.local.js file relating to RabbitMQ is:

rabbitmq: {
    enabled: true,
    host: "172.17.0.1",
    port: 5672,
    queue: "client.jobs.write",
  }

The component-config.local.js file:

{
  "loopback-component-explorer": {
    "mountPath": "/explorer"
  },
  "../node_modules/loopback-component-mq/lib": {
    "options": {
      "restPort": 5672,
      "acls": [
        {
          "accessType": "*",
          "principalType": "ROLE",
          "principalId": "$unauthenticated",
          "permission": "DENY"
        }
      ]
    },
    "topology": {
      "connection": {
        "uri": "amqp://guest:guest@172.17.0.1:5672/"
      },
      "exchanges": [
        {
          "name": "jobs.write",
          "type": "topic",
          "persistent": true
        }
      ],
      "queues": [
        {
          "name": "client.jobs.write",
          "subscribe": true
        }
      ],
      "bindings": [
        {
          "exchange": "jobs.write",
          "target": "client.jobs.write",
          "keys": [
            "jobqueue"
          ]
        }
      ]
    }
  }

I use docker compose to start up my instance of Scicat, here is the docker compose part for catamel. I am also starting rabbitmq as a service:

services:
 rabbitmq:
    image: library/rabbitmq:latest
    ports:
      - "5692:5692"
  catamel:
    image: dacat/catamel:latest
    depends_on:
      - mongodb
      - openldap
      - rabbitmq
    volumes:
      - ./config/catamel/config.local.js:/usr/src/app/server/config.local.js
      - ./config/catamel/config.local.js:/usr/src/app/server/component-config.local.js
      - ./config/catamel/providers.json:/usr/src/app/server/providers.json

Any help with how to connect and anything I have missed would be appreciated! Many thanks.

[stephan271]

Are there any error messages from the catamel logs ? As far as I remember (too long ago) I set up the queues manually in Rabbitmq before connection.

[LAShemilt]

So far nothing in the catamel logs. When I set up the queues manually before in python I was getting a heart beat error in RabbitMQ on opening scicat. On googling, it seems that this might have been caused by my connection in python. I will have a go again but this time set things up through a rabbitMQ config file or through the CLI.

Another question, is there anyway one can test the connection to rabbitMQ through the loopback explorer (or with another method)? I have just been posting jobs because I saw in the architecture diagrams that a job entry in mongo triggered the messaging system so that is where I started, but I wonder if there is a better way.

Thanks again for your help.

[stephan271]

I also only know this way, creating a job...no direct option via loopback explorer

[stephan271]

What branch did you use ? Could you give me the log output when catamel starts up ?

[stephan271]

In fact what we use here at PSI is not the Rabbitmq guest user, but a user which is assigned administrator rights. This user "user" is created via the RabbitMQ GUI, using port 15672. The line "uri": "amqp://guest:guest@172.17.0.1:5672/" is adjusted then with user:password

The user "user" then must have the following virtual host settings (you can ignore the sls line, I think)

One other thing I noticed: why do you use port 5672 in component-config.local.js, but 5692 in your service definition ?

[LAShemilt]

Thanks for this, good spot on the ports. I have put this back to the correct ports but still has not solved the issues. There was also a second problem in the docker-compose.yaml where I copy the component-config.local.js to the wrong location. I have fixed both these issues but it is still not working. My next attempt was to look into the ACLs and users. I had set up the queues and exchange manually but still was not sending messages. I will follow your direction on setting up the user and let you know.

Many thanks for looking into this.

[LAShemilt]

I am currently using the rfi's docker-compose file to spin up scicat locally. This pulls the dacat/catamel:latest image, which I assume checks out the current master branch.

The catamel logs look like this for start up and receiving a post request for a job:

[+] Logger set to: Console

Datasource host mongodb, database dacat

Adding ACLS for UserIdentity

INFO - Adding relations to User

{"relation":"UserSetting"}

INFO - Adding ACLS for User related models

{"relation":"UserSetting"}

Account already exists: proposalIngestor

Index on field datasetlifecycle.archivable created successfully

Index on field datasetlifecycle.retrievable created successfully

Index on field datasetlifecycle.publishable created successfully

Index on field datasetlifecycle.archiveStatusMessage created successfully

Index on field datasetlifecycle.retrieveStatusMessage created successfully

Text Index on Dataset created successfully

Text Index on Sample created successfully

Text Index on Proposal created successfully

Text Index on OrigDatablock created successfully

Role already exists: proposalingestor

Rolemapping already exists: proposalIngestor proposalingestor

Database Autoupdate result: undefined

Role already exists: globalaccess

Rolemapping already exists: proposalIngestor globalaccess

Account already exists: archiveManager

Role already exists: archivemanager

Rolemapping already exists: archiveManager archivemanager

Role already exists: globalaccess

Rolemapping already exists: archiveManager globalaccess

Account already exists: ingestor

Role already exists: ingestor

Rolemapping already exists: ingestor ingestor

Role already exists: globalaccess

Rolemapping already exists: ingestor globalaccess

Account already exists: admin

Role already exists: admin

Rolemapping already exists: admin admin

Role already exists: globalaccess

Rolemapping already exists: admin globalaccess

Web server listening at: http://localhost:3000

Browse your REST API at http://localhost:3000/explorer

[+] Logger set to: Console

Datasource host mongodb, database dacat

Adding ACLS for UserIdentity

INFO - Adding relations to User

{"relation":"UserSetting"}

INFO - Adding ACLS for User related models

{"relation":"UserSetting"}

Account already exists: proposalIngestor

Index on field datasetlifecycle.archivable created successfully

Index on field datasetlifecycle.retrievable created successfully

Index on field datasetlifecycle.publishable created successfully

Index on field datasetlifecycle.archiveStatusMessage created successfully

Index on field datasetlifecycle.retrieveStatusMessage created successfully

Text Index on Dataset created successfully

Text Index on Sample created successfully

Text Index on OrigDatablock created successfully

Text Index on Proposal created successfully

Role already exists: proposalingestor

Rolemapping already exists: proposalIngestor proposalingestor

Database Autoupdate result: undefined

Role already exists: globalaccess

Rolemapping already exists: proposalIngestor globalaccess

Account already exists: archiveManager

Role already exists: archivemanager

Rolemapping already exists: archiveManager archivemanager

Role already exists: globalaccess

Rolemapping already exists: archiveManager globalaccess

Account already exists: ingestor

Role already exists: ingestor

Rolemapping already exists: ingestor ingestor

Role already exists: globalaccess

Rolemapping already exists: ingestor globalaccess

Account already exists: admin

Role already exists: admin

Rolemapping already exists: admin admin

Role already exists: globalaccess

Rolemapping already exists: admin globalaccess

Web server listening at: http://localhost:3000

Browse your REST API at http://localhost:3000/explorer

INFO - Fetching metadataKeys

{"fields":{"mode":{},"isPublished":false,"metadataKey":""},"limits":{},"options":{"prohibitHiddenPropertiesInQuery":true,"maxDepthOfQuery":12,"maxDepthOfData":32,"accessToken":{"id":"E4InXUXXwbClch5Zxg2TBHoGXYht3lCFwwCk2yb95SAd4pGYdFLgi02oYFGFUOOf","ttl":1209600,"created":"2020-11-27T13:33:12.924Z","userId":"5fa51991c48be430dbc57623"},"authorizedRoles":{"$authenticated":true,"$everyone":true},"currentUser":"ingestor","currentUserEmail":"scicatingestor@your.site","currentGroups":["globalaccess","ingestor","ingestor"]},"blacklist":["/.*_date/","/runNumber/","/Entrych*./","/entryCh*./","/FMC-PICO*./","/BW_measurement*./","/Linearity_measurement*./","/Pulse_measurement*./"]}

INFO - Found datasets

{"count":10}

INFO - Raw metadata array

{"count":10}

INFO - Curated metadataKeys

{"count":0}

I have managed to set up a rabbitmq server with the management console and have added a user as you suggest. I connect as this user through the python client on the correct ports and receive messages on the correct queue. However when I start Scicat I have no evidence of connection.

When I send a job through the loopback explorer I get a 200 reponse but no messages generated or connections seen.

I appreciate that this error can not be easily replicated - is there some message that should appear in the catamel logs on connection that I can look out for?

Many thanks

[LAShemilt]

In response to my last question, I have found this in the server.js script:

if ("queue" in configLocal) {
    var msg = "Queue configured to be ";
    switch (configLocal.queue) {
        case "rabbitmq":
            console.log(msg + "RabbitMQ");
            break;
        case "kafka":
            console.log(msg + "Apache Kafka");
            break;
        default:
            console.log("Queuing system not configured.");
            break;
    }
}

I noticed that I was missing the queue: 'rabbitmq' line in the config.local.js and have added it. However I still don't see any of the above messages. Should I be seeing something like this?

[stephan271]

You did not tell me yet which branch are you using... AT startup you should see the "Queue configured to be RabbitMQ" message. DO you see it ?

[LAShemilt]

Sorry I am using a docker-compose file from the rfi which pulls in the latest image of dacat/catamel and catamel , and adds our local configs. I have cloned these images 12 days ago and I see there has been updates in the last week.

Thanks for letting me know about the message. I do not see the message on set up but I will pull the latest docker image again.

[stephan271]

I do not know which version the docker images have. It is usually safer to create your own image... Anyway: depending on the version you may need a different syntax inside job.json:

index 3c92c91..49f9304 100644
--- a/common/models/job.json
+++ b/common/models/job.json
@@ -8,10 +8,12 @@
         "validateUpsert": true
     },
     "replaceOnPUT": false,
-    "MessageQueue": {
-        "producers": {
-            "publishJob": {
-                "exchange": "jobs.write"
+    "mixins": {
+        "MessageQueue": {
+            "producers": {
+                "publishJob": {
+                    "exchange": "jobs.write"
+                }
             }
         }
     },```

[LAShemilt]

Thanks for the tip, I will try making the image myself, from the latest release branch.

[LAShemilt]

This has now worked. Catamel tries to contact rabbitMQ but fails due to security issues on the rabbitMQ side catamel_1 | [+] Logger set to: Console catamel_1 | Queue configured to be RabbitMQ catamel_1 | Email settings detected catamel_1 | Datasource host mongodb_data, database dacat catamel_1 | Adding ACLS for UserIdentity catamel_1 | INFO - Adding relations to User catamel_1 | {"relation":"UserSetting"} catamel_1 | INFO - Adding ACLS for User related models catamel_1 | {"relation":"UserSetting"} catamel_1 | Unable to configure Rabbit topology No endpoints could be reached

I will work on my rabbitMQ server, but will close this issue for now.