search

Data-to-Insight-Center / komadu

Provenance collection and visualization tool based on W3C PROV standard
Apache License 2.0
14 stars 6 forks source link

Running Komadu on Mesos cluster: Web service or Standalone server? #5

Open idea-launch-lab opened 9 years ago

idea-launch-lab commented 9 years ago

Hi @isurues We have an Apache Mesos cluster within which we'd like to deploy Komadu. Ideally, we'd like komadu to run in its own VM on the cluster and be able to receive event-style notifications from a web app and other process running on another, production-level, VM. Essentially, we'd like to be able to send prov notification events from other VMs to komdu (running in its own VM).

What I'd like to know is which of the two methods of running Komadu, web service or standalone server, would be suitable for our scenario. Please also clarify what accommodations, if any, would we need to make to enable Komadu to function within this multi-VM envrionment.

Thanks,

Sid

isurues commented 9 years ago

Hi Sid,

According to your description, I think it'll be more suitable to use RabbitMQ messaging channel for your use case. Here are the reasons.

  1. If you have multiple VMs which try to publish events into Komadu, it's more efficient to have a central messaging bus which can receive all events and pump into Komadu. If you use the Web Service channel, each VM will have to create 1-1 connections with Komadu and publish events.
  2. If you collect provenance in the critical path of your application, using RabbitMQ will reduce the overhead compared to Web Services. Because it just sends the event to RabbitMQ bus and return.

You won't need anything special on your VM to Komadu in either mode. If you decide to use RabbitMQ, you'll have to install it in the same VM or another one depending on your load. All you need are network connectivity and java.

Let us know if you need further help.

Thanks, Isuru

idea-launch-lab commented 9 years ago

Thanks, Isuru, that's helpful information. Looks like option that uses RabbitMQ should work in our case.

idea-launch-lab commented 9 years ago

Another question I have about deployment: we have a three-tier environment starting with user's local machine / desktop, dev environment, and production environment.

I am wondering if Komadu running on a single VM can serve the different environments. I looked at Komadu config file and one needs to specify database location and credentials. Is it possible to start Komadu web service that points to different instances of MySQL DB? May be create different collections / tables for the three environments? Or do we need three different instances of Komadu/MySQL/RabbitMQ, one for each of the environments? Thanks.

idea-launch-lab commented 9 years ago

@isurues A question about Komadu ingest and query API. We have deployed Komadu as a message-oriented service (RabbitMQ). From the manual it appears that the current ingest/query API is geared towards axis2 server setup (through komadu-axis2-client-core-1.0.jar). Is there a Komadu API to send / receive messages using RabbitMQ messaging framework?

We'd like to send provenance notifications to Komadu server from external sources and it is not clear from Komadu manual how we would do that within the messaging framework. Thanks.

gabrielzhouquan commented 9 years ago

Hi Sid,

Please refer to Komadu user guide as here:

https://github.com/Data-to-Insight-Center/komadu/blob/master/docs/KomaduUserGuide.pdf

From Page 15-16, the section covers messaging client usage with bash script.

For Java APIs, you could find send/receive message APIs in the package below:

package edu.indiana.d2i.komadu.client.messaging;

And use Notification and Query class.

Please let me know if you have further questions.

best,

Quan Zhou

On Thu, Aug 20, 2015 at 1:03 PM, sid thakur notifications@github.com wrote:

@isurues https://github.com/isurues A question about Komadu ingest and query API. We have deployed Komadu as a message-oriented service (RabbitMQ). From the manual it appears that the current ingest/query API is geared towards axis2 server setup (through komadu-axis2-client-core-1.0.jar). Is there a Komadu API to send / receive messages using RabbitMQ messaging framework?

We'd like to send provenance notifications to Komadu server from external sources and it is not clear from Komadu manual how we would do that within the messaging framework. Thanks.

— Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-133155751 .

idea-launch-lab commented 9 years ago

@gabrielzhouquan Thanks, I will look up that information. Essentially, we have a web stack (Node.js) within which we'd like to capture events and send notifications to Komadu. Komadu is running on a different VM than our web stack, so need to figure out how to send data between the two servers.

gabrielzhouquan commented 9 years ago

Hi Sid,

Glad that could help you with this. For Node.js, there is a amqp.node lib which can be used to send and receive notification from Komadu messaging server. Currently this is not integrated into Komadu messaging client. :(

Best,

Gabriel Zhou

On Thu, Aug 20, 2015 at 1:22 PM, sid thakur notifications@github.com wrote:

@gabrielzhouquan https://github.com/gabrielzhouquan Thanks, I will look up that information. Essentially, we have a web stack (Node.js) within which we'd like to capture events and send notifications to Komadu. Komadu is running on a different VM than our web stack, so need to figure out how to send data between the two servers.

— Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-133159927 .

idea-launch-lab commented 9 years ago

Hi Gabriel, Thanks for the helpful pointer. I will look up amqp library for node.js.

Sid

idea-launch-lab commented 9 years ago

@gabrielzhouquan I have some more questions about using Komadu through RabbitMQ-based messaging client. I am able to run RabbitMQ server and have better understanding now of how to create and run producer/consumer from my application (e.g., using AMQP for sending / receiving messages from RabbitMQ).

$ sudo rabbitmqctl list_queues
Listing queues ...
KomaduQueueQueryRequest 0
KomaduQueueQueryResponse    0
KomaduQueue_Notification    0

I see those queues even though I am not running Komadu. I see that following parameters are specified in client-core-messaging/config/komadu.properties:

messaging.exchangename=KomaduExchange
messaging.queuename=KomaduQueue
messaging.routingkey=KomaduKey

but am not sure I see how RabbitMQ sets up these queues? I didn't find any information on configuring RabbitMQ server in documentation to set these queues explicitly. 

#!/usr/bin/env python
import pika

connection = pika.BlockingConnection(pika.ConnectionParameters('<server_IP>'))
channel = connection.channel()
channel.basic_publish(exchange='',
                      routing_key='hello', #use KomaduQueue here
                      body='Hello World!')

An example of using messaging client that addresses above questions would be helpful, but any other info will be appreciated.

Thanks!

Sid

idea-launch-lab commented 9 years ago

I was able to partly figure out second part of my question above (namely, how to forward messages to Komadu via RabbitMQ). Here is my sample sender python code:

#!/usr/bin/env python                                                                                                                                       
import pika
import logging
logging.basicConfig()

connection = pika.BlockingConnection (pika.ConnectionParameters (host='localhost'))
channel = connection.channel ()

queueName    = 'KomaduQueue_Notification'
exchangeName = 'KomaduExchange_Notification'
routingKey   = 'KomaduQueue_Notification'

#channel.queue_declare(queue = queueName)                                                                                                                   

sampleNotificationFilePath = '/home/cluster/prov-components/komadu-1.0/client-core-messaging/samples/agent_agent.xml'

data = ''
with open (sampleNotificationFilePath, "r") as myfile:
    data=myfile.read().replace('\n', '')

print 'data: ', data

channel.basic_publish(exchange = '',
                      routing_key = routingKey,
                      body=data)

print " [x] Sent 'Hello World!'"

Running above script produces message ADD_AGENT_AGENT_RELATION on Komadu console, which I assume means that a notification was successfully inserted.

Clarification of first question above (namely, how does rabbit set up various queues) and any other info would still be helpful. Thanks.

gabrielzhouquan commented 9 years ago

Hi Sid,

It is a good question!

In the configuration, we have the KomaduQueueName which can be set up.

And for messaging bus, a client send a notification and should get a response. So we append several postfix to KomaduQueueName("Request, “Response”, “_Notification”) to set up 3 queues to handle incoming queries and notifications.

{KomaduQueueName}+”Request” — Queue to receive incoming query request {KomaduQueueName}+”Response” — Queue to send back query response {KomaduQueueName}+”_Notification” — Queue to receive notification and send back response

Please let me if this clears your question.

For templates of notifications, you can find from this url:

https://github.com/Data-to-Insight-Center/komadu/tree/master/client-core-messaging/samples https://github.com/Data-to-Insight-Center/komadu/tree/master/client-core-messaging/samples

Please let me know if you have any questions.

Best, ———————————————————————————

Quan (Gabriel) Zhou Email:quzhou@umail.iu.edu Phone:(812)-361-5159 Ph.D Student, Computer Science, Data To Insight Center, School of Informatics and Computing, Indiana University Bloomington

On Aug 24, 2015, at 3:55 PM, sid thakur notifications@github.com wrote:

@gabrielzhouquan https://github.com/gabrielzhouquan I have some more questions about using Komadu through RabbitMQ-based messaging client. I am able to run RabbitMQ server and have some understanding of how to create and run producer/consumer from my application (e.g., using AMQP for sending / receiving messages from RabbitMQ).

I am running RabbitMQ in a VM (co-hosted with Komadu and MySql DB). When I list queues, here's what I see: $ sudo rabbitmqctl list_queues Listing queues ... KomaduQueueQueryRequest 0 KomaduQueueQueryResponse 0 KomaduQueue_Notification 0 I see those queues even though I am not running Komadu. I see that following parameters are specified in client-core-messaging/config/komadu.properties:

messaging.exchangename=KomaduExchange messaging.queuename=KomaduQueue messaging.routingkey=KomaduKey but am not sure I see how RabbitMQ sets up these queues? I didn't find any information on configuring RabbitMQ server in documentation to set these queues explicitly.

Assuming that RabbitMQ has queues listening on KomaduQueue, how do I go about using that queue?

For sending prov notifications, do I send a notification message to RabbitMQ on KomaduQueue_Notification queue? What will/should happen then? How is the message forwarded and ingested by Komadu from thereon?

What content should I be sending in body of messages sent to RabbitMQ's KomaduQueue? I am assuming that I would need to do the following as shown on RabbitMQ manual https://www.rabbitmq.com/tutorials/tutorial-one-python.html to send message to KomaduQueue:

!/usr/bin/env python

import pika

connection = pika.BlockingConnection(pika.ConnectionParameters('')) channel = connection.channel() channel.basic_publish(exchange='', routing_key='hello', #use KomaduQueue here body='Hello World!') Similar question regarding query and KomaduQueueQueryRequest/Response An example of using messaging client that addresses above questions would be helpful, but any other info will be appreciated.

Thanks!

Sid

— Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-134357766.

idea-launch-lab commented 9 years ago

Gabriel, Thanks for your clarification. That is helpful information in figuring out internals of the plumbing. Not urgent, but I am curious to know where is RabbitMQ picking up these config parameters? Are they set in RabbitMQ's config file when Komdu is built? I ask because these queues are alive when Rabbit is run alone (Komadu server is not started). I don't remember setting the queue/channel names explicitly when I built Rabbit.

Again, thanks for being so responsive to my questions :)

idea-launch-lab commented 9 years ago

Here are read outs from my console on queues and exchanges in Rabbit that appear by default (even when komadu server is not running):

[cluster@mesos-stag-1 ~]$ sudo rabbitmqctl list_queues 
Listing queues ...
KomaduQueueQueryRequest 0
KomaduQueueQueryResponse    0
KomaduQueue_Notification    0
[cluster@mesos-stag-1 ~]$ sudo rabbitmqctl list_exchanges 
Listing exchanges ...
    direct
KomaduExchangeQueryRequest  direct
KomaduExchangeQueryResponse direct
KomaduExchange_Notification direct
amq.direct  direct
amq.fanout  fanout
amq.headers headers
amq.match   headers
amq.rabbitmq.log    topic
amq.rabbitmq.trace  topic
amq.topic   topic
[cluster@mesos-stag-1 ~]$
gabrielzhouquan commented 9 years ago

Hi Sid,

This config file is picked up every time when RabbitMQ is started.

You can view Rabbit MQ as a daemon which is running forever to take in notifications and send out response. And that’s why we call this method as Komadu Standalone server which is running all by its own. And all the messages exchange is done by the AMQP daemon.

Hope this answers your questions!

Best,

———————————————————————————

Quan (Gabriel) Zhou Email:quzhou@umail.iu.edu Phone:(812)-361-5159 Ph.D Student, Computer Science, Data To Insight Center, School of Informatics and Computing, Indiana University Bloomington

On Aug 25, 2015, at 2:42 PM, sid thakur notifications@github.com wrote:

Here are read outs from my console on queues and exchanges in Rabbit that appear by default (even when komadu server is not running):

[cluster@mesos-stag-1 ~]$ sudo rabbitmqctl list_queues Listing queues ... KomaduQueueQueryRequest 0 KomaduQueueQueryResponse 0 KomaduQueue_Notification 0 [cluster@mesos-stag-1 ~]$ sudo rabbitmqctl list_exchanges Listing exchanges ... direct KomaduExchangeQueryRequest direct KomaduExchangeQueryResponse direct KomaduExchange_Notification direct amq.direct direct amq.fanout fanout amq.headers headers amq.match headers amq.rabbitmq.log topic amq.rabbitmq.trace topic amq.topic topic [cluster@mesos-stag-1 ~]$ — Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-134697531.

idea-launch-lab commented 9 years ago

Great, thank you.

idea-launch-lab commented 9 years ago

Hi Gabriel, Could you please provide some clarifications on following two questions. Thanks.

(1) Is it possible to log provenance notification messages using JSON format instead of the default XML schema provided in komad-1.0/client-core-messaging/config/komadu-ingest/query_schema.xsd?

(2) Komadu is able to ingest notifications, but when I see the log file komadu.log I see error message unable to process raw notification in print out below (see after contents of the notification). What is happening there?

01 Sep 2015 11:51:08,122 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: One Message received]
<kom:addAgentAgentRelationship xmlns:kom="http://komadu.d2i.indiana.edu">
  <kom:delegateAgent>
    <kom:userAgent>
      <kom:agentID>agent1_58</kom:agentID>
      <kom:attributes>
        <kom:attribute>
          <kom:property>phone</kom:property>
          <kom:value>8123334455</kom:value>
        </kom:attribute>
        <kom:attribute>
          <kom:property>address</kom:property>
          <kom:value>10th Street, Bloomington, IN, 47408</kom:value>
        </kom:attribute>
      </kom:attributes>
      <kom:fullName>Isuru Suriarachchi</kom:fullName>
      <kom:affiliation>testaff</kom:affiliation>
      <kom:email>isuriara@indiana.edu</kom:email>
    </kom:userAgent>
    <kom:type>PERSON</kom:type>
  </kom:delegateAgent>
  <kom:responsibleAgent>
    <kom:userAgent>
      <kom:agentID>agent2_58</kom:agentID>
      <kom:attributes>
        <kom:attribute>
          <kom:property>phone</kom:property>
          <kom:value>8123337788</kom:value>
        </kom:attribute>
        <kom:attribute>
          <kom:property>address</kom:property>
          <kom:value>15th Street, Bloomington, IN, 47408</kom:value>
        </kom:attribute>
      </kom:attributes>
      <kom:fullName>John Fraser</kom:fullName>
      <kom:affiliation>testaff2</kom:affiliation>
      <kom:email>john@indiana.edu</kom:email>
    </kom:userAgent>
    <kom:type>PERSON</kom:type>
  </kom:responsibleAgent>
  <kom:delegation>
    <kom:delegateAgentID>agent1_58</kom:delegateAgentID>
    <kom:responsibleAgentID>agent2_58</kom:responsibleAgentID>
    <kom:attributes>
      <kom:attribute>
        <kom:property>type</kom:property>
        <kom:value>contract</kom:value>
      </kom:attribute>
    </kom:attributes>
  </kom:delegation>
</kom:addAgentAgentRelationship>
01 Sep 2015 11:51:08,156 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: Listening to Messaging Sy
stem]
01 Sep 2015 11:51:08,815 [AsynProc-1] ERROR edu.indiana.d2i.komadu.ingest.db.BaseDBIngester  -Unable to process raw notification with internalID 10
edu.indiana.d2i.komadu.ingest.IngestException: Max retries 10 for the lock reached for the object: agent1_58
    at edu.indiana.d2i.komadu.util.KomaduUtils.manageDBLock(KomaduUtils.java:100)
    at edu.indiana.d2i.komadu.ingest.db.BaseDBIngesterImplementer.addNewAgent(BaseDBIngesterImplementer.java:730)
    at edu.indiana.d2i.komadu.ingest.db.BaseDBIngesterImplementer.storeAgentAgentRelationship(BaseDBIngesterImplementer.java:269)
    at edu.indiana.d2i.komadu.ingest.db.BaseDBIngester.processNotifications(BaseDBIngester.java:133)
    at edu.indiana.d2i.komadu.ingest.AsyncRawNotificationProcessor.run(AsyncRawNotificationProcessor.java:99)
    at java.lang.Thread.run(Thread.java:745)
gabrielzhouquan commented 9 years ago

Hi Sid,

For question 1,

Current Komadu version only supports xml format. Sorry for the inconvenience.

For quesiton2,

The bug should happen due the sql database lock Could you try the following command using root account:

GRANT ALL ON komadu.* TO 'komaduuser'@'localhost' IDENTIFIED BY 'komadupwd';

GRANT SELECT ON mysql.proc TO 'komaduuser'@'localhost';

Many thanks!

Best,

Quan Zhou

On Sep 1, 2015, at 11:56 AM, sid thakur notifications@github.com wrote:

Max retries 10 for the lock reached for the object

idea-launch-lab commented 9 years ago

@gabrielzhouquan Thanks for the clarifications.

A question about configuring Komadu to accept data from different types of clients. We'd like to use a single instance of Komadu messaging service to serve three different environments: prod environment, dev/staging env, and local dev. Last one being localhost on a developer machine. We'd like all three to send messages to the same server, but with different identifiers.

My preliminary guess is that this could be achieved by starting komadu messaging service using different config files - one each for the three types of clients, and having different credentials for RabbitMQ and different queue and channel names.

Is this what you all would recommend or is there a better/different way of achieving multi-client support?

gabrielzhouquan commented 9 years ago

Hi Sid,

For this question about multi-client support, two options could be available:

1/ Setting up multiple messaging client: As you mentioned, one each has different queue and channel names in order to receive notifications and queries for three environments: prod, dev/staging and local;

2/ Using prefix to classify notification/query: For each notification, you could specify agent/entity/activity ID with a prefix as prod, dev and local_. For queries according to different environments, prefix can be used for classification.

Hope this can help! Please let me know if this makes sense to you.

Best,

Quan Zhou

On Sep 1, 2015, at 2:51 PM, sid thakur notifications@github.com wrote:

@gabrielzhouquan https://github.com/gabrielzhouquan Thanks for the clarifications.

A question about configuring Komadu to accept data from different types of clients. We'd like to use a single instance of Komadu messaging service to serve three different environments: prod environment, dev/staging env, and local dev. Last one being localhost on a developer machine. We'd like all three to send messages to the same server, but with different identifiers.

My preliminary guess is that this could be achieved by starting komadu messaging service using different config files - one each for the three types of clients, and having different credentials for RabbitMQ and different queue and channel names.

Is this what you all would recommend or is there a better/different way of achieving multi-client support?

— Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-136826265.

idea-launch-lab commented 9 years ago

Okay, thanks @gabrielzhouquan Re. option 2 above, did you imply using a single messaging server but different message identifiers? If so, I'm wondering if serving the three different environments (prod, dev, local) through one messaging server would adversely affect performance.

idea-launch-lab commented 9 years ago

Hi @gabrielzhouquan I am having some trouble getting query response queue. I am using AMQP in node.js to send prov messages to the messaging server. I am able to successfully send messages on the queue KomaduQueue_Notification and KomaduQueueQueryRequest. However, I don't see any response being placed on the KomaduQueueQueryRequest queue.

If I check the queues set up by RabbitMQ after sending a query request (below), I see a zero against the query response queue. I don't think I have any listener attached to the response queue that would remove the message, so not sure what's going on. Thanks. 

$ sudo rabbitmqctl list_queues
Listing queues ...
KomaduQueueQueryRequest 0
KomaduQueueQueryResponse    0
KomaduQueue_Notification    0

Below is code I am using:

// Sender
    var connection = amqp.createConnection (
        {
        host: '127.0.0.1',
        port: 5672,
        vhost: '/',
        login: '<username>',
        password: '<password>',
        connectionTimeout: 10000, 
        authMechanism: 'AMQPLAIN',
        vhost: '/',
        noDelay: true,
        ssl: { 
            enabled : false
        }
        }
    );

    // wait for connection to become established
    connection.on ('ready', function (q) {
        //q.bind (queueName);
        console.log ('[AMQP] connection ready..');

        var sendMessage = function(connection, queue_name, payload) {
        var encoded_payload = payload;
                console.log ('publish...');
        connection.publish(queue_name, encoded_payload);
        }
        sendMessage (connection, 'KomaduQueueQueryRequest', provDoc);
    });
// Receiver
    var connection = amqp.createConnection (
    {
        host: '127.0.0.1',
        port: 5672,
        vhost: '/',
        login: '<username>',
        password: '<password>',
        connectionTimeout: 10000, 
        authMechanism: 'AMQPLAIN',
        vhost: '/',
        noDelay: true,
        ssl: { 
        enabled : false
        }
    }
    );

    connection.on('ready', function () {
    connection.queue ("KomaduQueueQueryResponse", function (queue){
        queue.bind ('#'); 
        queue.subscribe (function (message) {
        var encoded_payload = unescape (message.data)
        var payload = JSON.parse (encoded_payload)
        console.log ('Recieved a message:')
        console.log (payload)
        });
    });
    });
gabrielzhouquan commented 9 years ago

Could you please print out the console log output from your node script?

Thx!

On Sep 4, 2015, at 1:52 PM, sid thakur notifications@github.com wrote:

Hi @gabrielzhouquan https://github.com/gabrielzhouquan I am having some trouble getting query response queue. I am using AMQP in node.js to send prov messages to the messaging server. I am able to successfully send messages on the queue KomaduQueue_Notification and KomaduQueueQueryRequest. However, I don't see any response being placed on the KomaduQueueQueryRequest queue.

If I check the queues set up by RabbitMQ after sending a query request (below), I see a zero against the query response queue. I don't think I have any listener attached to the response queue that would remove the message, so not sure what's going on. Thanks.

$ sudo rabbitmqctl list_queues Listing queues ... KomaduQueueQueryRequest 0 KomaduQueueQueryResponse 0 KomaduQueue_Notification 0 Below is code I am using:

// Sender var connection = amqp.createConnection ( { host: '127.0.0.1', port: 5672, vhost: '/', login: '', password: '', connectionTimeout: 10000, authMechanism: 'AMQPLAIN', vhost: '/', noDelay: true, ssl: { enabled : false } } );

// wait for connection to become established
connection.on ('ready', function (q) {
    //q.bind (queueName);
    console.log ('[AMQP] connection ready..');

    var sendMessage = function(connection, queue_name, payload) {
    var encoded_payload = payload;
    connection.publish(queue_name, encoded_payload);
    }
    sendMessage (connection, 'KomaduQueueQueryRequest', provDoc);
});

// Receiver var connection = amqp.createConnection ( { host: '127.0.0.1', port: 5672, vhost: '/', login: '', password: '', connectionTimeout: 10000, authMechanism: 'AMQPLAIN', vhost: '/', noDelay: true, ssl: { enabled : false } } );

connection.on('ready', function () {
connection.queue ("KomaduQueueQueryResponse", function (queue){
    queue.bind ('#'); 
    queue.subscribe (function (message) {
    var encoded_payload = unescape (message.data)
    var payload = JSON.parse (encoded_payload)
    console.log ('Recieved a message:')
    console.log (payload)
    });
});
});

— Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-137805248.

idea-launch-lab commented 9 years ago

I don't see a log or console output. The subscriber in the Receiver part of code binds to the queue but I don't think the subscribe function is called because the komadu response queue is always empty. Thanks.

gabrielzhouquan commented 9 years ago

Hi Sid,

In yr senders code, when amqp connection is ready, console will have console.log ('[AMQP] connection ready..'); If console output is not generated, it could be that the amqp connection is not configured properly.

You may refer to this link :https://github.com/squaremo/amqp.node/tree/master/examples/tutorials https://github.com/squaremo/amqp.node/tree/master/examples/tutorials It provides several good examples.

On Sep 4, 2015, at 2:40 PM, sid thakur notifications@github.com wrote:

I don't see a log or console output. The subscriber in the Receiver part of code binds to the queue but I don't think the subscribe function is called because the komadu response queue is always empty. Thanks.

— Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-137816439.

idea-launch-lab commented 9 years ago

Okay, sorry about the confusion. Yes, connection work fine and I see the message '[AMQP] connection ready..' in console output for both sender and receiver. Looks like the receiver is in a polling mode.. [I added a line in receiver within connection.on('ready',.. to see if connection was setup. Below is read-out from console:

[Query sent to KomaduQueueQueryRequest]: <quer:findEntityRequest xmlns:quer="http://komadu.d2i.indiana.edu/query">  <quer:fileName>data1.txt_58</quer:fileName>  <quer:fileOwnerID>jerry112</quer:fileOwnerID>  <quer:fileSize>500</quer:fileSize></quer:findEntityRequest>

2015-09-04T18:52:37.628Z - debug: [doc-db]  [PROV_EMIT] creating connection
2015-09-04T18:52:37.633Z - debug: [doc-db]  [PROV_RECEIVE] creating connection
[AMQP SENDER] connection ready..
publish...
[AMQP RECEIVER] connection ready..
[AMQP RECEIVER] connection ready..
[AMQP RECEIVER] connection ready..
[AMQP RECEIVER] connection ready..
[AMQP RECEIVER] connection ready..
[AMQP RECEIVER] connection ready..
[AMQP RECEIVER] connection ready..
[AMQP RECEIVER] connection ready..
[AMQP RECEIVER] connection ready..
[AMQP RECEIVER] connection ready..
[AMQP RECEIVER] connection ready..
[AMQP RECEIVER] connection ready..
[AMQP RECEIVER] connection ready..
:
idea-launch-lab commented 9 years ago

Hi, I am having trouble while trying to retrieve query response in node.js - amqp setup. I am able to send a query request successfully, and can see it being placed on the queue KomaduQueueQueryRequest. Komadu seems to be consuming the query request - I can see counter for query request queue being decremented. However, I don't see a corresponding response on the queue KomaduQueueQueryResponse. 

connecting to queue: KomaduQueueQueryRequest    Sending query: "query_find_activity.xml"
 [x] Sent '<quer:findActivityRequest xmlns:quer="http://komadu.d2i.indiana.edu/query">  <quer:name>workflow1</quer:name>  <quer:nextActivityID>service1_58</quer:nextActivityID></quer:findActivityRequest>'
 [*] Waiting for messages...

I have a client-side listener attached to the query response queue but it never receives any data.

Here's an Interesting observation: While my node.js client-side listener is polling, I run the command line script in komadu/client-core-messaging/bin/query.sh ... which sends another query request. This time the node.js client-side listener receives some response and the command line query.sh is left hanging. I see following in node.js server stdout:

 [x] Received '<quer:findEntityResponse xmlns:quer="http://komadu.d2i.indiana.edu/query">
  <quer:uniqueFileURIList/>
</quer:findEntityResponse>'

Thanks.

idea-launch-lab commented 9 years ago

@isurues / @gabrielzhouquan I'd appreciate any inputs regd. my last question in this thread. Basically, I am having trouble getting response from query. Thanks!

gabrielzhouquan commented 9 years ago

Hi Sid,

I missed the last thread.

From your description, AMQP client can get query response when you issue the query from JAVA client. However, AMQP cannot get response when you use AMQP client to issue the query.

It probably is caused by connection from AMQP client to Komadu Request queue.

If possible, could you share me two things?

1/ Your AMQP client codes; 2/ Under {komadu root}/service-core/, could you share the komadu.log to me?

Many thanks!

Best,

Quan Zhou

On Sep 22, 2015, at 11:22 AM, sid thakur notifications@github.com wrote:

@isurues https://github.com/isurues / @gabrielzhouquan https://github.com/gabrielzhouquan I'd appreciate any inputs regd. my last question in above thread. Thanks!

— Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-142322457.

idea-launch-lab commented 9 years ago

Hi, Great, thanks! Is there a way to send you the log file - it is ~1.5M. I can post the code here. Sid

gabrielzhouquan commented 9 years ago

Hi Sid,

thanks!

Could you attach your logs to the issues in GitHub? (drag & drop should work)

best,

Gabriel Zhou

Could you attach your log to

On Sep 22, 2015, at 2:30 PM, sid thakur notifications@github.com wrote:

Hi, Great, thanks! Is there a way to send you the log file - it is ~1.5M. I can post the code here. Sid

— Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-142374111.

idea-launch-lab commented 9 years ago

AMQP code used to send messages to komadu server prov_emit () and receive messages prov_receive () Note that I user text version of sample XML notification files and construct a single-line text before sending it as message on KomaduQueue_Notification

var amqp      = require('amqplib/callback_api');

// Send message to komadu server
exports.prov_emit = function (data, queueName) {

    // form provenance document
    var provDoc = ''; 

    // set path and name of data file
    var filePath = './data/';
    if (queueName === 'KomaduQueue_Notification') {
    filePath += 'activity_activity.txt';
    }
    else if (queueName === 'KomaduQueueQueryRequest') {
    filePath += 'query_find_activity.txt';
    }

    var array = fs.readFileSync (filePath).toString ().split ("\n");

    for(i in array) {
    provDoc += '' + array [i];
    }
    //console.log (provDoc);

    // for querying Komadu use queue name 'KomaduQueueQueryRequest'
    // for inserting provenance notification use queue name 'KomaduQueue_Notification'

    // error handler function
    function bail(err, conn) {
    logger.debug (err);
    if (conn) {
        conn.close(function() {
        // DO NOTHING
        });
    }
    }

    function on_connect(err, conn) {
    console.log ('connecting to queue: ' + queueName);
    if (err !== null) 
        return bail(err);

    var q = queueName;
    var msg = provDoc; //'Hello World!';

    function on_channel_open(err, ch) {

        if (err !== null) {
        return bail(err, conn);
        }

        ch.assertQueue(q, {}, function(err, ok) {

        if (err !== null) return bail(err, conn);

        ch.sendToQueue(q, new Buffer(msg));
        console.log(" [x] Sent '%s'", msg);
        logger.debug ('[PROV_SEND] Provenance notification sent to Komadu server');

        // close channel and connection
        ch.close(function() {
            console.info ("Closing connection...");
            conn.close(); 
        });
        });
    }

    conn.createChannel (on_channel_open);
    }

    // create an amqp connection through tunnel
    amqp.connect('amqp://userid:userpass@127.0.0.1:5672', on_connect);
};

// Receive message from Komadu
exports.prov_receive = function () {
    var queueName = 'KomaduQueueQueryResponse';

    // error handler function
    function bail(err, conn) {
    logger.debug (err);
    if (conn) {
        conn.close(function() {
        // DO NOTHING
        });
    }
    }

    function on_connect(err, conn) {
    // check if connection failed
    if (err !== null) {
        return bail(err);
    }

    process.once('SIGINT', function() {
        conn.close();
    });

    var q = queueName;

    function on_channel_open(err, ch) {

        ch.assertQueue(q, {}, function(err, ok) {

        if (err !== null) {
            return bail(err, conn);
        }

        // message callback
        ch.consume(q, function(msg) {
            console.log(" [x] Received '%s'", msg.content.toString());
            logger.debug ('[PROV_RECEIVE] Provenance notification received from Komadu server');
        }, {noAck: true}, function(_consumeOk) { // consume callback
            console.log(' [*] Waiting for messages. To exit press CTRL+C');
        });
        });
    }
    conn.createChannel(on_channel_open);
    }

    amqp.connect('amqp://userid:userpass@127.0.0.1:5672', on_connect);
};
gabrielzhouquan commented 9 years ago

Hi Sid,

Many thanks!

Let me run a few tests and will get back to you ASAP.

best,

Quan Zhou

On Sep 22, 2015, at 2:36 PM, sid thakur notifications@github.com wrote:

AMQP code used to send messages to komadu server prov_emit () and receive messages prov_receive () Note that I user text version of sample XML notification files and construct a single-line text before sending it as message on KomaduQueue_Notification

var amqp = require('amqplib/callback_api');

// Send message to komadu server exports.prov_emit = function (data, queueName) {

// form provenance document
var provDoc = ''; 

// set path and name of data file
var filePath = './data/';
if (queueName === 'KomaduQueue_Notification') {
filePath += 'activity_activity.txt';
}
else if (queueName === 'KomaduQueueQueryRequest') {
filePath += 'query_find_activity.txt';
}

var array = fs.readFileSync (filePath).toString ().split ("\n");

for(i in array) {
provDoc += '' + array [i];
}
//console.log (provDoc);

// for querying Komadu use queue name 'KomaduQueueQueryRequest'
// for inserting provenance notification use queue name 'KomaduQueue_Notification'

// error handler function
function bail(err, conn) {
logger.debug (err);
if (conn) {
    conn.close(function() {
    // DO NOTHING
    });
}
}

function on_connect(err, conn) {
console.log ('connecting to queue: ' + queueName);
if (err !== null) 
    return bail(err);

var q = queueName;
var msg = provDoc; //'Hello World!';

function on_channel_open(err, ch) {

    if (err !== null) {
    return bail(err, conn);
    }

    ch.assertQueue(q, {}, function(err, ok) {

    if (err !== null) return bail(err, conn);

    ch.sendToQueue(q, new Buffer(msg));
    console.log(" [x] Sent '%s'", msg);
    logger.debug ('[PROV_SEND] Provenance notification sent to Komadu server');

    // close channel and connection
    ch.close(function() {
        console.info ("Closing connection...");
        conn.close(); 
    });
    });
}

conn.createChannel (on_channel_open);
}

// create an amqp connection through tunnel
amqp.connect('amqp://userid:userpass@127.0.0.1:5672', on_connect);

};

// Receive message from Komadu exports.prov_receive = function () { var queueName = 'KomaduQueueQueryResponse';

// error handler function
function bail(err, conn) {
logger.debug (err);
if (conn) {
    conn.close(function() {
    // DO NOTHING
    });
}
}

function on_connect(err, conn) {
// check if connection failed
if (err !== null) {
    return bail(err);
}

process.once('SIGINT', function() {
    conn.close();
});

var q = queueName;

function on_channel_open(err, ch) {

    ch.assertQueue(q, {}, function(err, ok) {

    if (err !== null) {
        return bail(err, conn);
    }

    // message callback
    ch.consume(q, function(msg) {
        console.log(" [x] Received '%s'", msg.content.toString());
        logger.debug ('[PROV_RECEIVE] Provenance notification received from Komadu server');
    }, {noAck: true}, function(_consumeOk) { // consume callback
        console.log(' [*] Waiting for messages. To exit press CTRL+C');
    });
    });
}
conn.createChannel(on_channel_open);
}

amqp.connect('amqp://userid:userpass@127.0.0.1:5672', on_connect);

}; — Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-142377507.

idea-launch-lab commented 9 years ago

Looks like I can only attach a PNG/GIF/JPG file, and not a plain text (log) file.

idea-launch-lab commented 9 years ago

Btw, just so you know, I am creating an ssh tunnel to the server that is running Komadu web service. Once the tunnel is active I can send messages to port 5672 as if it is on localhost. Below is a read-out of netstat command showing 5672 attached to localhost.

[sthakur@localhost skylr]$ netstat -ntpl
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:5672          0.0.0.0:*               LISTEN      26929/ssh           
:
gabrielzhouquan commented 9 years ago

Hi Sid,

Got is.

In your Komadu.log,

Could you find related output as below?

22 Sep 2015 13:51:05,570 [Thread-1] INFO edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable -[Karma server: Query] quer:informationDetailLevelFINE/quer:informationDetailLevel quer:entityURIfile://foo/bar/data2.txt_58/quer:entityURI quer:entityTypeFILE/quer:entityType /quer:getEntityGraphRequest 22 Sep 2015 13:51:05,570 [Thread-1] INFO edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable -[Karma server: Query Response Routing Key] 9e9d94cc-733a-4626-8167-705285783673 22 Sep 2015 13:51:05,637 [Thread-1] INFO edu.indiana.d2i.komadu.query.graph.GraphGenerator -Query has not been cached. 22 Sep 2015 13:51:06,201 [Thread-1] INFO edu.indiana.d2i.komadu.query.graph.GraphGenerator -Caching graph. 22 Sep 2015 13:51:06,272 [Thread-1] INFO edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable -[Komadu server: Send Query Response to Client] 22 Sep 2015 13:51:06,277 [Thread-1] INFO edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable -[Komadu server: Listening Queries from Messaging System]

This is the output log from my AMQP client. If you find related log output, please share to me and I could see the exceptions and errors over there.

many thanks!

best,

Quan Zhou

On Sep 22, 2015, at 2:43 PM, sid thakur notifications@github.com wrote:

Btw, just so you know, I am creating an ssh tunnel to the server that is running Komadu web service. Once the tunnel is active I can send messages to port 5672 as if it is on localhost. Below is a read-out of netstat command showing 5672 attached to localhost.

[sthakur@localhost skylr]$ netstat -ntpl (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:5672 0.0.0.0:* LISTEN 26929/ssh
: — Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-142380886.

idea-launch-lab commented 9 years ago

Okay, below is a snippet that contained 'QueryRecieveRunnable` from last few attempts. I don't see data for more recent attempts though. Is there a way to reset the log before trying to query?

01 Sep 2015 09:42:59,701 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Listening Queries from Messaging System]
01 Sep 2015 09:44:59,422 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Starting Notification Deamon [0]  for receiving notifications from Client.
01 Sep 2015 09:44:59,425 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Notification Deamon [0] Started.
01 Sep 2015 09:44:59,425 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Starting Query Deamon [0] for receiving queries from Client.
01 Sep 2015 09:44:59,425 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: Listening to Messaging System]
01 Sep 2015 09:44:59,426 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Query Deamon [0] Started.
01 Sep 2015 09:44:59,426 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Listening Queries from Messaging System]
01 Sep 2015 09:48:08,918 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Starting Notification Deamon [0]  for receiving notifications from Client.
01 Sep 2015 09:48:08,920 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Notification Deamon [0] Started.
01 Sep 2015 09:48:08,920 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Starting Query Deamon [0] for receiving queries from Client.
01 Sep 2015 09:48:08,921 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: Listening to Messaging System]
01 Sep 2015 09:48:08,921 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Query Deamon [0] Started.
01 Sep 2015 09:48:08,922 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Listening Queries from Messaging System]
01 Sep 2015 09:51:24,838 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Starting Notification Deamon [0]  for receiving notifications from Client.
01 Sep 2015 09:51:24,840 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Notification Deamon [0] Started.
01 Sep 2015 09:51:24,840 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Starting Query Deamon [0] for receiving queries from Client.
01 Sep 2015 09:51:24,840 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Query Deamon [0] Started.
01 Sep 2015 09:51:24,842 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: Listening to Messaging System]
01 Sep 2015 09:51:24,845 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Listening Queries from Messaging System]
01 Sep 2015 09:52:23,066 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Starting Notification Deamon [0]  for receiving notifications from Client.
01 Sep 2015 09:52:23,069 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Notification Deamon [0] Started.
01 Sep 2015 09:52:23,069 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Starting Query Deamon [0] for receiving queries from Client.
01 Sep 2015 09:52:23,070 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: Listening to Messaging System]
01 Sep 2015 09:52:23,070 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Query Deamon [0] Started.
01 Sep 2015 09:52:23,071 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Listening Queries from Messaging System]
01 Sep 2015 10:11:39,752 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Starting Notification Deamon [0]  for receiving notifications from Client.
01 Sep 2015 10:11:39,754 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Notification Deamon [0] Started.
01 Sep 2015 10:11:39,754 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Starting Query Deamon [0] for receiving queries from Client.
01 Sep 2015 10:11:39,755 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: Listening to Messaging System]
01 Sep 2015 10:11:39,755 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Query Deamon [0] Started.
01 Sep 2015 10:11:39,756 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Listening Queries from Messaging System]
gabrielzhouquan commented 9 years ago

Hi Sid,

Sure.

rm -rf {service-core}/komadu.log //remove log {service-core}/KomaduServer.sh {service-core]/config/komadu.properties //restart komadu server

Many thanks!

Best,

Gabriel Zhou

On Sep 22, 2015, at 3:01 PM, sid thakur notifications@github.com wrote:

Okay, below is a snippet that contained 'QueryRecieveRunnable` from last few attempts. I don't see data for more recent attempts though. Is there a way to reset the log before trying to query?

01 Sep 2015 09:42:59,701 [Thread-1] INFO edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable -[Komadu server: Listening Queries from Messaging System] 01 Sep 2015 09:44:59,422 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Starting Notification Deamon [0] for receiving notifications from Client. 01 Sep 2015 09:44:59,425 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Notification Deamon [0] Started. 01 Sep 2015 09:44:59,425 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Starting Query Deamon [0] for receiving queries from Client. 01 Sep 2015 09:44:59,425 [Thread-0] INFO edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable -[Komadu server: Listening to Messaging System] 01 Sep 2015 09:44:59,426 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Query Deamon [0] Started. 01 Sep 2015 09:44:59,426 [Thread-1] INFO edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable -[Komadu server: Listening Queries from Messaging System] 01 Sep 2015 09:48:08,918 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Starting Notification Deamon [0] for receiving notifications from Client. 01 Sep 2015 09:48:08,920 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Notification Deamon [0] Started. 01 Sep 2015 09:48:08,920 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Starting Query Deamon [0] for receiving queries from Client. 01 Sep 2015 09:48:08,921 [Thread-0] INFO edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable -[Komadu server: Listening to Messaging System] 01 Sep 2015 09:48:08,921 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Query Deamon [0] Started. 01 Sep 2015 09:48:08,922 [Thread-1] INFO edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable -[Komadu server: Listening Queries from Messaging System] 01 Sep 2015 09:51:24,838 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Starting Notification Deamon [0] for receiving notifications from Client. 01 Sep 2015 09:51:24,840 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Notification Deamon [0] Started. 01 Sep 2015 09:51:24,840 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Starting Query Deamon [0] for receiving queries from Client. 01 Sep 2015 09:51:24,840 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Query Deamon [0] Started. 01 Sep 2015 09:51:24,842 [Thread-0] INFO edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable -[Komadu server: Listening to Messaging System] 01 Sep 2015 09:51:24,845 [Thread-1] INFO edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable -[Komadu server: Listening Queries from Messaging System] 01 Sep 2015 09:52:23,066 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Starting Notification Deamon [0] for receiving notifications from Client. 01 Sep 2015 09:52:23,069 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Notification Deamon [0] Started. 01 Sep 2015 09:52:23,069 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Starting Query Deamon [0] for receiving queries from Client. 01 Sep 2015 09:52:23,070 [Thread-0] INFO edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable -[Komadu server: Listening to Messaging System] 01 Sep 2015 09:52:23,070 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Query Deamon [0] Started. 01 Sep 2015 09:52:23,071 [Thread-1] INFO edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable -[Komadu server: Listening Queries from Messaging System] 01 Sep 2015 10:11:39,752 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Starting Notification Deamon [0] for receiving notifications from Client. 01 Sep 2015 10:11:39,754 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Notification Deamon [0] Started. 01 Sep 2015 10:11:39,754 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Starting Query Deamon [0] for receiving queries from Client. 01 Sep 2015 10:11:39,755 [Thread-0] INFO edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable -[Komadu server: Listening to Messaging System] 01 Sep 2015 10:11:39,755 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Query Deamon [0] Started. 01 Sep 2015 10:11:39,756 [Thread-1] INFO edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable -[Komadu server: Listening Queries from Messaging System] — Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-142386600.

idea-launch-lab commented 9 years ago

Okay, thanks. Just one difference, I have a log in {service-core}/bin/komadu.log There is no {service-core}/komadu.log file. Does that matter?

gabrielzhouquan commented 9 years ago

Hi Sid,

The log is generated in the directory where you run KomaduServer.sh.

It is the same. :)

best,

Gabriel Zhou

On Sep 22, 2015, at 3:07 PM, sid thakur notifications@github.com wrote:

Okay, thanks. Just one difference, I have a log in {service-core}/bin/komadu.log There is no {service-core}/komadu.log file. Does that matter?

— Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-142388035.

idea-launch-lab commented 9 years ago

I realized after posted the note :) thanks. Sending you the log in a bit...

idea-launch-lab commented 9 years ago

I don't see a komadu.log file being written out any long in {service-core}/. folder. I ran ./client-core-messaging/bin/sendNotification.sh too, but no log file is being generated.

gabrielzhouquan commented 9 years ago

Hi Sid,

Under which dir do you start the server? If you still can’t find, Could you restart the Komadu server?

Best,

Quan Zhou

On Sep 22, 2015, at 3:17 PM, sid thakur notifications@github.com wrote:

I don't see a komadu.log file being written out any long in {service-core}/. folder. I ran ./client-core-messaging/bin/sendNotification.sh too, but no log file is being generated.

— Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-142390185.

idea-launch-lab commented 9 years ago

I am starting the server while inside the folder komadu-1.0/.

[cluster@server1 komadu-1.0]$ ls -l
total 56
drwxrwxr-x  5 cluster cluster  4096 Jul 22 13:15 axis2-client-core
drwxrwxr-x 12 cluster cluster  4096 Sep 15 16:07 client-core-messaging
drwxrwxr-x  2 cluster cluster  4096 Jul 24  2014 docs
-rw-rw-r--  1 cluster cluster 16700 Sep 22 15:23 komadu.log
-rw-rw-r--  1 cluster cluster  7804 Jul 24  2014 pom.xml
-rw-rw-r--  1 cluster cluster  3081 Jul 24  2014 README.md
drwxrwxr-x  6 cluster cluster  4096 Jul 22 13:17 service-core
drwxrwxr-x  4 cluster cluster  4096 Jul 22 13:15 service-core-aar
drwxrwxr-x  5 cluster cluster  4096 Jul 24  2014 visualization

[cluster@server1 komadu-1.0]$ ./service-core/bin/KomaduServer.sh ./service-core/config/komadu.properties

$
idea-launch-lab commented 9 years ago

Okay, I found it in komadu root folder ./komadu-1.0/.

22 Sep 2015 15:13:29,764 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Starting Notification Deamon [0]  for receiving notifications from Client.
22 Sep 2015 15:13:29,766 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Notification Deamon [0] Started.
22 Sep 2015 15:13:29,766 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Starting Query Deamon [0] for receiving queries from Client.
22 Sep 2015 15:13:29,767 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: Listening to Messaging System]
22 Sep 2015 15:13:29,767 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Query Deamon [0] Started.
22 Sep 2015 15:13:29,768 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Listening Queries from Messaging System]
22 Sep 2015 15:13:36,676 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: One Query received]
<quer:findActivityRequest xmlns:quer="http://komadu.d2i.indiana.edu/query">  <quer:name>workflow1</quer:name>  <quer:nextActivityID>service1_58</quer:nextActivityID></quer:findActivityRequest>
22 Sep 2015 15:13:36,676 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: One Message received]
<kom:addActivityActivityRelationship xmlns:kom="http://komadu.d2i.indiana.edu"> <kom:activity1>   <kom:workflowInformation>     <kom:workflowID>workflow1_58</kom:workflowID>     <kom:workflowNodeID>node1</kom:workflowNodeID>     <kom:timestep>20</kom:timestep>     <kom:attributes>       <kom:attribute>         <kom:property>domain</kom:property>         <kom:value>scientific</kom:value>       </kom:attribute>     </kom:attributes>     <kom:instanceOf>       <kom:instanceOfID>slosh_workflow</kom:instanceOfID>       <kom:version>1.0.0</kom:version>       <kom:creationTime>2014-05-19T13:30:14.706-04:00</kom:creationTime>     </kom:instanceOf>   </kom:workflowInformation>   <kom:location>Azure</kom:location> </kom:activity1> <kom:activity2>   <kom:serviceInformation>     <kom:workflowID>workflow1_58</kom:workflowID>     <kom:workflowNodeID>node1</kom:workflowNodeID>     <kom:timestep>20</kom:timestep>     <kom:attributes>       <kom:attribute>         <kom:property>domain</kom:property>         <kom:value>scientific</kom:value>       </kom:attribute>     </kom:attributes>     <kom:instanceOf>       <kom:instanceOfID>slosh_workflow</kom:instanceOfID>       <kom:version>1.0.0</kom:version>       <kom:creationTime>2014-05-19T13:30:14.729-04:00</kom:creationTime>     </kom:instanceOf>     <kom:serviceID>service1_58</kom:serviceID>   </kom:serviceInformation>   <kom:location>Azure</kom:location> </kom:activity2> <kom:communication>   <kom:informedActivityID>service1_58</kom:informedActivityID>   <kom:informantActivityID>workflow1_58</kom:informantActivityID>   <kom:attributes>     <kom:attribute>       <kom:property>dummy_att</kom:property>       <kom:value>communication_att</kom:value>     </kom:attribute>   </kom:attributes> </kom:communication></kom:addActivityActivityRelationship>
22 Sep 2015 15:13:36,676 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Karma server: Invalid Query Request]

22 Sep 2015 15:13:36,680 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Listening Queries from Messaging System]
22 Sep 2015 15:13:37,208 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: Listening to Messaging System]
22 Sep 2015 15:13:37,681 [AsynProc-0] ERROR edu.indiana.d2i.komadu.ingest.db.BaseDBIngester  -Unable to process raw notification with internalID 52
edu.indiana.d2i.komadu.ingest.IngestException: Max retries 10 for the lock reached for the object: workflow1_58
    at edu.indiana.d2i.komadu.util.KomaduUtils.manageDBLock(KomaduUtils.java:100)
    at edu.indiana.d2i.komadu.ingest.db.BaseDBIngesterImplementer.addNewActivity(BaseDBIngesterImplementer.java:1189)
    at edu.indiana.d2i.komadu.ingest.db.BaseDBIngesterImplementer.storeActivityActivityRelationship(BaseDBIngesterImplementer.java:174)
    at edu.indiana.d2i.komadu.ingest.db.BaseDBIngester.processNotifications(BaseDBIngester.java:123)
    at edu.indiana.d2i.komadu.ingest.AsyncRawNotificationProcessor.run(AsyncRawNotificationProcessor.java:99)
    at java.lang.Thread.run(Thread.java:745)
22 Sep 2015 15:14:03,859 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: One Query received]
<quer:findActivityRequest xmlns:quer="http://komadu.d2i.indiana.edu/query">  <quer:name>workflow1</quer:name>  <quer:nextActivityID>service1_58</quer:nextActivityID></quer:findActivityRequest>
22 Sep 2015 15:14:03,859 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Karma server: Invalid Query Request]

22 Sep 2015 15:14:03,859 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Listening Queries from Messaging System]
22 Sep 2015 15:14:03,867 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: One Message received]
<kom:addActivityActivityRelationship xmlns:kom="http://komadu.d2i.indiana.edu"> <kom:activity1>   <kom:workflowInformation>     <kom:workflowID>workflow1_58</kom:workflowID>     <kom:workflowNodeID>node1</kom:workflowNodeID>     <kom:timestep>20</kom:timestep>     <kom:attributes>       <kom:attribute>         <kom:property>domain</kom:property>         <kom:value>scientific</kom:value>       </kom:attribute>     </kom:attributes>     <kom:instanceOf>       <kom:instanceOfID>slosh_workflow</kom:instanceOfID>       <kom:version>1.0.0</kom:version>       <kom:creationTime>2014-05-19T13:30:14.706-04:00</kom:creationTime>     </kom:instanceOf>   </kom:workflowInformation>   <kom:location>Azure</kom:location> </kom:activity1> <kom:activity2>   <kom:serviceInformation>     <kom:workflowID>workflow1_58</kom:workflowID>     <kom:workflowNodeID>node1</kom:workflowNodeID>     <kom:timestep>20</kom:timestep>     <kom:attributes>       <kom:attribute>         <kom:property>domain</kom:property>         <kom:value>scientific</kom:value>       </kom:attribute>     </kom:attributes>     <kom:instanceOf>       <kom:instanceOfID>slosh_workflow</kom:instanceOfID>       <kom:version>1.0.0</kom:version>       <kom:creationTime>2014-05-19T13:30:14.729-04:00</kom:creationTime>     </kom:instanceOf>     <kom:serviceID>service1_58</kom:serviceID>   </kom:serviceInformation>   <kom:location>Azure</kom:location> </kom:activity2> <kom:communication>   <kom:informedActivityID>service1_58</kom:informedActivityID>   <kom:informantActivityID>workflow1_58</kom:informantActivityID>   <kom:attributes>     <kom:attribute>       <kom:property>dummy_att</kom:property>       <kom:value>communication_att</kom:value>     </kom:attribute>   </kom:attributes> </kom:communication></kom:addActivityActivityRelationship>
22 Sep 2015 15:14:03,894 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: Listening to Messaging System]
22 Sep 2015 15:14:04,708 [AsynProc-1] ERROR edu.indiana.d2i.komadu.ingest.db.BaseDBIngester  -Unable to process raw notification with internalID 53
edu.indiana.d2i.komadu.ingest.IngestException: Max retries 10 for the lock reached for the object: workflow1_58
    at edu.indiana.d2i.komadu.util.KomaduUtils.manageDBLock(KomaduUtils.java:100)
    at edu.indiana.d2i.komadu.ingest.db.BaseDBIngesterImplementer.addNewActivity(BaseDBIngesterImplementer.java:1189)
    at edu.indiana.d2i.komadu.ingest.db.BaseDBIngesterImplementer.storeActivityActivityRelationship(BaseDBIngesterImplementer.java:174)
    at edu.indiana.d2i.komadu.ingest.db.BaseDBIngester.processNotifications(BaseDBIngester.java:123)
    at edu.indiana.d2i.komadu.ingest.AsyncRawNotificationProcessor.run(AsyncRawNotificationProcessor.java:99)
    at java.lang.Thread.run(Thread.java:745)
22 Sep 2015 15:14:32,733 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: One Message received]
<kom:addAgentAgentRelationship xmlns:kom="http://komadu.d2i.indiana.edu">
  <kom:delegateAgent>
    <kom:userAgent>
      <kom:agentID>agent1_58</kom:agentID>
      <kom:attributes>
        <kom:attribute>
          <kom:property>phone</kom:property>
          <kom:value>8123334455</kom:value>
        </kom:attribute>
        <kom:attribute>
          <kom:property>address</kom:property>
          <kom:value>10th Street, Bloomington, IN, 47408</kom:value>
        </kom:attribute>
      </kom:attributes>
      <kom:fullName>Isuru Suriarachchi</kom:fullName>
      <kom:affiliation>testaff</kom:affiliation>
      <kom:email>isuriara@indiana.edu</kom:email>
    </kom:userAgent>
    <kom:type>PERSON</kom:type>
  </kom:delegateAgent>
  <kom:responsibleAgent>
    <kom:userAgent>
      <kom:agentID>agent2_58</kom:agentID>
      <kom:attributes>
        <kom:attribute>
          <kom:property>phone</kom:property>
          <kom:value>8123337788</kom:value>
        </kom:attribute>
        <kom:attribute>
          <kom:property>address</kom:property>
          <kom:value>15th Street, Bloomington, IN, 47408</kom:value>
        </kom:attribute>
      </kom:attributes>
      <kom:fullName>John Fraser</kom:fullName>
      <kom:affiliation>testaff2</kom:affiliation>
      <kom:email>john@indiana.edu</kom:email>
    </kom:userAgent>
    <kom:type>PERSON</kom:type>
  </kom:responsibleAgent>
  <kom:delegation>
    <kom:delegateAgentID>agent1_58</kom:delegateAgentID>
    <kom:responsibleAgentID>agent2_58</kom:responsibleAgentID>
    <kom:attributes>
      <kom:attribute>
        <kom:property>type</kom:property>
        <kom:value>contract</kom:value>
      </kom:attribute>
    </kom:attributes>
  </kom:delegation>
</kom:addAgentAgentRelationship>
22 Sep 2015 15:14:32,789 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: Listening to Messaging System]
22 Sep 2015 15:14:32,880 [AsynProc-1] ERROR edu.indiana.d2i.komadu.ingest.db.BaseDBIngester  -Unable to process raw notification with internalID 54
edu.indiana.d2i.komadu.ingest.IngestException: Max retries 10 for the lock reached for the object: agent1_58
    at edu.indiana.d2i.komadu.util.KomaduUtils.manageDBLock(KomaduUtils.java:100)
    at edu.indiana.d2i.komadu.ingest.db.BaseDBIngesterImplementer.addNewAgent(BaseDBIngesterImplementer.java:730)
    at edu.indiana.d2i.komadu.ingest.db.BaseDBIngesterImplementer.storeAgentAgentRelationship(BaseDBIngesterImplementer.java:269)
    at edu.indiana.d2i.komadu.ingest.db.BaseDBIngester.processNotifications(BaseDBIngester.java:133)
    at edu.indiana.d2i.komadu.ingest.AsyncRawNotificationProcessor.run(AsyncRawNotificationProcessor.java:99)
    at java.lang.Thread.run(Thread.java:745)
22 Sep 2015 15:15:06,761 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: One Query received]
0167b618-a9ac-4875-a99c-163963d8f807#<quer:getAgentGraphRequest xmlns:quer="http://komadu.d2i.indiana.edu/query">
  <quer:informationDetailLevel>FINE</quer:informationDetailLevel>
  <quer:agentID>agent1_58</quer:agentID>
</quer:getAgentGraphRequest>
22 Sep 2015 15:15:06,761 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Karma server: Query] <quer:getAgentGraphRequest xmlns:quer="http://komadu.d2i.indiana.edu/query">
  <quer:informationDetailLevel>FINE</quer:informationDetailLevel>
  <quer:agentID>agent1_58</quer:agentID>
</quer:getAgentGraphRequest>
22 Sep 2015 15:15:06,761 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Karma server: Query Response Routing Key] 0167b618-a9ac-4875-a99c-163963d8f807
22 Sep 2015 15:15:06,878 [Thread-1] INFO  edu.indiana.d2i.komadu.query.graph.GraphGenerator  -Query has not been cached.
22 Sep 2015 15:15:07,650 [Thread-1] INFO  edu.indiana.d2i.komadu.query.graph.GraphGenerator  -Caching graph.
22 Sep 2015 15:15:07,771 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Send Query Response to Client]
22 Sep 2015 15:15:07,781 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Listening Queries from Messaging System]
22 Sep 2015 15:15:36,239 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: One Query received]
<quer:findActivityRequest xmlns:quer="http://komadu.d2i.indiana.edu/query">  <quer:name>workflow1</quer:name>  <quer:nextActivityID>service1_58</quer:nextActivityID></quer:findActivityRequest>
22 Sep 2015 15:15:36,239 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Karma server: Invalid Query Request]

22 Sep 2015 15:15:36,239 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Listening Queries from Messaging System]
22 Sep 2015 15:15:36,244 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: One Message received]
<kom:addActivityActivityRelationship xmlns:kom="http://komadu.d2i.indiana.edu"> <kom:activity1>   <kom:workflowInformation>     <kom:workflowID>workflow1_58</kom:workflowID>     <kom:workflowNodeID>node1</kom:workflowNodeID>     <kom:timestep>20</kom:timestep>     <kom:attributes>       <kom:attribute>         <kom:property>domain</kom:property>         <kom:value>scientific</kom:value>       </kom:attribute>     </kom:attributes>     <kom:instanceOf>       <kom:instanceOfID>slosh_workflow</kom:instanceOfID>       <kom:version>1.0.0</kom:version>       <kom:creationTime>2014-05-19T13:30:14.706-04:00</kom:creationTime>     </kom:instanceOf>   </kom:workflowInformation>   <kom:location>Azure</kom:location> </kom:activity1> <kom:activity2>   <kom:serviceInformation>     <kom:workflowID>workflow1_58</kom:workflowID>     <kom:workflowNodeID>node1</kom:workflowNodeID>     <kom:timestep>20</kom:timestep>     <kom:attributes>       <kom:attribute>         <kom:property>domain</kom:property>         <kom:value>scientific</kom:value>       </kom:attribute>     </kom:attributes>     <kom:instanceOf>       <kom:instanceOfID>slosh_workflow</kom:instanceOfID>       <kom:version>1.0.0</kom:version>       <kom:creationTime>2014-05-19T13:30:14.729-04:00</kom:creationTime>     </kom:instanceOf>     <kom:serviceID>service1_58</kom:serviceID>   </kom:serviceInformation>   <kom:location>Azure</kom:location> </kom:activity2> <kom:communication>   <kom:informedActivityID>service1_58</kom:informedActivityID>   <kom:informantActivityID>workflow1_58</kom:informantActivityID>   <kom:attributes>     <kom:attribute>       <kom:property>dummy_att</kom:property>       <kom:value>communication_att</kom:value>     </kom:attribute>   </kom:attributes> </kom:communication></kom:addActivityActivityRelationship>
22 Sep 2015 15:15:36,275 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: Listening to Messaging System]
22 Sep 2015 15:15:37,047 [AsynProc-0] ERROR edu.indiana.d2i.komadu.ingest.db.BaseDBIngester  -Unable to process raw notification with internalID 55
edu.indiana.d2i.komadu.ingest.IngestException: Max retries 10 for the lock reached for the object: workflow1_58
    at edu.indiana.d2i.komadu.util.KomaduUtils.manageDBLock(KomaduUtils.java:100)
    at edu.indiana.d2i.komadu.ingest.db.BaseDBIngesterImplementer.addNewActivity(BaseDBIngesterImplementer.java:1189)
    at edu.indiana.d2i.komadu.ingest.db.BaseDBIngesterImplementer.storeActivityActivityRelationship(BaseDBIngesterImplementer.java:174)
    at edu.indiana.d2i.komadu.ingest.db.BaseDBIngester.processNotifications(BaseDBIngester.java:123)
    at edu.indiana.d2i.komadu.ingest.AsyncRawNotificationProcessor.run(AsyncRawNotificationProcessor.java:99)
    at java.lang.Thread.run(Thread.java:745)
22 Sep 2015 15:23:24,331 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Starting Notification Deamon [0]  for receiving notifications from Client.
22 Sep 2015 15:23:24,333 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Notification Deamon [0] Started.
22 Sep 2015 15:23:24,333 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Starting Query Deamon [0] for receiving queries from Client.
22 Sep 2015 15:23:24,333 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: Listening to Messaging System]
22 Sep 2015 15:23:24,333 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Query Deamon [0] Started.
22 Sep 2015 15:23:24,334 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Listening Queries from Messaging System]
22 Sep 2015 15:25:55,433 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: One Query received]
<quer:findActivityRequest xmlns:quer="http://komadu.d2i.indiana.edu/query">  <quer:name>workflow1</quer:name>  <quer:nextActivityID>service1_58</quer:nextActivityID></quer:findActivityRequest>
22 Sep 2015 15:25:55,433 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Karma server: Invalid Query Request]

22 Sep 2015 15:25:55,433 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Listening Queries from Messaging System]
22 Sep 2015 15:25:55,434 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: One Message received]
<kom:addActivityActivityRelationship xmlns:kom="http://komadu.d2i.indiana.edu"> <kom:activity1>   <kom:workflowInformation>     <kom:workflowID>workflow1_58</kom:workflowID>     <kom:workflowNodeID>node1</kom:workflowNodeID>     <kom:timestep>20</kom:timestep>     <kom:attributes>       <kom:attribute>         <kom:property>domain</kom:property>         <kom:value>scientific</kom:value>       </kom:attribute>     </kom:attributes>     <kom:instanceOf>       <kom:instanceOfID>slosh_workflow</kom:instanceOfID>       <kom:version>1.0.0</kom:version>       <kom:creationTime>2014-05-19T13:30:14.706-04:00</kom:creationTime>     </kom:instanceOf>   </kom:workflowInformation>   <kom:location>Azure</kom:location> </kom:activity1> <kom:activity2>   <kom:serviceInformation>     <kom:workflowID>workflow1_58</kom:workflowID>     <kom:workflowNodeID>node1</kom:workflowNodeID>     <kom:timestep>20</kom:timestep>     <kom:attributes>       <kom:attribute>         <kom:property>domain</kom:property>         <kom:value>scientific</kom:value>       </kom:attribute>     </kom:attributes>     <kom:instanceOf>       <kom:instanceOfID>slosh_workflow</kom:instanceOfID>       <kom:version>1.0.0</kom:version>       <kom:creationTime>2014-05-19T13:30:14.729-04:00</kom:creationTime>     </kom:instanceOf>     <kom:serviceID>service1_58</kom:serviceID>   </kom:serviceInformation>   <kom:location>Azure</kom:location> </kom:activity2> <kom:communication>   <kom:informedActivityID>service1_58</kom:informedActivityID>   <kom:informantActivityID>workflow1_58</kom:informantActivityID>   <kom:attributes>     <kom:attribute>       <kom:property>dummy_att</kom:property>       <kom:value>communication_att</kom:value>     </kom:attribute>   </kom:attributes> </kom:communication></kom:addActivityActivityRelationship>
22 Sep 2015 15:25:55,931 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: Listening to Messaging System]
22 Sep 2015 15:25:56,623 [AsynProc-0] ERROR edu.indiana.d2i.komadu.ingest.db.BaseDBIngester  -Unable to process raw notification with internalID 56
edu.indiana.d2i.komadu.ingest.IngestException: Max retries 10 for the lock reached for the object: workflow1_58
    at edu.indiana.d2i.komadu.util.KomaduUtils.manageDBLock(KomaduUtils.java:100)
    at edu.indiana.d2i.komadu.ingest.db.BaseDBIngesterImplementer.addNewActivity(BaseDBIngesterImplementer.java:1189)
    at edu.indiana.d2i.komadu.ingest.db.BaseDBIngesterImplementer.storeActivityActivityRelationship(BaseDBIngesterImplementer.java:174)
    at edu.indiana.d2i.komadu.ingest.db.BaseDBIngester.processNotifications(BaseDBIngester.java:123)
    at edu.indiana.d2i.komadu.ingest.AsyncRawNotificationProcessor.run(AsyncRawNotificationProcessor.java:99)
    at java.lang.Thread.run(Thread.java:745)
22 Sep 2015 15:27:55,484 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: One Query received]
<quer:findActivityRequest xmlns:quer="http://komadu.d2i.indiana.edu/query">  <quer:name>workflow1</quer:name>  <quer:nextActivityID>service1_58</quer:nextActivityID></quer:findActivityRequest>
22 Sep 2015 15:27:55,484 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Karma server: Invalid Query Request]

22 Sep 2015 15:27:55,484 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Listening Queries from Messaging System]
22 Sep 2015 15:27:55,490 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: One Message received]
<kom:addActivityActivityRelationship xmlns:kom="http://komadu.d2i.indiana.edu"> <kom:activity1>   <kom:workflowInformation>     <kom:workflowID>workflow1_58</kom:workflowID>     <kom:workflowNodeID>node1</kom:workflowNodeID>     <kom:timestep>20</kom:timestep>     <kom:attributes>       <kom:attribute>         <kom:property>domain</kom:property>         <kom:value>scientific</kom:value>       </kom:attribute>     </kom:attributes>     <kom:instanceOf>       <kom:instanceOfID>slosh_workflow</kom:instanceOfID>       <kom:version>1.0.0</kom:version>       <kom:creationTime>2014-05-19T13:30:14.706-04:00</kom:creationTime>     </kom:instanceOf>   </kom:workflowInformation>   <kom:location>Azure</kom:location> </kom:activity1> <kom:activity2>   <kom:serviceInformation>     <kom:workflowID>workflow1_58</kom:workflowID>     <kom:workflowNodeID>node1</kom:workflowNodeID>     <kom:timestep>20</kom:timestep>     <kom:attributes>       <kom:attribute>         <kom:property>domain</kom:property>         <kom:value>scientific</kom:value>       </kom:attribute>     </kom:attributes>     <kom:instanceOf>       <kom:instanceOfID>slosh_workflow</kom:instanceOfID>       <kom:version>1.0.0</kom:version>       <kom:creationTime>2014-05-19T13:30:14.729-04:00</kom:creationTime>     </kom:instanceOf>     <kom:serviceID>service1_58</kom:serviceID>   </kom:serviceInformation>   <kom:location>Azure</kom:location> </kom:activity2> <kom:communication>   <kom:informedActivityID>service1_58</kom:informedActivityID>   <kom:informantActivityID>workflow1_58</kom:informantActivityID>   <kom:attributes>     <kom:attribute>       <kom:property>dummy_att</kom:property>       <kom:value>communication_att</kom:value>     </kom:attribute>   </kom:attributes> </kom:communication></kom:addActivityActivityRelationship>
22 Sep 2015 15:27:55,519 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: Listening to Messaging System]
22 Sep 2015 15:27:55,878 [AsynProc-1] ERROR edu.indiana.d2i.komadu.ingest.db.BaseDBIngester  -Unable to process raw notification with internalID 57
edu.indiana.d2i.komadu.ingest.IngestException: Max retries 10 for the lock reached for the object: workflow1_58
    at edu.indiana.d2i.komadu.util.KomaduUtils.manageDBLock(KomaduUtils.java:100)
    at edu.indiana.d2i.komadu.ingest.db.BaseDBIngesterImplementer.addNewActivity(BaseDBIngesterImplementer.java:1189)
    at edu.indiana.d2i.komadu.ingest.db.BaseDBIngesterImplementer.storeActivityActivityRelationship(BaseDBIngesterImplementer.java:174)
    at edu.indiana.d2i.komadu.ingest.db.BaseDBIngester.processNotifications(BaseDBIngester.java:123)
    at edu.indiana.d2i.komadu.ingest.AsyncRawNotificationProcessor.run(AsyncRawNotificationProcessor.java:99)
    at java.lang.Thread.run(Thread.java:745)
gabrielzhouquan commented 9 years ago

Hi Sid,

Got the problem. See the following log output:

quer:nameworkflow1/quer:name quer:nextActivityIDservice1_58/quer:nextActivityID/quer:findActivityRequest 22 Sep 2015 15:15:36,239 [Thread-1] INFO edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable -[Karma server: Invalid Query Request] The query format is invalid. Line delimiter is needed to send the query to Komadu server. (“\n”) Best, Quan Zhou > On Sep 22, 2015, at 3:29 PM, sid thakur notifications@github.com wrote: > > manageDBLock(KomaduUtils.java:100)
idea-launch-lab commented 9 years ago

Okay, thanks! I will try and fix that. I think converting XML to serialized string might be the issue.

gabrielzhouquan commented 9 years ago

Hi Sid,

No problem at all.

Glad I can help you with this! :)

Best,

Quan Zhou

On Sep 22, 2015, at 3:43 PM, sid thakur notifications@github.com wrote:

Okay, thanks! I will try and fix that. I think converting XML to serialized string might be the issue.

— Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-142397122.

idea-launch-lab commented 8 years ago

@gabrielzhouquan Hi, I am still having trouble getting Komadu to process my query and return a result. Below is read-out from komadu.log. I am sending querying with line delimiters, but still getting the 'invalid query' error. Could you please take a look when you get a chance? Thanks.

[cluster@server komadu-1.0]$ more komadu.log 
28 Sep 2015 15:16:17,606 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Starting Notification Deamon [0]  for receiving notifications from C
lient.
28 Sep 2015 15:16:17,608 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Notification Deamon [0] Started.
28 Sep 2015 15:16:17,608 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Starting Query Deamon [0] for receiving queries from Client.
28 Sep 2015 15:16:17,609 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: Listening to Messaging System]
28 Sep 2015 15:16:17,609 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Query Deamon [0] Started.
28 Sep 2015 15:16:17,610 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Listening Queries from Messaging System]
28 Sep 2015 15:16:26,561 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: One Query received]
<quer:findEntityRequest xmlns:quer="http://komadu.d2i.indiana.edu/query">
  <quer:fileName>data1.txt_58</quer:fileName>
  <quer:fileOwnerID>jerry112</quer:fileOwnerID>
  <quer:fileSize>500</quer:fileSize>
</quer:findEntityRequest>

28 Sep 2015 15:16:26,562 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Karma server: Invalid Query Request]

28 Sep 2015 15:16:26,562 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Listening Queries from Messaging System]
gabrielzhouquan commented 8 years ago

Hi Sid,

It is weird.

Here is my output:

28 Sep 2015 15:26:27,425 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Starting Notification Deamon [0] for receiving notifications from Client. 28 Sep 2015 15:26:27,427 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Notification Deamon [0] Started. 28 Sep 2015 15:26:27,427 [Thread-0] INFO edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable -[Komadu server: Listening to Messaging System] 28 Sep 2015 15:26:27,427 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Starting Query Deamon [0] for receiving queries from Client. 28 Sep 2015 15:26:27,428 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Query Deamon [0] Started. 28 Sep 2015 15:26:27,428 [Thread-1] INFO edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable -[Komadu server: Listening Queries from Messaging System] 28 Sep 2015 15:27:38,917 [Thread-1] INFO edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable -[Komadu server: One Query received] 9e84712d-3cb7-4ba6-b689-6b97cebfc570# quer:fileNamedata1.txt_58/quer:fileName quer:fileOwnerIDjerry112/quer:fileOwnerID quer:fileSize500/quer:fileSize /quer:findEntityRequest 28 Sep 2015 15:27:38,918 [Thread-1] INFO edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable -[Karma server: Query] quer:fileNamedata1.txt_58/quer:fileName quer:fileOwnerIDjerry112/quer:fileOwnerID quer:fileSize500/quer:fileSize /quer:findEntityRequest 28 Sep 2015 15:27:38,918 [Thread-1] INFO edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable -[Karma server: Query Response Routing Key] 9e84712d-3cb7-4ba6-b689-6b97cebfc570 28 Sep 2015 15:27:39,295 [Thread-1] INFO edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable -[Komadu server: Send Query Response to Client] 28 Sep 2015 15:27:39,300 [Thread-1] INFO edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable -[Komadu server: Listening Queries from Messaging System]

Could I have a look at your updated js codes with delimiter added?

Many thanks!

best,

Quan Zhou

On Sep 28, 2015, at 3:21 PM, sid thakur notifications@github.com wrote:

@gabrielzhouquan https://github.com/gabrielzhouquan Hi, I am still having trouble getting Komadu to process my query and return a result. Below is read-out from komadu.log. I am sending querying with line delimiters, but still getting the 'invalid query' error. Could you please take a look when you get a chance? Thanks.

[cluster@server komadu-1.0]$ more komadu.log 28 Sep 2015 15:16:17,606 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Starting Notification Deamon [0] for receiving notifications from C lient. 28 Sep 2015 15:16:17,608 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Notification Deamon [0] Started. 28 Sep 2015 15:16:17,608 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Starting Query Deamon [0] for receiving queries from Client. 28 Sep 2015 15:16:17,609 [Thread-0] INFO edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable -[Komadu server: Listening to Messaging System] 28 Sep 2015 15:16:17,609 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Query Deamon [0] Started. 28 Sep 2015 15:16:17,610 [Thread-1] INFO edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable -[Komadu server: Listening Queries from Messaging System] 28 Sep 2015 15:16:26,561 [Thread-1] INFO edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable -[Komadu server: One Query received]

quer:fileNamedata1.txt_58/quer:fileName quer:fileOwnerIDjerry112/quer:fileOwnerID quer:fileSize500/quer:fileSize /quer:findEntityRequest 28 Sep 2015 15:16:26,562 [Thread-1] INFO edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable -[Karma server: Invalid Query Request] 28 Sep 2015 15:16:26,562 [Thread-1] INFO edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable -[Komadu server: Listening Queries from Messaging System] — Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-143849109.
idea-launch-lab commented 8 years ago

Here you go; I add line delimiter after reading in text file with query. Thanks.


exports.prov_emit ('', 'KomaduQueueQueryRequest');

// Emit
exports.prov_emit = function (data, queueName) {

    // form provenance document
    var provDoc = '';

    // set path and name of data file
    var filePath = './data/';
    if (queueName === 'KomaduQueue_Notification') {
    filePath += 'activity_activity.txt';//
    }
    else if (queueName === 'KomaduQueueQueryRequest') {
    filePath += 'query_find_entity.txt';
    }

    var array = fs.readFileSync (filePath).toString ().split ("\n");
    for(var i in array) {
    provDoc += array [i];

//>>>     ADDING LINE DELIMITER

    if (i < (array.length - 1)) {
        provDoc += '\n';
    }

//<<<

    }
    console.log ("[PROV] Document sent:\n" + provDoc);

    // for querying Komadu use queue name 'KomaduQueueQueryRequest'
    // for inserting provenance notification use queue name 'KomaduQueue_Notification'

    // error handler function
    function bail(err, conn) {
    logger.debug (err);
    if (conn) {
        conn.close(function() {
        // DO NOTHING
        });
    }
    }

    function on_connect(err, conn) {
    console.log ('connecting to queue: ' + queueName);
    if (err !== null) 
        return bail(err);

    var q = queueName;
    var msg = provDoc; //'Hello World!';

    function on_channel_open(err, ch) {

        if (err !== null) {
        return bail(err, conn);
        }

        ch.assertQueue(q, {}, function(err, ok) {

        if (err !== null) return bail(err, conn);

        ch.sendToQueue(q, new Buffer(msg));
        console.log(" [x] Sent '%s'", msg);
        logger.debug ('[PROV_SEND] Provenance notification sent to Komadu server');

        // close channel and connection
        ch.close(function() {
            console.info ("Closing connection...");
            conn.close(); 
        });
        });
    }

    conn.createChannel (on_channel_open);
    }

    // create an amqp connection through tunnel
    amqp.connect('amqp://kuser:resuk@127.0.0.1:5672', on_connect);
};