Running Komadu on Mesos cluster: Web service or Standalone server?

[idea-launch-lab]

Hi @isurues We have an Apache Mesos cluster within which we'd like to deploy Komadu. Ideally, we'd like komadu to run in its own VM on the cluster and be able to receive event-style notifications from a web app and other process running on another, production-level, VM. Essentially, we'd like to be able to send prov notification events from other VMs to komdu (running in its own VM).

What I'd like to know is which of the two methods of running Komadu, web service or standalone server, would be suitable for our scenario. Please also clarify what accommodations, if any, would we need to make to enable Komadu to function within this multi-VM envrionment.

Thanks,

Sid

[gabrielzhouquan]

Hi Sid,

Got the problem! Sorry that I didn’t see it at the first place.

For Komadu query, the client should send query format as following:

+”#”+ For your code, please add the response routing key (UUID) to the beginning of provDoc: var array = fs.readFileSync (filePath).toString ().split ("\n”); var uuid = guid(); provDoc += uuid+”#”; << add routing key and # sign ``` for(var i in array) { provDoc += array [i]; ``` //>>> ADDING LINE DELIMITER ``` if (i < (array.length - 1)) { provDoc += '\n'; } ``` function guid() { function s4() { return Math.floor((1 + Math.random()) \* 0x10000) .toString(16) .substring(1); } return s4() + s4() + '-' + s4() + '-' + s4() + '-' + s4() + '-' + s4() + s4() + s4(); } Please let me if this works for you. Many thanks! Best, Gabriel Zhou > On Sep 28, 2015, at 3:50 PM, sid thakur notifications@github.com wrote: > > Here you go; I add line delimiter after reading in text file with query. > Thanks. > > exports.prov_emit ('', 'KomaduQueueQueryRequest'); > > // Emit > exports.prov_emit = function (data, queueName) { > > ``` > // form provenance document > var provDoc = ''; > > // set path and name of data file > var filePath = './data/'; > if (queueName === 'KomaduQueue_Notification') { > filePath += 'activity_activity.txt';// > } > else if (queueName === 'KomaduQueueQueryRequest') { > filePath += 'query_find_entity.txt'; > } > > var array = fs.readFileSync (filePath).toString ().split ("\n"); > for(var i in array) { > provDoc += array [i]; > ``` > > //>>> ADDING LINE DELIMITER > > ``` > if (i < (array.length - 1)) { > provDoc += '\n'; > } > ``` > > //<<< > > ``` > } > console.log ("[PROV] Document sent:\n" + provDoc); > > // for querying Komadu use queue name 'KomaduQueueQueryRequest' > // for inserting provenance notification use queue name 'KomaduQueue_Notification' > > // error handler function > function bail(err, conn) { > logger.debug (err); > if (conn) { > conn.close(function() { > // DO NOTHING > }); > } > } > > function on_connect(err, conn) { > console.log ('connecting to queue: ' + queueName); > if (err !== null) > return bail(err); > > var q = queueName; > var msg = provDoc; //'Hello World!'; > > function on_channel_open(err, ch) { > > if (err !== null) { > return bail(err, conn); > } > > ch.assertQueue(q, {}, function(err, ok) { > > if (err !== null) return bail(err, conn); > > ch.sendToQueue(q, new Buffer(msg)); > console.log(" [x] Sent '%s'", msg); > logger.debug ('[PROV_SEND] Provenance notification sent to Komadu server'); > > // close channel and connection > ch.close(function() { > console.info ("Closing connection..."); > conn.close(); > }); > }); > } > > conn.createChannel (on_channel_open); > } > > // create an amqp connection through tunnel > amqp.connect('amqp://kuser:resuk@127.0.0.1:5672', on_connect); > ``` > > }; > — > Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-143856185.

[idea-launch-lab]

Thanks, I'll try it out and let you know. @gabrielzhouquan

[idea-launch-lab]

@gabrielzhouquan Okay, this time there is no error in komadu log. But client does not receive response. I think something to do with my receiver. Thanks.

Komadu log output:

30 Sep 2015 09:46:33,899 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Starting Notification Deamon [0]  for receiving notifications from C
lient.
30 Sep 2015 09:46:33,901 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Notification Deamon [0] Started.
30 Sep 2015 09:46:33,901 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Starting Query Deamon [0] for receiving queries from Client.
30 Sep 2015 09:46:33,901 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: Listening to Messaging System]
30 Sep 2015 09:46:33,901 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Query Deamon [0] Started.
30 Sep 2015 09:46:33,902 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Listening Queries from Messaging System]
30 Sep 2015 09:46:40,613 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: One Query received]
3d429c86-89da-dd3e-2ced-fe7d24e335f8#<quer:findEntityRequest xmlns:quer="http://komadu.d2i.indiana.edu/query">
  <quer:fileName>data1.txt_58</quer:fileName>
  <quer:fileOwnerID>jerry112</quer:fileOwnerID>
  <quer:fileSize>500</quer:fileSize>
</quer:findEntityRequest>

30 Sep 2015 09:46:40,613 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Karma server: Query] <quer:findEntityRequest xmlns:quer=
"http://komadu.d2i.indiana.edu/query">
  <quer:fileName>data1.txt_58</quer:fileName>
  <quer:fileOwnerID>jerry112</quer:fileOwnerID>
  <quer:fileSize>500</quer:fileSize>
</quer:findEntityRequest>
30 Sep 2015 09:46:40,613 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Karma server: Query Response Routing Key] 3d429c86-89da-
dd3e-2ced-fe7d24e335f8
30 Sep 2015 09:46:41,158 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Send Query Response to Client]
30 Sep 2015 09:46:41,171 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Listening Queries from Messaging System]

Receiver code. The function ch.consume (q, function(msg) {..} with msg doesn't get called.

exports.prov_receive = function () {

    console.log ("[PROV_RECEIVE] Inside receiver code");

    var queueName = 'KomaduQueueQueryResponse';

    // error handler function
    function bail(err, conn) {
    logger.debug (err);
    if (conn) {
        conn.close(function() {
        // DO NOTHING
        });
    }
    }

    function on_connect(err, conn) {
    // check if connection failed
    if (err !== null) {
        return bail(err);
    }

    process.once('SIGINT', function() {
        conn.close();
    });

    var q = queueName;

    function on_channel_open(err, ch) {

        ch.assertQueue(q, {}, function(err, ok) {
        console.log ('[PROV_RECEIVE] channel ok? ' + JSON.stringify(ok));
        if (err !== null) {
            return bail(err, conn);
        }

        // message callback
        ch.consume (q, function(msg) {

            console.log (" [x] Received '%s'", msg.content.toString());
            logger.debug ('[PROV_RECEIVE] Provenance notification received from Komadu server');
            ch.close ();

        }, {noAck: false}, function(_consumeOk) { // consume callback
            console.log ('_consumeOk: ' + JSON.stringify (_consumeOk));
            console.log (' [*] Waiting for messages. To exit press CTRL+C');
        });
        });
    }
    console.log ('[PROV_RECEIVE] queue: ' + queueName);

    conn.createChannel(on_channel_open);
    }

    amqp.connect('amqp://kuser:resuk@127.0.0.1:5672', on_connect);
};

Calling emit and receive in following order:

    exports.prov_emit ('', 'KomaduQueueQueryRequest');
    exports.prov_receive ();

Console output on my server:

[PROV_SEND] Inside notification send code
[PROV_SEND] UUID: 3d429c86-89da-dd3e-2ced-fe7d24e335f8

[PROV_RECEIVE] Inside receiver code

[PROV_SEND] connecting to queue: KomaduQueueQueryRequest

[PROV_RECEIVE] queue: KomaduQueueQueryResponse
[PROV_SEND] Provenance notification sent to Komadu server
[PROV_RECEIVE] channel ok? {"queue":"KomaduQueueQueryResponse","messageCount":0,"consumerCount":0}
_consumeOk: null
 [*] Waiting for messages. To exit press CTRL+C
[PROV_SEND] Closing connection...

[idea-launch-lab]

@isurues When you get a chance, could you please take a look at previous post Re. issue getting response back from server. I am guessing that @gabrielzhouquan is busy. Thank you all.

[gabrielzhouquan]

Hi Sid,

For the receiver code, Could you refer to the following code I write for connecting to a server and listening on a queue? It may give you a clear view about the usage of AMQP js lib.

var amqp = require('amqp');

var connection = amqp.createConnection({ host: 'dev.rabbitmq.com' });

// Wait for connection to become established. connection.on('ready', function () { // Use the default 'amq.topic' exchange connection.queue('my-queue', function (q) { // Catch all messages q.bind('#');

  // Receive messages
  q.subscribe(function (message) {
    // Print messages to stdout
    console.log(message);
  });

}); });

Best,

Quan Zhou

On Oct 7, 2015, at 9:40 AM, sid thakur notifications@github.com wrote:

@isurues https://github.com/isurues When you get a chance, could you please take a look at previous post Re. issue getting response back from server. I am guessing that @gabrielzhouquan https://github.com/gabrielzhouquan is busy. Thank you all.

— Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-146199099.

[idea-launch-lab]

Thanks. I tried the amqp lib you suggested, but there is no response from server. Among other thigns, I couldn't create a queue using KomaduQueueQueryResponse I don't understand why I need a generic my-queue to be able to set up a queue. Thanks.

exports.prov_receive_2 = function () {

    var connection = node_amqp.createConnection ({host:'localhost', port:'5672', vhost: '/', login:'kuser', password:'resuk'});

    // Wait for connection to become established.
    connection.on('ready', function () {
        console.log ('[PROV_REC] connection ready.. ');

    // Use the default 'amq.topic' exchange
    connection.queue('my-queue', function (q) {

        console.log ('[PROV_REC] queue is open');
        console.log ('[PROV_REC] queue name: ' + q.name);
        console.log ('[PROV_REC] waiting for response');
        // Catch all messages 
        q.bind('#');

        // Receive messages 
        q.subscribe(function (message) {
        // Print messages to stdout
        console.log('[PROV_REC] RESULT: ' + message);
        });
    });
    });
};

I call the the receiver after I close the sender, to make sure that there is no overlap. Below is readout from my stdout:

2015-10-13T20:01:22.520Z - debug: [doc-db] [PROV_SEND] Provenance notification sent to Komadu server
[PROV_SEND] Closing connection...
[PROV_REC] connection ready.. 
[PROV_REC] queue is open
[PROV_REC] queue name: my-queue
[PROV_REC] waiting for response

Komadu log file reports that response is sent to client:

13 Oct 2015 16:01:22,527 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Karma server: Qu
ery] <quer:findEntityRequest xmlns:quer="http://komadu.d2i.indiana.edu/query">
  <quer:fileName>data1.txt_58</quer:fileName>
  <quer:fileOwnerID>jerry112</quer:fileOwnerID>
  <quer:fileSize>500</quer:fileSize>
</quer:findEntityRequest>
13 Oct 2015 16:01:22,527 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Karma server: Qu
ery Response Routing Key] c76a176b-c425-9b8c-fabc-509a86438adc
13 Oct 2015 16:01:22,536 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: S
end Query Response to Client]
13 Oct 2015 16:01:22,541 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: L
istening Queries from Messaging System]

[idea-launch-lab]

@gabrielzhouquan Any hints at how I could debug the issue with query response? Please see prev. post. Appreciate your time for helping us install and use K, and apologize for this recurring issue. Thanks!

[gabrielzhouquan]

Hi Sid,

No problem at all.

I have wrote a functional javascript Komadu query client for you as below:

query.js:

!/usr/bin/env node

var amqp = require('amqplib/callback_api');

amqp.connect('amqp://localhost', function(err, conn) { conn.createChannel(function(err, ch) { var queueName = 'KomaduQueueQueryRequest'; var fs = require("fs"); // form provenance document var provDoc = '';

var uuid = guid();
provDoc += uuid+'#';

// set path and name of data file
var filePath = './';
if (queueName === 'KomaduQueue_Notification') {
filePath += 'activity_activity.xml';//
}
else if (queueName === 'KomaduQueueQueryRequest') {
filePath += 'query_entity_graph.xml';
}

provDoc += fs.readFileSync (filePath).toString();

console.log ("[PROV] Document sent:\n" + provDoc);
var ex = 'KomaduExchangeQueryRequest';
var routingKey = 'KomaduKeyQueryRequest';

ch.assertExchange(ex, 'direct', {durable: true});
ch.publish(ex, routingKey, new Buffer(provDoc));
console.log(" [x] Sent %s: '%s'", routingKey, provDoc);

var ex2 = 'KomaduExchangeQueryResponse';

ch.assertExchange(ex2, 'direct', {durable: false});

ch.assertQueue('KomaduQueueQueryResponse', {exclusive: true}, function(err, q) {
  console.log(' [*] Waiting for logs...');
    ch.bindQueue(q.queue, ex2, uuid);
  ch.consume(q.queue, function(msg) {
    console.log(" [x] %s: '%s'", msg.fields.routingKey, msg.content.toString());
  }, {noAck: true});
});

});

setTimeout(function() { conn.close(); process.exit(0) }, 5000); });

function guid() { function s4() { return Math.floor((1 + Math.random()) * 0x10000) .toString(16) .substring(1); } return s4() + s4() + '-' + s4() + '-' + s4() + '-' + s4() + '-' + s4() + s4() + s4(); } For usage:

1/ Install amqplib:

npm install amqplib

2/ Run query.js:

node query.js

Example result is as below:

[PROV] Document sent: 8beb9f5b-8198-6b27-d869-d0986cd954a1# quer:informationDetailLevelFINE/quer:informationDetailLevel quer:entityURIfile://foo/bar/data2.txt_58/quer:entityURI quer:entityTypeFILE/quer:entityType /quer:getEntityGraphRequest [x] Sent KomaduKeyQueryRequest: '8beb9f5b-8198-6b27-d869-d0986cd954a1# quer:informationDetailLevelFINE/quer:informationDetailLevel quer:entityURIfile://foo/bar/data2.txt_58/quer:entityURI quer:entityTypeFILE/quer:entityType /quer:getEntityGraphRequest' [*] Waiting for logs... [x] 8beb9f5b-8198-6b27-d869-d0986cd954a1: '

File/kom:type file://foo/bar/data2.txt_58/kom:url 500/kom:size 2014-05-19 13:30:15.0/kom:creation-date data2.txt_58/kom:file-name dummy_md5_data2.txt_58/kom:md5_checksum /prov:entity File/kom:type file://foo/bar/data6.txt_58/kom:url 500/kom:size 2014-05-19 13:30:16.0/kom:creation-date data6.txt_58/kom:file-name dummy_md5_data6.txt_58/kom:md5_checksum /prov:entity dummy_val/ext:dummy_derivation /prov:wasDerivedFrom /prov:document /prov:document /quer:getEntityGraphResponse' Please let me know if you have further questions or problems. Many thanks! best, Gabriel Zhou > On Oct 19, 2015, at 10:01 AM, sid thakur notifications@github.com wrote: > > @gabrielzhouquan https://github.com/gabrielzhouquan Any hints at how I could debug the issue with query response? Please see prev. post. Appreciate your time for helping us install and use K, and apologize for this recurring issue. Thanks! > > — > Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-149221469.

[idea-launch-lab]

@gabrielzhouquan Thank you! I will test it with my application.

[idea-launch-lab]

Hi @gabrielzhouquan , I am seeing following issue in Komadu log file when I try to send a notification event. I used the query.js code you posted above and modified it to send a notification. Any clues would be helpful. It seems to be due to XML parsing, but I did not see this earlier.

Does it have to do with Komadu XSD schema?

Thanks.

[cluster]$ more komadu.log 
05 Nov 2015 10:42:11,603 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Starting Notification Deamon [0]  for receiving notification
s from Client.
05 Nov 2015 10:42:11,606 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Notification Deamon [0] Started.
05 Nov 2015 10:42:11,606 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Starting Query Deamon [0] for receiving queries from Client.
05 Nov 2015 10:42:11,606 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Query Deamon [0] Started.
05 Nov 2015 10:42:11,608 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: Listening to Messaging Sys
tem]
05 Nov 2015 10:42:11,610 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Listening Queries from Messaging 
System]
05 Nov 2015 10:42:18,418 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: One Message received]
02c497af-0314-e0df-0e5a-a4cd50adba80#<kom:addActivityEntityRelationship xmlns:kom="http://komadu.d2i.indiana.edu">
 <kom:activity>
   <kom:serviceInformation>
     <kom:workflowID>workflow1_58</kom:workflowID>
     <kom:workflowNodeID>node1</kom:workflowNodeID>
     <kom:timestep>20</kom:timestep>
     <kom:attributes>
       <kom:attribute>
         <kom:property>domain</kom:property>
         <kom:value>scientific</kom:value>
       </kom:attribute>
     </kom:attributes>
     <kom:instanceOf>
       <kom:instanceOfID>slosh_workflow</kom:instanceOfID>
       <kom:version>1.0.0</kom:version>
       <kom:creationTime>2014-05-19T13:30:14.729-04:00</kom:creationTime>
     </kom:instanceOf>
     <kom:serviceID>service1_58</kom:serviceID>
   </kom:serviceInformation>
   <kom:location>Azure</kom:location>
 </kom:activity>
 <kom:entity>
   <kom:file>
     <kom:fileURI>file://foo/bar/data1.txt_58</kom:fileURI>
     <kom:ownerDN>jerry112</kom:ownerDN>
     <kom:createDate>2014-05-19T13:30:14.954-04:00</kom:createDate>
     <kom:size>500</kom:size>
     <kom:md5sum>dummy_md5_data1.txt_58</kom:md5sum>
     <kom:fileName>data1.txt_58</kom:fileName>
   </kom:file>
 </kom:entity>
 <kom:usage>
   <kom:activityID>service1_58</kom:activityID>
   <kom:entityID>file://foo/bar/data1.txt_58</kom:entityID>
   <kom:timestamp>2014-05-19T13:30:14.963-04:00</kom:timestamp>
   <kom:attributes>
     <kom:attribute>
       <kom:property>total_time</kom:property>
       <kom:value>0.53h</kom:value>
     </kom:attribute>
   </kom:attributes>
   <kom:location>Location2</kom:location>
 </kom:usage>
</kom:addActivityEntityRelationship>

05 Nov 2015 10:42:18,768 [Thread-0] ERROR edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -
org.apache.xmlbeans.XmlException: error: Unexpected element: CDATA
    at org.apache.xmlbeans.impl.store.Locale$SaxLoader.load(Locale.java:3471)
    at org.apache.xmlbeans.impl.store.Locale.parse(Locale.java:706)
    at org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:690)
    at org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:677)
    at org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(SchemaTypeLoaderBase.java:208)
    at org.apache.xmlbeans.XmlObject$Factory.parse(XmlObject.java:580)
    at edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable.run(NotificationReceiverRunnable.java:45)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.xml.sax.SAXParseException; systemId: file:; lineNumber: 1; columnNumber: 1; Unexpected element: CDATA
    at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.reportFatalError(Piccolo.java:1038)
    at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.parse(Piccolo.java:723)
    at org.apache.xmlbeans.impl.store.Locale$SaxLoader.load(Locale.java:3439)
    ... 7 more
05 Nov 2015 10:42:18,770 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: Listening to Messaging Sys
tem]

[gabrielzhouquan]

Hi Sid,

I have just published Komadu javascript messaging client today. This node module can be found at:

1. Github repo: https://github.com/Data-to-Insight-Center/komadu/tree/master/client-core-messaging-js https://github.com/Data-to-Insight-Center/komadu/tree/master/client-core-messaging-js
2. npm global repo: https://www.npmjs.com/package/client-core-messaging-js https://www.npmjs.com/package/client-core-messaging-js

The README.md will give you a detailed usage guide with this module.

Please let me know if you encounter any problem when you try out this module.

Many thanks!

Best,

Quan (Gabriel) Zhou

On Nov 5, 2015, at 10:48 AM, sid thakur notifications@github.com wrote:

Hi @gabrielzhouquan https://github.com/gabrielzhouquan , I am seeing following issue in Komadu log file when I try to send a notification event. I used the query.js code you posted above and modified it to send a notification. Any clues would be helpful. It seems to be due to XML parsing, but I did not see this earlier.

Does it have to do with Komadu XSD schema?

Thanks.

[cluster]$ more komadu.log 05 Nov 2015 10:42:11,603 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Starting Notification Deamon [0] for receiving notification s from Client. 05 Nov 2015 10:42:11,606 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Notification Deamon [0] Started. 05 Nov 2015 10:42:11,606 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Starting Query Deamon [0] for receiving queries from Client. 05 Nov 2015 10:42:11,606 [main] INFO edu.indiana.d2i.komadu.messaging.MessageDaemons -Query Deamon [0] Started. 05 Nov 2015 10:42:11,608 [Thread-0] INFO edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable -[Komadu server: Listening to Messaging Sys tem] 05 Nov 2015 10:42:11,610 [Thread-1] INFO edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable -[Komadu server: Listening Queries from Messaging System] 05 Nov 2015 10:42:18,418 [Thread-0] INFO edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable -[Komadu server: One Message received] 02c497af-0314-e0df-0e5a-a4cd50adba80# kom:activity kom:serviceInformation kom:workflowIDworkflow1_58/kom:workflowID kom:workflowNodeIDnode1/kom:workflowNodeID kom:timestep20/kom:timestep kom:attributes kom:attribute kom:propertydomain/kom:property kom:valuescientific/kom:value /kom:attribute /kom:attributes kom:instanceOf kom:instanceOfIDslosh_workflow/kom:instanceOfID kom:version1.0.0/kom:version kom:creationTime2014-05-19T13:30:14.729-04:00/kom:creationTime /kom:instanceOf kom:serviceIDservice1_58/kom:serviceID /kom:serviceInformation kom:locationAzure/kom:location /kom:activity kom:entity kom:file kom:fileURIfile://foo/bar/data1.txt_58/kom:fileURI kom:ownerDNjerry112/kom:ownerDN kom:createDate2014-05-19T13:30:14.954-04:00/kom:createDate kom:size500/kom:size kom:md5sumdummy_md5_data1.txt_58/kom:md5sum kom:fileNamedata1.txt_58/kom:fileName /kom:file /kom:entity kom:usage kom:activityIDservice1_58/kom:activityID kom:entityIDfile://foo/bar/data1.txt_58/kom:entityID kom:timestamp2014-05-19T13:30:14.963-04:00/kom:timestamp kom:attributes kom:attribute kom:propertytotal_time/kom:property kom:value0.53h/kom:value /kom:attribute /kom:attributes kom:locationLocation2/kom:location /kom:usage /kom:addActivityEntityRelationship

05 Nov 2015 10:42:18,768 [Thread-0] ERROR edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable - org.apache.xmlbeans.XmlException: error: Unexpected element: CDATA at org.apache.xmlbeans.impl.store.Locale$SaxLoader.load(Locale.java:3471) at org.apache.xmlbeans.impl.store.Locale.parse(Locale.java:706) at org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:690) at org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:677) at org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(SchemaTypeLoaderBase.java:208) at org.apache.xmlbeans.XmlObject$Factory.parse(XmlObject.java:580) at edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable.run(NotificationReceiverRunnable.java:45) at java.lang.Thread.run(Thread.java:745) Caused by: org.xml.sax.SAXParseException; systemId: file:; lineNumber: 1; columnNumber: 1; Unexpected element: CDATA at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.reportFatalError(Piccolo.java:1038) at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.parse(Piccolo.java:723) at org.apache.xmlbeans.impl.store.Locale$SaxLoader.load(Locale.java:3439) ... 7 more 05 Nov 2015 10:42:18,770 [Thread-0] INFO edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable -[Komadu server: Listening to Messaging Sys tem] — Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-154099074.

[idea-launch-lab]

Hi, That is great, thank you. I will take a look and let you know. Best, Sid

[idea-launch-lab]

@gabrielzhouquan Thanks, I checked out the node module and have integrated it with my code. Send notification works fine. Only a couple issues:

[1] In query, following line generates exception below. Changing to durable:true does not result in error and the query is executed and response returned without error: ch.assertExchange(response_ex, 'direct', {durable: false});

[PROV] Query sent sucessfully!!
2015-11-06T21:27:36.674Z - error: [local] -->[ Unhandled exception: Error: Channel closed by server: 406 (PRECONDITION-FAILED) with message "PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'KomaduExchangeQueryResponse' in vhost '/': received 'false' but current is 'true'" ]
Error: Channel closed by server: 406 (PRECONDITION-FAILED) with message "PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'KomaduExchangeQueryResponse' in vhost '/': received 'false' but current is 'true'"
    at Channel.C.accept (/home/sthakur/.../node_modules/client-core-messaging-js/node_modules/amqplib/lib/channel.js:406:17)
    at Connection.mainAccept [as accept] (/home/sthakur/.../node_modules/client-core-messaging-js/node_modules/amqplib/lib/connection.js:63:33)
    at Socket.go (/home/sthakur/.../node_modules/client-core-messaging-js/node_modules/amqplib/lib/connection.js:474:48)
    at Socket.emit (events.js:104:17)
    at emitReadable_ (_stream_readable.js:424:10)
    at emitReadable (_stream_readable.js:418:7)
    at readableAddChunk (_stream_readable.js:174:11)
    at Socket.Readable.push (_stream_readable.js:126:10)
    at TCP.onread (net.js:538:20)

[2] Even though records seem to added via send notification, query almost always returns a sparse result (blank). I have been using the standard notification and query data in the komadu/samples/. folder.

[PROV] Query sent sucessfully!!
[INFO] Waiting for response...
[PROV] Query response received:
<quer:getActivityDetailResponse xmlns:quer="http://komadu.d2i.indiana.edu/query">
  <quer:activityDetailList/>
</quer:getActivityDetailResponse>
[INFO] Query response successfully write to ./data/output.txt
[INFO]Execution time: -469ms

Should I blow away the database and create a new one from scratch?

Thanks!

[gabrielzhouquan]

Hi Sid,

1. For first question about response exchange, we should set durable to be false; The problem you have should be due an existing response exchange.

Could you do the following?

rabbitmqctl stop_app rabbitmqctl reset rabbitmqctl start_app to reset the rabbitmq server.

If this doesn’t work, another brute force method is as below:

cd /var/lib/rabbitmq/mnesia remove all history node records from this directory and then restart rabbitmq and Komadu server.

1. For the query question, since the query is successfully returned, it is not the problem from JS client.

Possibly it is due to conflicting raw provenance notifications in the backend database. Could you drop the Komadu database and setup a new one?

Best,

Quan (Gabriel) Zhou

On Nov 6, 2015, at 4:41 PM, sid thakur notifications@github.com wrote:

@gabrielzhouquan https://github.com/gabrielzhouquan Thanks, I checked out the node module and have integrated it with my code. Send notification works fine. Only a couple issues:

[1] In query, following line generates exception below. Changing to durable:true does not result in error and the query is executed and response returned without error: ch.assertExchange(response_ex, 'direct', {durable: false});

[PROV] Query sent sucessfully!! 2015-11-06T21:27:36.674Z - error: [local] -->[ Unhandled exception: Error: Channel closed by server: 406 (PRECONDITION-FAILED) with message "PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'KomaduExchangeQueryResponse' in vhost '/': received 'false' but current is 'true'" ] Error: Channel closed by server: 406 (PRECONDITION-FAILED) with message "PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'KomaduExchangeQueryResponse' in vhost '/': received 'false' but current is 'true'" at Channel.C.accept (/home/sthakur/.../node_modules/client-core-messaging-js/node_modules/amqplib/lib/channel.js:406:17) at Connection.mainAccept as accept at Socket.go (/home/sthakur/.../node_modules/client-core-messaging-js/nodemodules/amqplib/lib/connection.js:474:48) at Socket.emit (events.js:104:17) at emitReadable (_stream_readable.js:424:10) at emitReadable (_stream_readable.js:418:7) at readableAddChunk (_stream_readable.js:174:11) at Socket.Readable.push (_stream_readable.js:126:10) at TCP.onread (net.js:538:20) [2] Even though records seem to added via send notification, query almost always returns a sparse result (blank). I have been using the standard notification and query data in the komadu/samples/. folder.

[PROV] Query sent sucessfully!! [INFO] Waiting for response... [PROV] Query response received:

quer:activityDetailList/ /quer:getActivityDetailResponse [INFO] Query response successfully write to ./data/output.txt [INFO]Execution time: -469ms Should I blow away the database and create a new one from scratch? Thanks! — Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-154550074.

[idea-launch-lab]

@gabrielzhouquan Thanks for your quick reply! Will try out your recommendations and let you know. Thanks!

[idea-launch-lab]

@gabrielzhouquan After re-setting RabbitMQ server and restarting it, I get following error upon running komadu server:

[cluster@mesos-stag-1 ~]$ runkomadu 

Komadu standalone server started...
Listening for incoming messages...

com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.
    at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:338)
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:590)
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:612)
    at edu.indiana.d2i.komadu.messaging.Receiver.createConnection(Receiver.java:81)
    at edu.indiana.d2i.komadu.messaging.Receiver.<init>(Receiver.java:75)
    at edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable.<init>(NotificationReceiverRunnable.java:28)
    at edu.indiana.d2i.komadu.messaging.MessageDaemons.<init>(MessageDaemons.java:26)
    at edu.indiana.d2i.komadu.util.ServiceLauncher.startMessageReceiverDaemon(ServiceLauncher.java:128)
    at edu.indiana.d2i.komadu.util.ServiceLauncher.main(ServiceLauncher.java:205)

[gabrielzhouquan]

Check your username and password of messaging in komadu.config. After reset, rabbitmq will use default username “guest” with default password “guest”.

On Nov 6, 2015, at 5:02 PM, sid thakur notifications@github.com wrote:

@gabrielzhouquan https://github.com/gabrielzhouquan After re-setting RabbitMQ server and restarting it, I get following error upon running komadu server:

[cluster@mesos-stag-1 ~]$ runkomadu

Komadu standalone server started... Listening for incoming messages...

com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile. at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:338) at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:590) at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:612) at edu.indiana.d2i.komadu.messaging.Receiver.createConnection(Receiver.java:81) at edu.indiana.d2i.komadu.messaging.Receiver.(Receiver.java:75) at edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable.(NotificationReceiverRunnable.java:28) at edu.indiana.d2i.komadu.messaging.MessageDaemons.(MessageDaemons.java:26) at edu.indiana.d2i.komadu.util.ServiceLauncher.startMessageReceiverDaemon(ServiceLauncher.java:128) at edu.indiana.d2i.komadu.util.ServiceLauncher.main(ServiceLauncher.java:205) — Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-154555836.

[idea-launch-lab]

I guess I need to add users rabbitmqctl add_user <username> <password> That solved the problem. Thanks.

[idea-launch-lab]

@gabrielzhouquan I reset RabbitMQ server and added a new user ('myuser'), different than guest. I also dropped the komadu database and re-created a new one. But no I am getting following error when I attempt to start komadu. Am I missing something here? Thanks.

[cluster@mesos-stag-1 service-core]$ ./bin/KomaduServer.sh ./config/komadu.properties

Komadu standalone server started...
Listening for incoming messages...

java.io.IOException
    at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106)
    at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102)
    at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:124)
    at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:374)
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:590)
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:612)
    at edu.indiana.d2i.komadu.messaging.Receiver.createConnection(Receiver.java:81)
    at edu.indiana.d2i.komadu.messaging.Receiver.<init>(Receiver.java:75)
    at edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable.<init>(NotificationReceiverRunnable.java:28)
    at edu.indiana.d2i.komadu.messaging.MessageDaemons.<init>(MessageDaemons.java:26)
    at edu.indiana.d2i.komadu.util.ServiceLauncher.startMessageReceiverDaemon(ServiceLauncher.java:128)
    at edu.indiana.d2i.komadu.util.ServiceLauncher.main(ServiceLauncher.java:205)
Caused by: com.rabbitmq.client.ShutdownSignalException: connection error
    at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67)
    at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33)
    at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:343)
    at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:216)
    at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118)
    ... 9 more
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:196)
    at java.net.SocketInputStream.read(SocketInputStream.java:122)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
    at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:288)
    at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95)
    at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:139)
    at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:532)
    at java.lang.Thread.run(Thread.java:745)
[cluster@mesos-stag-1 service-core]$ 

RabbitMQ status:

[cluster@mesos-stag-1 ~]$ statusrabbit 
Status of node 'rabbit@mesos-stag-1' ...
[{pid,9303},
 {running_applications,[{rabbit,"RabbitMQ","3.5.3"},
                        {os_mon,"CPO  CXC 138 46","2.2.7"},
                        {xmerl,"XML parser","1.2.10"},
                        {mnesia,"MNESIA  CXC 138 12","4.5"},
                        {sasl,"SASL  CXC 138 11","2.1.10"},
                        {stdlib,"ERTS  CXC 138 10","1.17.5"},
                        {kernel,"ERTS  CXC 138 10","2.14.5"}]},
 {os,{unix,linux}},
 {erlang_version,"Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:30] [kernel-poll:true]\n"},
 {memory,[{total,28466488},
          {connection_readers,0},
          {connection_writers,0},
          {connection_channels,0},
          {connection_other,2704},
          {queue_procs,2704},
          {queue_slave_procs,0},
          {plugins,0},
          {other_proc,9077792},
          {mnesia,59888},
          {mgmt_db,0},
          {msg_index,46864},
          {other_ets,789432},
          {binary,669320},
          {code,14686409},
          {atom,1363353},
          {other_system,1768022}]},
 {alarms,[]},
 {listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,6689787084},
 {disk_free_limit,50000000},
 {disk_free,9005576192},
 {file_descriptors,[{total_limit,924},
                    {total_used,3},
                    {sockets_limit,829},
                    {sockets_used,1}]},
 {processes,[{limit,1048576},{used,125}]},
 {run_queue,0},
 {uptime,194}]
[cluster@mesos-stag-1 ~]$ 

Komadu.properties file:

#****************************************************************************                                                                
#                                                                                                                                            
# path to log4j.properties file                                                                                                              
#                                                                                                                                            
#****************************************************************************                                                                
log4j.properties.path=/home/cluster/prov-components/komadu-1.0/service-core/config/log4j.properties

#****************************************************************************                                                                
#                                                                                                                                            
# information to use RabbitMQ messaging system                                                                                               
# this section applies only when RabbitMQ is used                                                                                            
#                                                                                                                                            
#****************************************************************************                                                                

#messaging.username=guest                                                                                                                    
#messaging.password=guest                                                                                                                    
#messaging.hostname=localhost                                                                                                                

messaging.username=<myuser>
messaging.password=<mypass>
messaging.hostname=localhost

messaging.hostport=5672
messaging.virtualhost=/
messaging.exchangename=KomaduExchange
messaging.queuename=KomaduQueue
messaging.routingkey=KomaduKey

[gabrielzhouquan]

Hi Sid,

Trace back to the root of the exception stack: java.net.SocketException: Connection reset

This should be due to the settings from RabbitMQ end or your environment settings which would be out of my scope.

My suggestion is that please check your rabbitmq log at the directory /var/log for details. Another way is to re-install RabbitMQ server which would be easier.

best,

Quan (Gabriel) Zhou

On Nov 9, 2015, at 2:55 PM, sid thakur notifications@github.com wrote:

@gabrielzhouquan https://github.com/gabrielzhouquan I reset RabbitMQ server and added a new user ('myuser'), different than guest. I also dropped the komadu database and re-created a new one. But no I am getting following error when I attempt to start komadu. Am I missing something here? Thanks.

[cluster@mesos-stag-1 service-core]$ ./bin/KomaduServer.sh ./config/komadu.properties

Komadu standalone server started... Listening for incoming messages...

java.io.IOException at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106) at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102) at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:124) at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:374) at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:590) at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:612) at edu.indiana.d2i.komadu.messaging.Receiver.createConnection(Receiver.java:81) at edu.indiana.d2i.komadu.messaging.Receiver.(Receiver.java:75) at edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable.(NotificationReceiverRunnable.java:28) at edu.indiana.d2i.komadu.messaging.MessageDaemons.(MessageDaemons.java:26) at edu.indiana.d2i.komadu.util.ServiceLauncher.startMessageReceiverDaemon(ServiceLauncher.java:128) at edu.indiana.d2i.komadu.util.ServiceLauncher.main(ServiceLauncher.java:205) Caused by: com.rabbitmq.client.ShutdownSignalException: connection error at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67) at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33) at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:343) at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:216) at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118) ... 9 more Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:196) at java.net.SocketInputStream.read(SocketInputStream.java:122) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read(BufferedInputStream.java:254) at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:288) at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95) at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:139) at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:532) at java.lang.Thread.run(Thread.java:745) [cluster@mesos-stag-1 service-core]$ RabbitMQ status:

[cluster@mesos-stag-1 ~]$ statusrabbit Status of node 'rabbit@mesos-stag-1' ... [{pid,9303}, {running_applications,[{rabbit,"RabbitMQ","3.5.3"}, {os_mon,"CPO CXC 138 46","2.2.7"}, {xmerl,"XML parser","1.2.10"}, {mnesia,"MNESIA CXC 138 12","4.5"}, {sasl,"SASL CXC 138 11","2.1.10"}, {stdlib,"ERTS CXC 138 10","1.17.5"}, {kernel,"ERTS CXC 138 10","2.14.5"}]}, {os,{unix,linux}}, {erlang_version,"Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:30] [kernel-poll:true]\n"}, {memory,[{total,28466488}, {connection_readers,0}, {connection_writers,0}, {connection_channels,0}, {connection_other,2704}, {queue_procs,2704}, {queue_slave_procs,0}, {plugins,0}, {other_proc,9077792}, {mnesia,59888}, {mgmt_db,0}, {msg_index,46864}, {other_ets,789432}, {binary,669320}, {code,14686409}, {atom,1363353}, {other_system,1768022}]}, {alarms,[]}, {listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]}, {vm_memory_high_watermark,0.4}, {vm_memory_limit,6689787084}, {disk_free_limit,50000000}, {disk_free,9005576192}, {file_descriptors,[{total_limit,924}, {total_used,3}, {sockets_limit,829}, {sockets_used,1}]}, {processes,[{limit,1048576},{used,125}]}, {run_queue,0}, {uptime,194}] [cluster@mesos-stag-1 ~]$ Komadu.properties file:

****

path to log4j.properties file

****

log4j.properties.path=/home/cluster/prov-components/komadu-1.0/service-core/config/log4j.properties

****

information to use RabbitMQ messaging system

this section applies only when RabbitMQ is used

****

messaging.username=guest

messaging.password=guest

messaging.hostname=localhost

messaging.username= messaging.password= messaging.hostname=localhost

messaging.hostport=5672 messaging.virtualhost=/ messaging.exchangename=KomaduExchange messaging.queuename=KomaduQueue messaging.routingkey=KomaduKey

— Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-155172626.

[idea-launch-lab]

Okay, thanks, will check on your suggestions.

[idea-launch-lab]

@gabrielzhouquan I fixed the problem by adding permissions for my non-default user using the following:

sudo rabbitmqctl set_permissions -p / my_user_id ".*" ".*" ".*"

Thanks.

[idea-launch-lab]

Hi @gabrielzhouquan I am trying to send custom prov notification XML data but getting error on komadu terminal when I try to add a new information type in komadu_ingest_schema.xsd

Thanks.

10 Nov 2015 15:04:42,442 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Listening Queries from Mes
saging System]
10 Nov 2015 15:06:01,425 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: One Message receive
d]
<kom:addActivityEntityRelationship xmlns:kom="http://komadu.d2i.indiana.edu">
 <kom:activity>
   <kom:queryInformation>
     <kom:queryID>query_dummy_1</kom:queryID>
     <kom:timestep>unix_time_stamp</kom:timestep>
     <kom:attributes>
       <kom:attribute>
     <kom:property>body</kom:property>
     <kom:value>query_body</kom:value>
       </kom:attribute>
     </kom:attributes>
     <kom:location>none</kom:location>
   </kom:queryInformation>
 </kom:activity>
 <kom:entity>
   <kom:block>
     <kom:blockURI>entity_dummy_1</kom:blockURI>
     <kom:blockContent>query_body</kom:blockContent>
   </kom:block>
 </kom:entity>
 <kom:usage>
   <kom:queryID>query_dummy_1</kom:queryID>
   <kom:entityID>entity_dummy_1</kom:entityID>
   <kom:timestamp>2014-05-19T13:30:14.963-04:00</kom:timestamp>
 </kom:usage>
</kom:addActivityEntityRelationship>

10 Nov 2015 15:06:01,846 [Thread-0] ERROR edu.indiana.d2i.komadu.ingest.db.BaseDBIngester  -Invalid notification in ActivityEntityRelationshi
p
10 Nov 2015 15:06:01,846 [Thread-0] ERROR edu.indiana.d2i.komadu.ingest.db.BaseDBIngester  -Message : Expected elements 'workflowInformation@
http://komadu.d2i.indiana.edu serviceInformation@http://komadu.d2i.indiana.edu methodInformation@http://komadu.d2i.indiana.edu activityInform
ation@http://komadu.d2i.indiana.edu' instead of 'queryInformation@http://komadu.d2i.indiana.edu' here in element activity@http://komadu.d2i.i
ndiana.edu
10 Nov 2015 15:06:01,846 [Thread-0] ERROR edu.indiana.d2i.komadu.ingest.db.BaseDBIngester  -Message : Expected elements 'workflowInformation@
http://komadu.d2i.indiana.edu serviceInformation@http://komadu.d2i.indiana.edu methodInformation@http://komadu.d2i.indiana.edu activityInform
ation@http://komadu.d2i.indiana.edu' before the end of the content in element activity@http://komadu.d2i.indiana.edu
10 Nov 2015 15:06:01,846 [Thread-0] ERROR edu.indiana.d2i.komadu.ingest.db.BaseDBIngester  -Message : Expected element 'activityID@http://kom
adu.d2i.indiana.edu' instead of 'queryID@http://komadu.d2i.indiana.edu' here in element usage@http://komadu.d2i.indiana.edu
10 Nov 2015 15:06:01,846 [Thread-0] ERROR edu.indiana.d2i.komadu.ingest.db.BaseDBIngester  -Message : Expected element 'activityID@http://kom
adu.d2i.indiana.edu' instead of 'entityID@http://komadu.d2i.indiana.edu' here in element usage@http://komadu.d2i.indiana.edu
10 Nov 2015 15:06:01,847 [Thread-0] ERROR edu.indiana.d2i.komadu.ingest.db.BaseDBIngester  -Message : Expected element 'activityID@http://kom
adu.d2i.indiana.edu' instead of 'timestamp@http://komadu.d2i.indiana.edu' here in element usage@http://komadu.d2i.indiana.edu
10 Nov 2015 15:06:01,847 [Thread-0] ERROR edu.indiana.d2i.komadu.ingest.db.BaseDBIngester  -Message : Expected element 'activityID@http://kom
adu.d2i.indiana.edu' before the end of the content in element usage@http://komadu.d2i.indiana.edu
10 Nov 2015 15:06:01,847 [Thread-0] ERROR edu.indiana.d2i.komadu.messaging.KomaduOperations  -Ingest Exception.
10 Nov 2015 15:06:01,847 [Thread-0] ERROR edu.indiana.d2i.komadu.messaging.KomaduOperations  -[Ljava.lang.StackTraceElement;@2b42b716
10 Nov 2015 15:06:01,848 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu server: Listening to Messag
ing System]

I modified the schema in client-core-messaging-js/config/komadu_ingest_schema.xsd at following places:

    <complexType name="activityType">
        <sequence>
            <!--<element name="activityID" type="anyURI"/>-->
            <choice>
                <element name="workflowInformation" type="komadu:workflowInformationType"/>
                <element name="serviceInformation" type="komadu:serviceInformationType"/>
                <element name="methodInformation" type="komadu:methodInformationType"/>
                <!--use generic activityInformation type if the activity is not a workflow, service or method-->
                <element name="activityInformation" type="komadu:activityInformationType"/>
        <!-- 
             [PROV: custom entry]
        -->
        <element name="queryInformation" type="komadu:queryInformationType" minOccurs="0"/>

            </choice>
            <element name="location" type="string" minOccurs="0"/>
        </sequence>
    </complexType>

:
:
    <!-- 
     [PROV: custom entry type]
    -->
    <complexType name="queryInformationType">
        <sequence>
            <element name="queryID" type="anyURI"/>
            <!--<element name="workflowNodeID" type="string" minOccurs="0"/>-->
            <element name="timestep" type="int" minOccurs="0"/>
            <element name="attributes" type="komadu:attributesType" minOccurs="0"/>
            <!--<element name="instanceOf" type="komadu:instanceOfType" minOccurs="0"/>-->
        </sequence>
    </complexType>
    <!-- 
     [custom]
    -->

[idea-launch-lab]

Hi @gabrielzhouquan could you please look at my previous posting when you get chance? Any guidelines on extending/modifying Komadu XSD would be very helpful. Thank you!

[gabrielzhouquan]

HI sid,

this new information type is onlyinserted into the client side but also need to be inserted into server slide in order to let the server know this new information type.

try to update the ingest xsd from the server side. Let me know if this works.

cd service-core/src/main/resources/META-INF vi komadu_ingest_schema.xsd

On Nov 19, 2015, at 10:09 AM, sid thakur notifications@github.com wrote:

Hi @gabrielzhouquan https://github.com/gabrielzhouquan could you please look at my previous posting when you get chance? Any guidelines on extending/modifying Komadu XSD would be very helpful. Thank you!

— Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-158102879.

[idea-launch-lab]

Okay, thanks. I had updated only the client side xsd - that makes sense. Thank you.

[idea-launch-lab]

Hi @gabrielzhouquan I get following error in komadu.log when I attempt to make simple query (below). I am trying to figure out if my query document is an issue or something else going on. Thanks!

[cluster@mesos-stag-1 prov-components]$ cls; more komadu.log 

30 Nov 2015 16:15:21,790 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Starting Notification Deamon 
[0]  for receiving notifications from Client.
30 Nov 2015 16:15:21,792 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Notification Deamon [0] Start
ed.
30 Nov 2015 16:15:21,792 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Starting Query Deamon [0] for
 receiving queries from Client.
30 Nov 2015 16:15:21,793 [main] INFO  edu.indiana.d2i.komadu.messaging.MessageDaemons  -Query Deamon [0] Started.
30 Nov 2015 16:15:21,794 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Li
stening Queries from Messaging System]
30 Nov 2015 16:15:21,795 [Thread-0] INFO  edu.indiana.d2i.komadu.messaging.NotificationReceiverRunnable  -[Komadu ser
ver: Listening to Messaging System]
30 Nov 2015 16:15:34,153 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: On
e Query received]
ce92d925-c39d-e608-9792-2c757598c2fd#<quer:findActivityRequest xmlns:quer="http://komadu.d2i.indiana.edu/query">
  <quer:workflowID>workflow_default</quer:workflowID>
</quer:findActivityRequest>

30 Nov 2015 16:15:34,153 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Karma server: Que
ry] <quer:findActivityRequest xmlns:quer="http://komadu.d2i.indiana.edu/query">
  <quer:workflowID>workflow_default</quer:workflowID>
</quer:findActivityRequest>
30 Nov 2015 16:15:34,153 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Karma server: Que
ry Response Routing Key] ce92d925-c39d-e608-9792-2c757598c2fd
30 Nov 2015 16:15:34,554 [Thread-1] ERROR edu.indiana.d2i.komadu.query.db.BaseDBQuerier  -Exiting findActivity() with
 SQL errors.
30 Nov 2015 16:15:34,554 [Thread-1] ERROR edu.indiana.d2i.komadu.query.db.BaseDBQuerier  -com.mysql.jdbc.exceptions.j
dbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL
 server version for the right syntax to use near 'AND a.context_workflow_uri LIKE '%workflow_default%'' at line 1
30 Nov 2015 16:15:34,555 [Thread-1] ERROR edu.indiana.d2i.komadu.messaging.KomaduOperations  -Error while Querying.
java.lang.NullPointerException
    at edu.indiana.d2i.komadu.messaging.KomaduOperations.queryProvenance(KomaduOperations.java:90)
    at edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable.run(QueryReceiverRunnable.java:62)
    at java.lang.Thread.run(Thread.java:745)
30 Nov 2015 16:15:34,567 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Se
nd Query Response to Client]
30 Nov 2015 16:15:34,574 [Thread-1] INFO  edu.indiana.d2i.komadu.messaging.QueryReceiverRunnable  -[Komadu server: Li
stening Queries from Messaging System]

query document:

<quer:findActivityRequest xmlns:quer="http://komadu.d2i.indiana.edu/query">
  <quer:workflowID>workflow_default</quer:workflowID>
</quer:findActivityRequest>

Existing record in MySql database that I expect to be returned:

      2 | 2015-11-30 14:30:20 | ADD_AGENT_ACTIVITY_RELATION    | ERROR             | <kom:addAgentActivityRelationship xmlns:kom="http://komadu.d2i.indiana.edu">
  <kom:agent>
    <kom:userAgent>
      <kom:agentID>skylr_user_0</kom:agentID>
      <kom:attributes>
        <kom:attribute>
          <kom:property>user_name</kom:property>
          <kom:value>user_name_from_dict</kom:value>
        </kom:attribute>
        <kom:attribute>
          <kom:property>timestamp</kom:property>
          <kom:value>curr_time</kom:value>
        </kom:attribute>
        <kom:attribute>
          <kom:property>address</kom:property>
          <kom:value>NCSU</kom:value>
        </kom:attribute>
      </kom:attributes>
      <kom:fullName>not captured</kom:fullName>
      <kom:affiliation>NCSU/LAS</kom:affiliation>
      <kom:email>not captured</kom:email>
    </kom:userAgent>
    <kom:type>PERSON</kom:type>
  </kom:agent>
  <kom:activity>
    <kom:workflowInformation>
      <kom:workflowID>workflow_default</kom:workflowID>
      <kom:workflowNodeID>node1</kom:workflowNodeID>
      <kom:timestep>0</kom:timestep>
      <kom:attributes>
        <kom:attribute>
          <kom:property>domain</kom:property>
          <kom:value>LAS</kom:value>
        </kom:attribute>
      </kom:attributes>
    </kom:workflowInformation>
    <kom:location>not captured</kom:location>
  </kom:activity>
  <kom:association>
    <kom:activityID>workflow_default</kom:activityID>
    <kom:agentID>skylr_user_0</kom:agentID>
    <kom:attributes>
      <kom:attribute>
        <kom:property>data_originator</kom:property>
        <kom:value>user</kom:value>
      </kom:attribute>
    </kom:attributes>
  </kom:association>
</kom:addAgentActivityRelationship> |
...

[idea-launch-lab]

@gabrielzhouquan Hi, any clues to issue in my last post? Thanks.

[gabrielzhouquan]

Hi Sid,

I can see you are using find activity request query xml.

Could u using the following schema instead?

quer:nameworkflow1/quer:name quer:nextActivityIDservice1_58/quer:nextActivityID /quer:findActivityRequest Element “name" is the name of your targeting activity and “nextActivityID" is the activity which is next to your targeting activity. (In case some activities share the same name) Best, Gabriel > On Dec 8, 2015, at 10:41 AM, sid thakur notifications@github.com wrote: > > @gabrielzhouquan https://github.com/gabrielzhouquan Hi, any clues to issue in my last post? Thanks. > > — > Reply to this email directly or view it on GitHub https://github.com/Data-to-Insight-Center/komadu/issues/5#issuecomment-162921528.