error: Error getting view results Error: missing

[iStooge]

I am always getting the following error when trying to connect to the cognitive UI via the given link bei Hubot:

[Fri Oct 28 2016 13:33:11 GMT+0200 (CEST)] - error: Error getting view results Error: missing at Request._callback (/.../node_modules/hubot-ibmcloud-cognitive-ui/node_modules/cloudant/node_modules/nano/lib/nano.js:248:15) at Request.self.callback (/.../Hubot/node_modules/request/request.js:187:22) at emitTwo (events.js:87:13) at Request.emit (events.js:172:7) at Request. (/.../Hubot/node_modules/request/request.js:1048:10) at emitOne (events.js:77:13) at Request.emit (events.js:169:7) at IncomingMessage. (/.../Hubot/node_modules/request/request.js:969:12) at emitNone (events.js:72:20) at IncomingMessage.emit (events.js:166:7) at endReadableNT (_stream_readable.js:921:12) at nextTickCallbackWith2Args (node.js:442:9) at process._tickCallback (node.js:356:17)

See the picture for whats happening in the browser

[mbecvarik]

It looks like it might be a problem with the database setup. Are you using it with our Cloudbot and hubot-ibmcloud-nlc modules?

[jlpadilla]

Also, I fixed a problem with hubot-ibmcloud-cognitive-lib that caused the database to not be correctly initialized in certain conditions, so please make sure you update this dependency to version 0.1.5 or later.

[iStooge]

@jlpadilla just updated to 0.1.5 and still not working...

@mbecvarik i have hubot-ibmcloud-nlc running. Not with cloudbot but with the standard Hubot.

Here are the packages i have installed form my package.json. Note that i installed hubot-ibmcloud-cognitive-ui directly from github with npm because it is not available with npm directly.

Perheps you can help me?

{ "dependencies": { "htmlparser": "1.7.6", "hubot": "^2.19.0", "hubot-auth": "^2.0.0", "hubot-diagnostics": "0.0.1", "hubot-dynamic-conversation": "^1.0.1", "hubot-env": "0.0.2", "hubot-environment": "0.0.9", "hubot-google-images": "^0.2.6", "hubot-google-translate": "^0.2.0", "hubot-help": "^0.2.0", "hubot-heroku-keepalive": "^1.0.2", "hubot-ibmcloud-activity-emitter": "0.0.3", "hubot-ibmcloud-cloudant": "0.0.9", "hubot-ibmcloud-cognitive-entities": "0.0.8", "hubot-ibmcloud-cognitive-lib": "^0.1.5", "hubot-maps": "0.0.2", "hubot-pugme": "^0.1.0", "hubot-recognizer": "^1.0.2", "hubot-redis-brain": "0.0.3", "hubot-reload-scripts": "^0.1.4", "hubot-rocketchat": "^1.0.4", "hubot-rules": "^0.1.1", "hubot-scripts": "^2.17.2", "hubot-shipit": "^0.2.0", "hubot-tweets": "0.0.1", "hubot-twitter-search": "^1.0.3", "i18n-2": "^0.6.3", "soupselect": "0.2.0" }, "devDependencies": { "hubot-ibmcloud-auth": ">=0.0.8", "hubot-ibmcloud-nlc": ">=0.0.20" }, "engines": { "node": "0.10.x" } }

[jlpadilla]

Here's some things to try.

• In the drop down near the top select the database 'nlc'.
• Check if you have documents in the Cloudant database: nlc.

• I think that your database might be missing the document botInfo, which identifies the database as a database that contains bot cognitive training data. If that's missing, the easiest way to get your UI working is to create this document in your Cloudant database. The simplest route is to create the doc using the Cloudant UI, otherwise you could use a simple script like this one.

  // Untested sample code to add botInfo document.
  const Cloudant = require('cloudant');
  let cloudant = Cloudant({account: env.username, password: env.userpass});
  let db = cloudant.use('nlc');
  db.insert({_id: 'botInfo', botName: 'myBot', localDbName: 'nlc'}, function(err, body){
  if (err) {
    console.log('Error inserting document: ', err);
  }
  else {
    console.log('Document has been inserted. ', body);
  }
  });

• Once you have added the botInfo document, refresh the UI a couple times. The first refresh should create the necessary views and hopefully the second refresh will load the UI.

[iStooge]

@jlpadilla : Thank you so much for your help! The hint with the missing document helped, so now i get a connection to my database. The problem is, that still nothing is listed there. Hubot did already some classifications with different types of confidence. But nothing is appearing in the database!

In which database will the information be stored? NLC or RR? Does some option has to be enabled that the system tracks the classifications done?

HUBOT_COGNITIVE_FEEDBACK_ENABLED is enabled...

Thank you so much so far!

[jlpadilla]

• The data that you are looking for is stored in the database nlc. rr is used for a different purpose.
• Since you can see the nlc and rr databases from the UI, I think your config is correct.
• Hubot stores the data about your classifications and confidence in a local database, which then synchronizes with your Cloudant database. By default this happens every 30 minutes. I recommend that you lower the sync interval process.env.SYNC_INTERVAL on your Hubot and verify that the new records are being copied by looking at the logs.

Hopefully you'll see the log message for complete, but here is the reference to possible error messages. https://github.com/ibm-cloud-solutions/hubot-ibmcloud-cognitive-lib/blob/master/src/lib/dbManager.js#L128

[iStooge]

The problem is that I still can't see the nlc database from the UI and it also gets not synced. Only the database rr is syncing. Is this still a problem with the configuration of the cloudant database nlc?

This is the document, i created in the cloudant database nlc: { "_id": "botInfo", "_rev": "1-e23a7ed5544d75bcd36fc3a71b994a77", "botName": "hubot", "localDbName": "databases/nlc" }

Or is the missing sync a local problem?

[jlpadilla]

Looks like you don't have any documents in the nlc database, other than the botInfo doc, that you created manually. If that's correct, then the problem is most likely with sync.

The database sync is initiated from your Hubot. It's interesting that you mention that rr is syncing. I assume you are seeing success log messages from the Hubot console. If that's correct your configuration is most likely ok because the sync function is common to both databases.

Let's start by checking the data in your local database. The local database is PouchDB, you can use the following node script to check if the nlc database has documents that need to be sync. If we have documents, then we need a closer look at sync, if there's no documents then it's something else. Note that documents with storageType: 'private' won't sync.

const PouchDB = require('pouchdb');
const db = new PouchDB('/absolute/path/to/databases/nlc');
db.allDocs({include_docs: true}).then((docs) => {
  docs.rows.map((record) => {
    if(record.doc.storageType !== 'private') {
      console.log('=====\n', record);
    }
  });
});

• You need to change the db location to the absolute path to your /databases/nlc directory, that's most likely the root of your hubot project, unless you are using HUBOT_DB_PATH.
• You may need to npm install pouchdb

[iStooge]

So I just checked the database with your script and there are documents in the Database. Here is the Output of nlc:

=====
 { id: 'EDC948CF-0E4B-0D2F-9C5E-1D638931CF25',
  key: 'EDC948CF-0E4B-0D2F-9C5E-1D638931CF25',
  value: { rev: '1-53c0161d3e8e1789402e9f6e160eb9f2' },
  doc:
   { type: 'classified',
     ts: 1478098899221,
     classification:
      { classifier_id: 'f48968x109-nlc-2306',
        url: 'https://gateway.watsonplatform.net/natural-language-classifier/api/v1/classifiers/f48968x109-nlc-2306',
        text: 'set the topic to automobile',
        top_class: 'topic.set',
        classes: [Object] },
     lowConfidenceThreshold: 0.05,
     highConfidenceThreshold: 0.8,
     botVersion: '0.0.0',
     botName: 'imo',
     _id: 'EDC948CF-0E4B-0D2F-9C5E-1D638931CF25',
     _rev: '1-53c0161d3e8e1789402e9f6e160eb9f2' } }

=====
 { id: 'IBMcloudCloudant_databasename',
  key: 'IBMcloudCloudant_databasename',
  value: { rev: '1-78a2e1401bf7374c01168f0f2e35e005' },
  doc:
   { values: [ 'nlc', 'rr' ],
     _id: 'IBMcloudCloudant_databasename',
     _rev: '1-78a2e1401bf7374c01168f0f2e35e005' } }

=====
 { id: 'botInfo',
  key: 'botInfo',
  value: { rev: '1-d43a4e08fcac850cee7f452ca6d52827' },
  doc:
   { botName: 'hubot',
     localDbName: 'databases/nlc',
     _id: 'botInfo',
     _rev: '1-d43a4e08fcac850cee7f452ca6d52827' } }

So it seems to be a problem with the sync... Another thing I noticed: In the folder databases there are also 2 additional folders. One is called nlc-mrview-dea30aa2c4e1366c50eb3c3851103efd/

Here is the Output of this database:

=====
 { id: '54question.get\u00004hubot-ideaSession_4_v0.0.1\u0000\u0000',
  key: '54question.get\u00004hubot-ideaSession_4_v0.0.1\u0000\u0000',
  value: { rev: '1-e60a32273941bd437c9971133a6f8e10' },
  doc:
   { _id: '54question.get\u00004hubot-ideaSession_4_v0.0.1\u0000\u0000',
     _rev: '1-e60a32273941bd437c9971133a6f8e10' } }

=====
 { id: '54question.get\u00004hubot-ideaSession_5_v0.0.1\u000031\u0000\u0000',
  key: '54question.get\u00004hubot-ideaSession_5_v0.0.1\u000031\u0000\u0000',
  value: { rev: '1-1273244add2b40ad5d4c8d886342119c' },
  doc:
   { _id: '54question.get\u00004hubot-ideaSession_5_v0.0.1\u000031\u0000\u0000',
     _rev: '1-1273244add2b40ad5d4c8d886342119c' } }
=====

[jlpadilla]

If I understand correctly, previously you created the botInfo document in your remote Cloudant database. Now, I noticed the botInfo document in the output you provided from your local database. Unless I'm missing something I believe that this document was created in the local database by the sync process, if that's correct we can confirm that sync worked. I don't think this document was created by your bot because it would have more fields and on I see that your bot name is imo from the first document pasted.

Did you only see the 3 documents pasted above in your database? The first document suggest that things are good and your bot classified your text correctly, at least that one time. But that record wouldn't show in the UI because it has type: 'classified', which means we got it right, so we don't need to do anything with it, we just collected the information to measure how good the bot is performing.

We need to see ore documents in the database. More specifically we need to see some documents with type: 'unclassified' or type: 'learned', which is what the UI works with.

At this point I think you need to interact more with your bot and check if your interactions are being recorded in the database. Try to send at least 10 messages to the bot, trying to ask questions that the bot isn't trained to handle, so we get more unclassified and learned records in the local db. The reason we need several interactions is because the bot captures the context of the conversation (what was said before and after the missed classification), so writing to the db gets delayed until we have collected the context data.

Hope this information helps. Looks like you are very close!

[iStooge]

I just tried to talk more with the bot, to get some data into the database. It worked locally, but again the data did not get synced.

So i just started all over and deleted the local database as well as the databases in cloudent. When i now start hubot, the databases get build locally. So also the document we just talked about with the botinfo gets created locally! It sayed hubot because I did not set the ennvironment variable HUBOT_NAME. So now when i check both the databases, in both there are the documents with the bot info.

Then the sync-process gets started and the database rr gets created in Cloudent. But not nlc. So I think the problem is still the syncing, but I have just no idea why rrgets synced and nlcnot...

[jlpadilla]

I think I found the bug.

It's in the hubot-ibmcloud-cognitive-lib project, when creating the DBManager object in this line

const nlcDb = new DBManager({localDbName: 'nlc', remoteDbName: env.cloudantDb});

you can see that it sets the remoteDbName, but when you look at env.js the property cloudantDb no longer exist.

So the database nlc isn't synced because an undefined remoteDbName means we should skip replication (see here).

Please open an issue on https://github.com/ibm-cloud-solutions/hubot-ibmcloud-cognitive-lib and verify that this change solves your problem. I'll issue a PR and publish a new version once I get confirmation that it resolved your problem.

[jlpadilla]

@iStooge I'm closing this issue because we are confident that the root cause is sync, which is in https://github.com/ibm-cloud-solutions/hubot-ibmcloud-cognitive-lib

Let's continue this discussion in the right repo. Thanks for your patience! :)

[iStooge]

Great!! Now the sync is working :) That was the problem. I will open the issue later.

But still there is the problem left, that i can't see the database nlc in the Cognitive UI... Does this have to to do anything with the same problem or is this a different problem?

See the screenshot:

[jlpadilla]

@iStooge I've created PR #23 with the fix for the UI problem. Please review it and we can merge it.

[iStooge]

Thank you so much for your help! Its working now :)