Problem in getting any new corpus to work on docker version of conTEXT

[Analect]

Hi. I followed your README for a docker install, which appeared to work OK. The only change to config.js I made was to mongodb:// setting, as suggested and when running the 'context' container I opt to link it to an existing mongodb container (although one that isn't using the -v option to a local host directory). I tried to add various RSS feeds for testing. http://ftalphaville.ft.com/feed/ http://feeds.bcci.co.uk/news/rss.xml?edition=int ... and a few others ... but each time, it crashes the instance and the container falls over, requiring me to do a 'docker start context' each time. On looking at 'docker logs context', I see a similar pattern each time ... a series of json messages (see below). The logs below refer to the bbc rss being added and then immediately crashing. The same 'Unhandled error event' happened for my various other attempts. I then used 'nsenter' to access the context container with a view to running npm test, which might throw light on the problem. I tried to run this from /usr/local/bin ... but it didn't work. Maybe it requires me to somehow stop the node process first? Any pointers on troubleshooting would be much appreciated. I don't suppose it's possible to attach an email archive (e.g. a *.pst file) as a corpus? Thanks again. Colum

{"name":"context","hostname":"b3ea3a3dd8d8","pid":1,"level":30,"msg":"[route-file]: captcha.js:","time":"2014-09-14T14:45:10.434Z","v":0} {"name":"context","hostname":"b3ea3a3dd8d8","pid":1,"level":30,"msg":"[route-file]: corpus.js:","time":"2014-09-14T14:45:10.438Z","v":0} {"name":"context","hostname":"b3ea3a3dd8d8","pid":1,"level":30,"msg":"[route-file]: user.js:","time":"2014-09-14T14:45:10.673Z","v":0} {"name":"context","hostname":"b3ea3a3dd8d8","pid":1,"level":30,"msg":"[route-file]: auth.facebook.js:","time":"2014-09-14T14:45:10.674Z","v":0} {"name":"context","hostname":"b3ea3a3dd8d8","pid":1,"level":30,"msg":"[route-file]: auth.google.js:","time":"2014-09-14T14:45:10.692Z","v":0} {"name":"context","hostname":"b3ea3a3dd8d8","pid":1,"level":30,"msg":"[route-file]: auth.linkedin.js:","time":"2014-09-14T14:45:10.712Z","v":0} {"name":"context","hostname":"b3ea3a3dd8d8","pid":1,"level":30,"msg":"[route-file]: auth.local.js:","time":"2014-09-14T14:45:10.731Z","v":0} {"name":"context","hostname":"b3ea3a3dd8d8","pid":1,"level":30,"msg":"[route-file]: auth.twitter.js:","time":"2014-09-14T14:45:10.735Z","v":0} {"name":"context","hostname":"b3ea3a3dd8d8","pid":1,"level":30,"msg":"[route-file]: auth.js:","time":"2014-09-14T14:45:10.753Z","v":0} {"name":"context","hostname":"b3ea3a3dd8d8","pid":1,"level":30,"msg":"[route-file]: corpus.js:","time":"2014-09-14T14:45:10.753Z","v":0} {"name":"context","hostname":"b3ea3a3dd8d8","pid":1,"level":30,"msg":"[route-file]: index.js:","time":"2014-09-14T14:45:10.754Z","v":0} {"name":"context","hostname":"b3ea3a3dd8d8","pid":1,"level":30,"msg":"\n listening on port 8080\n","time":"2014-09-14T14:45:10.758Z","v":0} {"name":"context","hostname":"b3ea3a3dd8d8","pid":1,"level":30,"msg":"closing websocket 5415aab4a08d620100937037","time":"2014-09-14T14:48:20.137Z","v":0} {"name":"context","hostname":"b3ea3a3dd8d8","pid":1,"level":30,"msg":"Starting progress socket at: ws://localhost:8081/corpus/5415aab4a08d620100937037","time":"2014-09-14T14:48:20.137Z","v":0} {"name":"context","hostname":"b3ea3a3dd8d8","pid":1,"level":30,"msg":"starting to process input from corpus 5415aab4a08d620100937037","time":"2014-09-14T14:48:20.138Z","v":0} {"name":"context","hostname":"b3ea3a3dd8d8","pid":1,"level":50,"msg":"error processing corpus input! { __v: 0,\n user: 5415a5f22ecc290100d8894c,\n input_type: 'feed',\n input: 'http://feeds.bbci.co.uk/news/rss.xml?edition=int',\n input_count: 50,\n name: 'BBC News',\n _id: 5415aab4a08d620100937037,\n files: [],\n nlp_api: 'DBpedia-Spotlight',\n creation_date: Sun Sep 14 2014 14:48:20 GMT+0000 (UTC),\n processed: false } [TypeError: Object false has no method 'setHeader']","time":"2014-09-14T14:48:20.153Z","v":0} {"name":"context","hostname":"b3ea3a3dd8d8","pid":1,"level":30,"msg":"closing websocket 5415abf0a08d620100937038","time":"2014-09-14T14:53:36.729Z","v":0} {"name":"context","hostname":"b3ea3a3dd8d8","pid":1,"level":30,"msg":"Starting progress socket at: ws://localhost:8081/corpus/5415abf0a08d620100937038","time":"2014-09-14T14:53:36.729Z","v":0} {"name":"context","hostname":"b3ea3a3dd8d8","pid":1,"level":30,"msg":"starting to process input from corpus 5415abf0a08d620100937038","time":"2014-09-14T14:53:36.730Z","v":0}

events.js:72 throw er; // Unhandled 'error' event ^ Error: listen EADDRINUSE at errnoException (net.js:904:11) at Server._listen2 (net.js:1042:14) at listen (net.js:1064:10) at net.js:1146:9 at dns.js:72:18 at process._tickCallback (node.js:419:13)

[yamalight]

It seems that conTEXT is not able to open WebSocket port inside of your docker container for some reason. Try changin defaultSocketPort in config.js to something else. Let me know if that helps.

[Analect]

Hi @yamalight for your response. I changed to port 9001 in config as you suggested and rebuilt the image and generated a new container. However, this doesn't appear to resolve the issue ... after I generated a new corpus, it doesn't appear to pull in any content. These are the logs from the 'context' container on my attempt to create two rss feeds.

ubuntu:~/Development/Tools/context$ docker logs context {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"[route-file]: captcha.js:","time":"2014-09-16T08:32:48.176Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"[route-file]: corpus.js:","time":"2014-09-16T08:32:48.181Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"[route-file]: user.js:","time":"2014-09-16T08:32:48.391Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"[route-file]: auth.facebook.js:","time":"2014-09-16T08:32:48.392Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"[route-file]: auth.google.js:","time":"2014-09-16T08:32:48.402Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"[route-file]: auth.linkedin.js:","time":"2014-09-16T08:32:48.423Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"[route-file]: auth.local.js:","time":"2014-09-16T08:32:48.441Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"[route-file]: auth.twitter.js:","time":"2014-09-16T08:32:48.444Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"[route-file]: auth.js:","time":"2014-09-16T08:32:48.460Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"[route-file]: corpus.js:","time":"2014-09-16T08:32:48.461Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"[route-file]: index.js:","time":"2014-09-16T08:32:48.461Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"\n listening on port 8080\n","time":"2014-09-16T08:32:48.464Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"closing websocket 5417f61e7d144701007ddc45","time":"2014-09-16T08:34:38.996Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"Starting progress socket at: ws://localhost:9001/corpus/5417f61e7d144701007ddc45","time":"2014-09-16T08:34:38.997Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"starting to process input from corpus 5417f61e7d144701007ddc45","time":"2014-09-16T08:34:38.998Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":50,"msg":"error processing corpus input! { __v: 0,\n user: 5415a5f22ecc290100d8894c,\n input_type: 'feed',\n input: 'http://ftalphaville.ft.com/feed/',\n input_count: 50,\n name: 'FTA3',\n _id: 5417f61e7d144701007ddc45,\n files: [],\n nlp_api: 'DBpedia-Spotlight',\n creation_date: Tue Sep 16 2014 08:34:38 GMT+0000 (UTC),\n processed: false } [TypeError: Object false has no method 'setHeader']","time":"2014-09-16T08:34:39.011Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"closing websocket 5417fe8d7d144701007ddc46","time":"2014-09-16T09:10:37.756Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"Starting progress socket at: ws://localhost:9001/corpus/5417fe8d7d144701007ddc46","time":"2014-09-16T09:10:37.757Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"starting to process input from corpus 5417fe8d7d144701007ddc46","time":"2014-09-16T09:10:37.757Z","v":0}

events.js:72 throw er; // Unhandled 'error' event ^ Error: listen EADDRINUSE at errnoException (net.js:904:11) at Server._listen2 (net.js:1042:14) at listen (net.js:1064:10) at net.js:1146:9 at dns.js:72:18 at process._tickCallback (node.js:419:13) {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"[route-file]: captcha.js:","time":"2014-09-16T13:23:41.113Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"[route-file]: corpus.js:","time":"2014-09-16T13:23:41.117Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"[route-file]: user.js:","time":"2014-09-16T13:23:41.338Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"[route-file]: auth.facebook.js:","time":"2014-09-16T13:23:41.339Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"[route-file]: auth.google.js:","time":"2014-09-16T13:23:41.351Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"[route-file]: auth.linkedin.js:","time":"2014-09-16T13:23:41.373Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"[route-file]: auth.local.js:","time":"2014-09-16T13:23:41.392Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"[route-file]: auth.twitter.js:","time":"2014-09-16T13:23:41.395Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"[route-file]: auth.js:","time":"2014-09-16T13:23:41.413Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"[route-file]: corpus.js:","time":"2014-09-16T13:23:41.414Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"[route-file]: index.js:","time":"2014-09-16T13:23:41.415Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"\n listening on port 8080\n","time":"2014-09-16T13:23:41.419Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"closing websocket 54183a6aeb1e750100a3994a","time":"2014-09-16T13:26:02.432Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"Starting progress socket at: ws://localhost:9001/corpus/54183a6aeb1e750100a3994a","time":"2014-09-16T13:26:02.433Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"starting to process input from corpus 54183a6aeb1e750100a3994a","time":"2014-09-16T13:26:02.435Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":50,"msg":"error processing corpus input! { __v: 0,\n user: 5415a5f22ecc290100d8894c,\n input_type: 'feed',\n input: 'http://feeds.bbci.co.uk/news/uk/rss.xml',\n input_count: 50,\n name: 'BBC UK',\n _id: 54183a6aeb1e750100a3994a,\n files: [],\n nlp_api: 'DBpedia-Spotlight',\n creation_date: Tue Sep 16 2014 13:26:02 GMT+0000 (UTC),\n processed: false } [TypeError: Object false has no method 'setHeader']","time":"2014-09-16T13:26:02.454Z","v":0}

[yamalight]

it seems that request.js is not properly loaded in feed module. can you please try running gulp test and copy input here?

[Analect]

Just to be clear, from where do I run that ... inside the 'context' docker container ... or back in the host where I cloned this repo? Thanks

[yamalight]

oh, sorry. should've started with that :) if you don't have node installed, then inside of the docker. otherwise outside of the docker should work too.

[Analect]

I'm probably being stupid here ... I ran it inside the container ... but suspect I probably need to navigate into somewhere within the file structure first where a gulpfile exists?

bin boot data dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var root@02fd477c229b:/# gulp test [14:07:14] Local gulp not found in / [14:07:14] Try running: npm install gulp root@02fd477c229b:/# npm install gulp gulp@3.8.8 node_modules/gulp ├── interpret@0.3.7 ├── pretty-hrtime@0.2.1 ├── deprecated@0.0.1 ├── archy@0.0.2 ├── minimist@1.1.0 ├── semver@3.0.1 ├── tildify@1.0.0 (user-home@1.0.0) ├── orchestrator@0.3.7 (stream-consume@0.1.0, sequencify@0.0.7, end-of-stream@0.1.5) ├── liftoff@0.12.1 (extend@1.3.0, minimist@0.2.0, resolve@0.7.4, findup-sync@0.1.3) ├── vinyl-fs@0.3.7 (graceful-fs@3.0.2, mkdirp@0.5.0, lodash@2.4.1, strip-bom@1.0.0, vinyl@0.4.3, through2@0.6.1, glob-stream@3.1.15, glob-watcher@0.0.6) ├── chalk@0.5.1 (escape-string-regexp@1.0.2, ansi-styles@1.1.0, supports-color@0.2.0, has-ansi@0.1.0, strip-ansi@0.3.0) └── gulp-util@3.0.1 (lodash._reinterpolate@2.4.1, dateformat@1.0.8, lodash@2.4.1, vinyl@0.4.3, through2@0.6.1, multipipe@0.1.1, lodash.template@2.4.1) root@02fd477c229b:/# gulp test [14:08:21] No gulpfile found root@02fd477c229b:/#

[yamalight]

you have to either install it globally (using npm install -g gulp) or run it from node_modules/.bin folder (using ./node_modules/.bin/gulp)

[Analect]

OK, I'm still trying to run 'gulp test' ... but it's still complaining it has no gulp file. Should I be running that gulp command against a specific 'test' folder or file located elsewhere in the container? Thanks for your patience!

bin boot data dev etc home lib lib64 media mnt node_modules opt proc root run sbin srv sys tmp usr var root@02fd477c229b:/# /node_modules/.bin/gulp test [14:27:41] No gulpfile found

[Analect]

OK ... figured out I should be navigating to the /opt/app folder in the container and then running gulp test there. Here is the output:

root@02fd477c229b:/opt/app# /node_modules/.bin/gulp test [14:52:16] Using gulpfile /opt/app/gulpfile.js [14:52:16] Starting 'jshint'... [14:52:16] Starting 'mocha'... [14:52:16] Starting 'jscs'...

Mocha test suite ✓ should be ok

DBPedia annotation suit [14:52:17] Finished 'jshint' after 1.13 s [14:52:17] 'jscs' errored after 1.12 s [14:52:17] Error in plugin 'gulp-jscs' Message: Operator + should not stick to following expression at config.js : 18 | 19 |// default db 20 |exports.db = 'mongodb://' + process.env.MONGODB_PORT_28017_TCP_ADDR +'/context'; ----------------------------------------------------------------------------^ 21 | 22 |// session config

Illegal space before opening round brace at inputProcessing/blogger.js : 158 | 159 |// module 160 |var BloggerProcessing = function () { --------------------------------^ 161 | ProgressReporter.call(this); 162 |

Illegal space before opening round brace at inputProcessing/direct.js : 28 | 29 |// module 30 |var DirectProcessing = function () { -------------------------------^ 31 | ProgressReporter.call(this); 32 |

Illegal space before opening round brace at inputProcessing/feed.js : 15 |var getFeedEntries = function(corpus, self) { 16 | // return promise 17 | return new Promise(function (resolve, reject) { -------------------------------^ 18 | // get vars from corpus 19 | var url = corpus.input;

Illegal space before opening round brace at inputProcessing/feed.js : 81 | req.on('error', function(err) { 82 | reject(err); 83 | }).on('response', function (res) { ----------------------------------^ 84 | var stream = this; 85 |

Illegal space before opening round brace at inputProcessing/feed.js : 103 | 104 |// module 105 |var FeedProcessing = function () { -----------------------------^ 106 | ProgressReporter.call(this); 107 |

Illegal space before opening round brace at inputProcessing/files.js : 52 | 53 |// module 54 |var FilesProcessing = function () { ------------------------------^ 55 | ProgressReporter.call(this); 56 |

Illegal space before opening round brace at inputProcessing/webpage.js : 31 | 32 |// module 33 |var WebpageProcessing = function () { --------------------------------^ 34 | ProgressReporter.call(this); 35 |

Illegal space before opening round brace at inputProcessing/wordpress.js : 186 | 187 |// module 188 |var WordpressProcessing = function () { ----------------------------------^ 189 | ProgressReporter.call(this); 190 |

Illegal space before opening round brace at modules/corpus.js : 6 | 7 |// config routes 8 |corpusModule.config(function ($stateProvider, $urlRouterProvider) { ----------------------------^ 9 | // main view 10 | $stateProvider.state('corpus', {

Operator - should stick to operand at controllers/corpus/facets.js : 59 | case 'article': 60 | if ($scope.entityFilter !== undefined) { 61 | filterEntity = item.entities.indexOf($scope.entityFilter) !== - ------------------------------------------------------------------------------------------^ 62 | 1; 63 | } else {

Operator - should stick to operand at controllers/corpus/facets.js : 72 | case 'entity': 73 | if ($scope.articleFilter !== undefined) { 74 | filterArticle = item.articles.indexOf($scope.articleFilter) !== - --------------------------------------------------------------------------------------------^ 75 | 1; 76 | } else {

Operator - should stick to operand at controllers/corpus/facets.js : 85 | case 'type': 86 | if ($scope.articleFilter !== undefined) { 87 | filterArticle = item.articles.indexOf($scope.articleFilter) !== - --------------------------------------------------------------------------------------------^ 88 | 1; 89 | } else {

Operator - should stick to operand at controllers/corpus/facets.js : 91 | } 92 | if ($scope.entityFilter !== undefined) { 93 | filterEntity = item.entities.indexOf($scope.entityFilter) !== - ------------------------------------------------------------------------------------------^ 94 | 1; 95 | } else { ✓ should get at least 29 entities (7821ms)

FOX annotation suit ✓ should get X entities (2373ms)

Blogger input processing suit ✓ should get 50 results (17964ms)

Direct input processing suit ✓ should get 1 results

Feed input processing suit Possibly unhandled TypeError: Object false has no method 'setHeader' at /opt/app/modules/inputProcessing/feed.js:79:13 at tryCatch2 (/opt/app/node_modules/bluebird/js/main/util.js:51:21) at Promise$_resolveFromResolver as _resolveFromResolver at new Promise (/opt/app/node_modules/bluebird/js/main/promise.js:82:37) at getFeedEntries (/opt/app/modules/inputProcessing/feed.js:17:12) at null. (/opt/app/modules/inputProcessing/feed.js:99:25) at tryBlock (/opt/app/node_modules/asyncawait/src/async/fiberManager.js:42:33) at runInFiber (/opt/app/node_modules/asyncawait/src/async/fiberManager.js:30:9) 1) should get 12 results

Files input processing suit ✓ should get 3 results

Slidewiki input processing suit 2) should get 50 results

Twitter input processing suit ✓ should get 100 results for new profile (9955ms)

Webpage input processing suit ✓ should get 1 results (1127ms)

Wordpress input processing suit 3) should get 50 results

8 passing (2m) 3 failing

1) Feed input processing suit should get 12 results: Error: timeout of 20000ms exceeded at null. (/opt/app/node_modules/gulp-mocha/node_modules/mocha/lib/runnable.js:157:19) at Timer.listOnTimeout as ontimeout

2) Slidewiki input processing suit should get 50 results: Error: timeout of 20000ms exceeded at null. (/opt/app/node_modules/gulp-mocha/node_modules/mocha/lib/runnable.js:157:19) at Timer.listOnTimeout as ontimeout

3) Wordpress input processing suit should get 50 results: Error: timeout of 20000ms exceeded at null. (/opt/app/node_modules/gulp-mocha/node_modules/mocha/lib/runnable.js:157:19) at Timer.listOnTimeout as ontimeout

stream.js:94 throw er; // Unhandled stream error in pipe. ^ Error: 3 tests failed. root@02fd477c229b:/opt/app#

[yamalight]

can you please try changing ./modules/inputProcessing/feed.js line 78-79 from:

req.setHeader('user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36')
    .setHeader('accept', 'text/html,application/xhtml+xml');

to:

req.setHeader('user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36');
req.setHeader('accept', 'text/html,application/xhtml+xml');

and see if the feed test passes?

[Analect]

Thanks Tim. I made those modifications, which appeared to fix 'some' of the rss feeds I have been testing. See the latest test report at the bottom with two tests still failing.

It's a great project and it's nice to be able to play around with a private version.

I've noticed a few things. There appears to be a limit of 10 articles when it ingests a feed, even if the feed contains 100s and I have set the limit on creation to the max 600. Is this an intended limitation?

Also, it only appears to process a portion of the ingested articles for entity recognition ... some don't have any 'entities' eventhough they contain relatively obvious ones that are contained in DBpedia.

Certain feeds still don't seem to process any articles and in some cases, they cause the docker container to crash. A case in point is this BBC feed: http://feeds.bbci.co.uk/news/uk/rss.xml On saving that as a new corpus, it crashes the app and the docker container crashes too. Here are the few pertinent log entries.

{"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"Starting progress socket at: ws://localhost:9001/corpus/54194f70f282100100594b20","time":"2014-09-17T09:08:00.361Z","v":0} {"name":"context","hostname":"02fd477c229b","pid":1,"level":30,"msg":"starting to process input from corpus 54194f70f282100100594b20","time":"2014-09-17T09:08:00.361Z","v":0}

events.js:72 throw er; // Unhandled 'error' event ^ Error: listen EADDRINUSE at errnoException (net.js:904:11) at Server._listen2 (net.js:1042:14) at listen (net.js:1064:10) at net.js:1146:9 at dns.js:72:18 at process._tickCallback (node.js:419:13)

This is the test results, referred to at top:

9 passing (55s) 2 failing

1) Slidewiki input processing suit should get 50 results: Error: timeout of 20000ms exceeded at null. (/opt/app/node_modules/gulp-mocha/node_modules/mocha/lib/runnable.js:157:19) at Timer.listOnTimeout as ontimeout

2) Wordpress input processing suit should get 50 results: Error: timeout of 20000ms exceeded at null. (/opt/app/node_modules/gulp-mocha/node_modules/mocha/lib/runnable.js:157:19) at Timer.listOnTimeout as ontimeout stream.js:94 throw er; // Unhandled stream error in pipe. ^ Error: 2 tests failed.

[yamalight]

There appears to be a limit of 10 articles when it ingests a feed, even if the feed contains 100s and I have set the limit on creation to the max 600. Is this an intended limitation?

That's because RSS usually gives only last X (usually 10-20) entries, and there's no way to get previous ones.

The crashes seem to be once again related to the websockets issue. Give me a few days, I'll add a config option to disable sockets at all. You can then use it without sockets (which will disable progress reporting, but that's a minor thing). I'll write here once it's done

[Analect]

Thanks. Once running, is it capable of processing new feeds to come in via RSS, so that the corpus grows over time ... or is it just possible to get that snapshot of 10 articles that happen to be available when you first set up the corpus?

[yamalight]

At the moment it just runs at the moment of execution, if you want to setup long running tasks, I'd say it'd be best to building a new tool on top of conTEXT base.

[Analect]

Understood. Thanks.

[yamalight]

Okay, just pushed the new config flag that allows disabling websockets progress reporting, please try using it. Also, please note that our instance of dbpedia spotlight seems to be having some issues right now.

[Analect]

Tim. I applied those changes in your latest commit and it is now working for the feed 'http://feeds.bbci.co.uk/news/uk/rss.xml', which had previously been crashing the container. Of the 50 items it ingests, it only seems to run entity-recognition on the first 5 or 6 of them. Perhaps this is related to you mentioning above that your instance of dbpedia spotlight is not working properly. I also tried with 'FOX', with more or less the same result (only a fraction of items getting processed for entity-recognition).

[yamalight]

alright, so the crashes are fixed, i'll close the issue then.

I'll have a look into the issues with fox and spotlight, but our public versions are always not very stable because of the random spikes of requests. if you want to use it for something serious, I'd recommend installing your own instance of spotlight and/or fox.

I'll comment here once I'd figured out the problem with our instances.