Mongodb error while fetching statements.

[thebuh]

I encountered 500 error while fetching statements in bulk without filters from a LRS. At this point i have around 70k statements in LRS and while dumping with limit of 1000 statements per page following "more" URL i'm stuck at offset of 22000 with 500 error from mongo "too much data for sort() with no index. add an index or specify a smaller limit". (first 21 pages are loading just fine)

Version 1.12.1

Steps to reproduce the bug On our server after 22 pages of 1000 statements with URL /data/xAPI/statements?limit=1000&offset=22000

Expected behaviour We should get 1000 statements.

Actual behaviour

{
"error": true
"success": false
"message": "localhost:27017: too much data for sort() with no index. add an index or specify a smaller limit"
"code": 500
"trace": "#0 [internal function]: MongoCursor->rewind() #1 /srv/vhosts/learninglocker/vendor/jenssegers/mongodb/src/Jenssegers/Mongodb/Query/Builder.php(260): iterator_to_array(Object(MongoCursor), false) #2 /srv/vhosts/learninglocker/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(1384): Jenssegers\Mongodb\Query\Builder->getFresh(Array) #3 /srv/vhosts/learninglocker/vendor/jenssegers/mongodb/src/Jenssegers/Mongodb/Query/Builder.php(124): Illuminate\Database\Query\Builder->get(Array) #4 /srv/vhosts/learninglocker/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Builder.php(411): Jenssegers\Mongodb\Query\Builder->get(Array) #5 /srv/vhosts/learninglocker/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Builder.php(151): Illuminate\Database\Eloquent\Builder->getModels(Array) #6 /srv/vhosts/learninglocker/app/locker/repository/Statement/EloquentIndexer.php(157): Illuminate\Database\Eloquent\Builder->get() #7 /srv/vhosts/learninglocker/app/locker/repository/Statement/EloquentRepository.php(44): Locker\Repository\Statement\EloquentIndexer->format(Object(Jenssegers\Mongodb\Eloquent\Builder), Object(Locker\Repository\Statement\IndexOptions)) #8 /srv/vhosts/learninglocker/app/controllers/xapi/StatementIndexController.php(46): Locker\Repository\Statement\EloquentRepository->index(Array) #9 /srv/vhosts/learninglocker/app/controllers/xapi/StatementController.php(81): Controllers\xAPI\StatementIndexController->index(Array) #10 /srv/vhosts/learninglocker/app/controllers/xapi/StatementController.php(52): Controllers\xAPI\StatementController->index() #11 /srv/vhosts/learninglocker/app/controllers/xapi/BaseController.php(43): Controllers\xAPI\StatementController->get() #12 [internal function]: Controllers\xAPI\BaseController->selectMethod() #13 /srv/vhosts/learninglocker/vendor/laravel/framework/src/Illuminate/Routing/Controller.php(231): call_user_func_array(Array, Array) #14 /srv/vhosts/learninglocker/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php(93): Illuminate\Routing\Controller->callAction('selectMethod', Array) #15 /srv/vhosts/learninglocker/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php(62): Illuminate\Routing\ControllerDispatcher->call(Object(Controllers\xAPI\StatementController), Object(Illuminate\Routing\Route), 'selectMethod') #16 /srv/vhosts/learninglocker/vendor/laravel/framework/src/Illuminate/Routing/Router.php(967): Illuminate\Routing\ControllerDispatcher->dispatch(Object(Illuminate\Routing\Route), Object(Illuminate\Http\Request), 'Controllers\xAP...', 'selectMethod') #17 [internal function]: Illuminate\Routing\Router->Illuminate\Routing\{closure}() #18 /srv/vhosts/learninglocker/vendor/laravel/framework/src/Illuminate/Routing/Route.php(109): call_user_func_array(Object(Closure), Array) #19 /srv/vhosts/learninglocker/vendor/laravel/framework/src/Illuminate/Routing/Router.php(1033): Illuminate\Routing\Route->run(Object(Illuminate\Http\Request)) #20 /srv/vhosts/learninglocker/vendor/laravel/framework/src/Illuminate/Routing/Router.php(1001): Illuminate\Routing\Router->dispatchToRoute(Object(Illuminate\Http\Request)) #21 /srv/vhosts/learninglocker/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(775): Illuminate\Routing\Router->dispatch(Object(Illuminate\Http\Request)) #22 /srv/vhosts/learninglocker/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(745): Illuminate\Foundation\Application->dispatch(Object(Illuminate\Http\Request)) #23 /srv/vhosts/learninglocker/vendor/itsgoingd/clockwork/Clockwork/Support/Laravel/ClockworkLegacyMiddleware.php(20): Illuminate\Foundation\Application->handle(Object(Illuminate\Http\Request), 1, true) #24 /srv/vhosts/learninglocker/vendor/laravel/framework/src/Illuminate/Session/Middleware.php(72): Clockwork\Support\Laravel\ClockworkLegacyMiddleware->handle(Object(Illuminate\Http\Request), 1, true) #25 /srv/vhosts/learninglocker/vendor/laravel/framework/src/Illuminate/Cookie/Queue.php(47): Illuminate\Session\Middleware->handle(Object(Illuminate\Http\Request), 1, true) #26 /srv/vhosts/learninglocker/vendor/laravel/framework/src/Illuminate/Cookie/Guard.php(51): Illuminate\Cookie\Queue->handle(Object(Illuminate\Http\Request), 1, true) #27 /srv/vhosts/learninglocker/vendor/stack/builder/src/Stack/StackedHttpKernel.php(23): Illuminate\Cookie\Guard->handle(Object(Illuminate\Http\Request), 1, true) #28 /srv/vhosts/learninglocker/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(641): Stack\StackedHttpKernel->handle(Object(Illuminate\Http\Request)) #29 /srv/vhosts/learninglocker/public/index.php(49): Illuminate\Foundation\Application->run() #30 {main}"
}

Server information OS: Ubuntu 14.04.4 mongo version: 2.4.9 php version: PHP 5.5.9

Mongo related settings in php:

mongo

MongoDB Support => enabled
Version => 1.4.5
SSL Support => enabled
Streams Support => enabled

Directive => Local Value => Master Value
mongo.allow_empty_keys => 0 => 0
mongo.chunk_size => 262144 => 262144
mongo.cmd => $ => $
mongo.default_host => localhost => localhost
mongo.default_port => 27017 => 27017
mongo.is_master_interval => 15 => 15
mongo.long_as_object => 0 => 0
mongo.native_long => 0 => 0
mongo.ping_interval => 5 => 5

Client information Any client.

Additional information I've tried to decrease page size. But even with page size of 100 error pops up at offset of around 23000.

[ryasmi]

Hi @thebuh, thanks for reporting this bug and using our template. We will now try to reproduce your issue.

[ryasmi]

@thebuh, can you please have a read through a blog post about indexes that we recently published and ensure that creating some of the indexes doesn't resolve your issue?

[thebuh]

Thank you, @ryansmith94 for quick response! Yep, it does indeed resolved the issue.

May i suggest adding something regarding indexes to http://docs.learninglocker.net/installation/ since blog posts are not very obvious place to look for that kind of info. And there no mention of this anywhere in installation docs.

[ryasmi]

Hey @thebuh, you're very welcome and I'm glad that resolved your issue.

Yes you may, I agree that it would be a good idea, so I'll do it now and close this issue when it's done.

[ht2]

Just FYI - the current version of develop has all the indexes mentioned in the blog post included. Following the installation steps regarding migrations would ensure these are loaded. (I appreciate however that not everyone works off the develop branch and as such the latest release does not have them).

Saying that, index management is simply part and parcel of running a database with an application - I would implore you that if you plan on running this at scale or in a production client facing environment to consult or hire expertise that can enable you to proficiently manage the application stack, as you will face many other challenges relating to the database and infrastructure that fall outside the purview of the application's documentation.