Problem with res.send of a MongoDB-ObjectID

[mm765]

Hi, i am having a problem with res.send() in decaf/modules/http/lib/Response.js

So far, everything worked, while ive been requesting files and some very simple json-data (like a login request returning if the user/password combination is valid).

Now i tried requesting/returning some data from the database (simple stuff - id (Objectid), username and language-code) and i get an internal server error. After about 2-3 hours of searching, i found out that the problem is the ObjectId - as soon as it is in the data that is being sent, the error happens, if i dont send it, no error.

the stacktrace: Stack Trace InternalError: Java class "[Ljava.lang.reflect.Constructor;" has no public instance field or method named "toJSON". (/usr/local/decaf/modules/http/lib/Response.js#185) at /usr/local/decaf/modules/http/lib/Response.js:178 (anonymous) at /opt/itk/modules/UserHandler.js:38 (anonymous) at /opt/itk/modules/UserHandler.js:12 (anonymous) at /opt/itk/libs/decaf-jolt/lib/Application.js:233 (anonymous) at /opt/itk/libs/decaf-jolt/lib/Application.js:170 (anonymous) at /usr/local/decaf/modules/http/lib/Child.js:69 (handleRequest) at /usr/local/decaf/modules/http/lib/Child.js:102 (Child) at /usr/local/decaf/modules/Threads/lib/Thread.js:197 (anonymous)

interestingly it shows the error happening in line 178 whereas, when i use the debugger, the error happens in line 182 of Response.js (which makes sense because i'm sending json-data, not text). . Please advise/help.

[mschwartz]

The ObjectId is a Java object. Convert it to hex with id.toHex().

On Tuesday, February 2, 2016, mm765 notifications@github.com wrote:

Hi, i am having a problem with res.send() in decaf/modules/http/lib/Response.js

So far, everything worked, while ive been requesting files and some very simple json-data (like a login request returning if the user/password combination is valid).

Now i tried requesting/returning some data from the database (simple stuff

• id (Objectid), username and language-code) and i get an internal server error. After about 2-3 hours of searching, i found out that the problem is the ObjectId - as soon as it is in the data that is being sent, the error happens, if i dont send it, no error.

the stacktrace: Stack Trace InternalError: Java class "[Ljava.lang.reflect.Constructor;" has no public instance field or method named "toJSON". (/usr/local/decaf/modules/http/lib/Response.js#185) at /usr/local/decaf/modules/http/lib/Response.js:178 (anonymous) at /opt/itk/modules/UserHandler.js:38 (anonymous) at /opt/itk/modules/UserHandler.js:12 (anonymous) at /opt/itk/libs/decaf-jolt/lib/Application.js:233 (anonymous) at /opt/itk/libs/decaf-jolt/lib/Application.js:170 (anonymous) at /usr/local/decaf/modules/http/lib/Child.js:69 (handleRequest) at /usr/local/decaf/modules/http/lib/Child.js:102 (Child) at /usr/local/decaf/modules/Threads/lib/Thread.js:197 (anonymous)

interestingly it shows the error happening in line 178 whereas, when i use the debugger, the error happens in line 182 of Response.js (which makes sense because i'm sending json-data, not text). . Please advise/help.

— Reply to this email directly or view it on GitHub https://github.com/decafjs/decaf/issues/16.

[mm765]

i tried It.toHexString() and i get the same error as above :( will try toString() and see what happens

[mm765]

Same thing with toString()

[mschwartz]

http://api.mongodb.org/java/current/org/bson/types/ObjectId.html

https://github.com/decafjs/decaf-mongodb/blob/master/lib/ObjectId.js#L20 Java ObjectId is returned.

On Tuesday, February 2, 2016, mm765 notifications@github.com wrote:

Same thing with toString()

— Reply to this email directly or view it on GitHub https://github.com/decafjs/decaf/issues/16#issuecomment-178959726.

[mm765]

Hmm.. maybe it returns a java string and that has to be converted into a javascript string ? One more thin i noticed when i tried the console.dir() is that i have an array in the data, consisting of objectids - and those get converted to strings - isnt that weird ?

[mm765]

This is the output of the dir() and the roles array consists of objectids in the database

(object) text Sessionuser: id [date] [getClass] [wait] [getTime] [notifyAll] [compareTo] [notify] [timeSecond] [hashCode] [getTimestamp] [getDate] [toStringMongod] [class] [timestamp] [getCounter] [toHexString] [machineIdentifier] [counter] [getMachineIdentifier] [equals] [toByteArray] [getTimeSecond] [toString] [getProcessIdentifier] [time] [processIdentifier] name Admin roles 0 5697d386b0210f3ea1417137 lang en

[mschwartz]

Heh... console.dir() is really smart about Java objects. console.log() just loops through the arguments calling System.out.print() on each - suitable for strings.

var s = String(javaString)

On Tuesday, February 2, 2016, mm765 notifications@github.com wrote:

Hmm.. maybe it returns a java string and that has to be converted into a javascript string ? One more thin i noticed when i tried the console.dir() is that i have an array in the data, consisting of objectids - and those get converted to strings - isnt that weird ?

— Reply to this email directly or view it on GitHub https://github.com/decafjs/decaf/issues/16#issuecomment-178961201.

[mschwartz]

You will need to convert Date() to string or milliseconds to send as json.

On Tuesday, February 2, 2016, mm765 notifications@github.com wrote:

This is the output of the dir() and the roles array consists of objectids in the database

(object) text http://string Sessionuser: id http://JavaObject [date] [getClass] [wait] [getTime] [notifyAll] [compareTo] [notify] [timeSecond] [hashCode] [getTimestamp] [getDate] [toStringMongod] [class] [timestamp] [getCounter] [toHexString] [machineIdentifier] [counter] [getMachineIdentifier] [equals] [toByteArray] [getTimeSecond] [toString] [getProcessIdentifier] [time] [processIdentifier] name http://string Admin roles http://array 0 http://string 5697d386b0210f3ea1417137 lang http://string en

— Reply to this email directly or view it on GitHub https://github.com/decafjs/decaf/issues/16#issuecomment-178961604.

[mm765]

hmm.. that paste omitted the most important stuff.. for the id it lists all the methods... for the roles: roles 0 5697d386b0210f3ea1417137 So it clearly sees that one as a string whereas the id is an objectId - in the database both are the same..

[mm765]

let me switch back to the older mongo-driver - maybe its a problem with the new driver i'm using.

[mm765]

well, that wasn't it either..

[mschwartz]

There are some things you cannot serialize as json. Date is one, function is another. You will have to process the record from mongo into a serializable object to send it.

Or use only serializable types in your documents. Instead of date, use milliseconds. Instead of ObjectId, use the string returned by new ObjectId().toHexString().

On Tuesday, February 2, 2016, mm765 notifications@github.com wrote:

let me switch back to the older mongo-driver - maybe its a problem with the new driver i'm using.

— Reply to this email directly or view it on GitHub https://github.com/decafjs/decaf/issues/16#issuecomment-178962546.

[mm765]

I just tried the toHexString again - and it returns a javaobject (console.dir shows it as such) - but as a string-object, not an objectid (different functions) - how do i convert that into a javascript-string ?

[mschwartz]

This is a snippet from my working project:

String(document._id.toHexString())

On Tuesday, February 2, 2016, mm765 notifications@github.com wrote:

I just tried the toHexString again - and it returns a javaobject (console.dir shows it as such) - but as a string-object, not an objectid (different functions) - how do i convert that into a javascript-string ?

— Reply to this email directly or view it on GitHub https://github.com/decafjs/decaf/issues/16#issuecomment-178964152.

[mschwartz]

http://www.newtonsoft.com/json/help/html/DatesInJSON.htm

On Tuesday, February 2, 2016, Mike Schwartz mykesx@gmail.com wrote:

This is a snippet from my working project:

String(document._id.toHexString())

On Tuesday, February 2, 2016, mm765 <notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

I just tried the toHexString again - and it returns a javaobject (console.dir shows it as such) - but as a string-object, not an objectid (different functions) - how do i convert that into a javascript-string ?

— Reply to this email directly or view it on GitHub https://github.com/decafjs/decaf/issues/16#issuecomment-178964152.

[mm765]

Yep, that did the trick - no more internal server error - thanks!

[mschwartz]

This is a web service, called via ajax:

var MongoDB  = require('decaf-mongodb').MongoDB,
    ObjectId = require('decaf-mongodb').ObjectId,
    db       = new MongoDB('mbrea');

db.use('events');

var methods = {
    save : function () {
        if (!req.user || req.role !== 'admin') {
            res.send({ success : false, message : 'Permission denied.' });
            return;
        }

        var userId = req.user._id,
            now    = new Date(),
            document;

        debugger;
        try {
            document = db.events.findOne({ _id : ObjectId(req.data.slug) });
        }
        catch (e) {
            try {
                document = db.events.findOne({ seo: req.data.seo });
            }
            catch (e) {
                res.send({ success : false, message : e.message + '\n' + e.stack });
                return;
            }
        }
        if (!document) {
            document = {
                _id     : ObjectId(),
                creator : userId,
                created : now
            }
        }

        var startDate = new Date(Date.parse(req.data.date + ' ' + req.data.startTime)),
            endDate   = new Date(Date.parse(req.data.date + ' ' + req.data.endTime)),
            year;

        year = startDate.getFullYear();
        if (year < 2000) {
            year += 100;
            startDate.setFullYear(year);
        }
        year = endDate.getFullYear();
        if (year < 2000) {
            year += 100;
            endDate.setFullYear(year);
        }
        decaf.extend(document, {
            title        : req.data.title,
            seo          : req.data.seo,
            startDate    : startDate,
            endDate      : endDate,
            location     : req.data.location,
            speakers     : req.data.speakers,
            body         : req.data.body,
            paymentTypes : req.data.paymentTypes,
            editor       : userId,
            edited       : now
        });

        if (document.seo) {
            try {
                var existing = db.events.findOne({ seo : document.seo });
                if (existing && String(existing._id) !== String(document._id)) {
                    res.send({ success : false, message : 'SEO URL already exists; it must be unique' });
                }
            }
            catch (e) {
                res.send({ success : false, message : e.message + '\n' + e.stack });
            }
        }
        try {
            db.events.save(document);
            res.send({ success : true, slug : String(document._id.toHexString()), seo : document.seo });
        }
        catch (e) {
            console.dir(e);
            res.send({ success : false, message : e.message + '\n' + e.stack });
        }
    },

    remove : function () {
        if (!req.user || req.role !== 'admin') {
            res.send({ success : false, message : 'Permission denied.' });
            return;
        }

        try {
            db.events.remove({ _id : ObjectId(req.data.slug) });
            res.send({ success : true });
        }
        catch (e) {
            res.send({ success : false, message : e.message });
        }
    }
};

var method = methods[ req.data.method ];
if (method) {
    return method();
}
else {
    throw new Error('Invalid method');
}

[mschwartz]

This is a controller, which generates HTML pages:

var Page     = require('Page'),
    page     = new Page(req, res),
    MongoDB  = require('decaf-mongodb').MongoDB,
    ObjectId = require('decaf-mongodb').ObjectId,
    db       = new MongoDB('mbrea'),
    moment   = require('moment');

db.use('users', 'events');

function timeFromNow(s) {
    var d    = new moment(s),
        diff = new moment().diff(d, 'days');

    if (diff > 7 || diff < -7) {
        return d.format('MMM Do h:mm A');
    }
    else {
        return d.calendar();
    }
}

function view(slug) {
    debugger;
    var item;
    try {
        item = db.events.findOne({ _id : ObjectId(slug) });
    }
    catch (e) {
        try {
            item = db.events.findOne({ seo : slug });
        }
        catch (e) {
            console.log(e.message + '\n' + e.stack);
        }
    }

    if (!item || !slug) {
        return 404;
    }
    var creator = db.users.findOne({ _id : item.creator }),
        editor  = db.users.findOne({ _id : item.editor });

    if (page.isAdmin) {
        page.addScript('/js/Events/detail.js');
    }

    var now          = new Date(),
        paymentTypes = (item.startDate > now) ? (item.paymentTypes.filter(function (o) {
            return o.memberRequired ? !!req.user : true;
        }) || []) : [],
        document     = {
            slug         : slug,
            title        : item.title,
            startDate    : timeFromNow(item.startDate.getTime()),
            endDate      : timeFromNow(item.endDate.getTime()),
            location     : item.location,
            speakers     : item.speakers,
            body         : item.body,
            paymentTypes : paymentTypes,
            created      : timeFromNow(item.created.getTime()),
            creator      : creator.firstName + ' ' + creator.lastName
        };

    if (+item.created !== +item.edited) {
        document.editor = editor.firstName + ' ' + editor.lastName;
        document.edited = timeFromNow(item.edited)
    }

    if (paymentTypes.length) {
        page.addScript('https://js.stripe.com/v2/');
        page.addScript('/js/payment-dialog.js');
        page.addScript('/js/Events/payment.js');
        var ndx = 0;
        decaf.each(paymentTypes, function (type) {
            type.ndx = ndx++;
        });
    }

    page.render('Events/Detail', {
        title               : item.title,
        breadcrumb          : [
            { href : '/events', text : 'Events' },
            { href : null, text : item.title }
        ],
        document            : document,
        apiKey              : Config.stripe.public_key,
        paymentTitle        : item.title,
        paymentOptions      : !!document.paymentTypes.length,
        paymentTypesEncoded : JSON.stringify(paymentTypes)
    });
}

function edit(slug) {
    if (!page.isAdmin) {
        res.redirect('/events/view/' + slug);
    }
    var title    = slug ? 'Edit Event' : 'Create Event',
        document; //  = slug ? db.events.findOne({ _id : ObjectId(slug) }) : {};

    debugger;
    try {
        document = db.events.findOne({ _id : ObjectId(slug) }) || {
                _id: new ObjectId(),
                startDate: new Date(),
                endDate: new Date()
            };
    }
    catch (e) {
        try {
            document = db.events.findOne({ seo : slug });
        }
        catch (e) {
            console.log(e.message + '\n' + e.stack);
        }
    }

    if (!document) {
        return 404;
    }

    page.addStylesheet('/bower/fontawesome/css/font-awesome.min.css');
    page.addStylesheet('/bower/summernote/dist/summernote.css');
    page.addStylesheet('/bower/eonasdan-bootstrap-datetimepicker/build/css/bootstrap-datetimepicker.min.css');

    page.addScript('/bower/summernote/dist/summernote.min.js');
    page.addScript('/bower/moment/moment.js');
    page.addScript('/bower/eonasdan-bootstrap-datetimepicker/build/js/bootstrap-datetimepicker.min.js');

    page.addScript('/js/Events/edit.js');

    if (document._id) {
        document.slug = String(document._id);
    }

    page.render('Events/Editor', {
        title        : title,
        breadcrumb   : [
            { href : '/events', text : 'Events' },
            { href : null, text : slug ? 'Edit' : 'Create' }
        ],
        document     : document,
        documentJSON : JSON.stringify({
            slug         : String(document._id.toString()),
            seo          : document.seo,
            title        : document.title,
            startDate    : document.startDate.getTime(),
            endDate      : document.endDate.getTime(),
            location     : document.location,
            speakers     : document.speakers,
            body         : document.body,
            paymentTypes : document.paymentTypes
        })
    });
}

// this needs to be paginated!
function list() {
    var events = [];
    decaf.each(db.events.find().sort({ startDate : -1 }).toArray(), function (item) {
        var creator = db.users.findOne({ _id : new ObjectId(item.creator) });
        events.push({
            slug      : item.seo || item._id.toString(),
            title     : item.title,
            speakers  : item.speakers,
            posted    : new moment(item.created.getTime()).calendar(),
            startDate : timeFromNow(item.startDate.getTime()),
            endDate   : timeFromNow(item.endDate.getTime()),
            creator   : creator.firstName + ' ' + creator.lastName
        });
    });
    page.render('Events/List', {
        title      : 'Events',
        breadcrumb : [
            { href : null, text : 'Events' }
        ],
        events     : events
    });
}

var verb = req.args[ 0 ],
    slug = req.args[ 1 ];

switch (verb) {
    case 'create':
        return edit();
    case 'edit':
        return edit(slug);
    case 'view':
        if (!slug) {
            res.redirect('/');
        }
        else {
            return view(slug);
        }
        break;
    default:
        return list();
        break;
}

[mschwartz]

This is lib/Page.js, a Page class used by the above controller (all controllers, actually):

var File = require('File'),
    MongoDB = require('decaf-mongodb').MongoDB,
    db = new MongoDB('mbrea'),
    TemplateManager = require('decaf-hoganjs').TemplateManager,
    viewManager = new TemplateManager('views');

db.use('pages');

function leadZero( s ) {
    s = '' + s;
    if (s.length === 1) {
        s = '0' + s;
    }
    return s;
}
function Page( req, res ) {
    var MobileDetect = require('mobile-detect'),
        browser = new MobileDetect(req.headers['user-agent']),
        backgroundImages = [ 'mission-beach-photo.png', 'mission-beach-photo2.jpg', 'mission-beach-photo3.jpg'],
        ndx = Math.floor(Math.random() * backgroundImages.length);

    this.bg = backgroundImages[ndx];
    this.req = req;
    this.res = res;
    this._scripts = [
        '/bower/jquery/dist/jquery.min.js',
        '/bower/jquery-cookie/jquery.cookie.js',
        '/bower/blockui/jquery.blockUI.js',
        '/bower/bootstrap/dist/js/bootstrap.min.js',
        '/bower/bootbox/bootbox.js',
        '/md5.js',
        '/js/mbrea.js'
    ];
    this._css = [
        '/bower/bootstrap/dist/css/bootstrap.min.css',
        '/bower/bootstrap/dist/css/bootstrap-theme.min.css',
        '/css/mbrea.css'
    ];

    this.isAdmin = req.role === 'admin';
    this.isPhone = browser.phone();
    this.isAdmin = this.isAdmin && !this.isPhone;
}

decaf.extend(Page.prototype, {
    addStylesheet : function( path ) {
        var me = this,
            paths = Array.prototype.slice.call(arguments, 0);
        decaf.each(paths, function( path ) {
            me._css.push(path);
        });
    },
    addScript     : function( path ) {
        var me = this,
            paths = Array.prototype.slice.call(arguments, 0);
        decaf.each(paths, function( path ) {
            me._scripts.push(path);
        })
    },

    render : function( tpl, o ) {
        var uri = this.req.uri.substr(1);
        o.bg = this.bg;
        o.css = this._css.concat(o.css || []);
        o.scripts = this._scripts.concat(o.scripts || []);
        if (!o.today) {
            var now = new Date();
            o.today = now.getFullYear() + '-' + leadZero(now.getMonth() + 1) + '-' + leadZero(now.getDate());
        }
        o.navigation = {
            news     : '',
            events   : '',
            members  : '',
            speakers : '',
            caravan  : '',
            area     : '',
            about    : '',
            settings : '',
            signin   : ''
        };
        o.user = this.req.user;
        o.isAdmin = this.isAdmin;
        o.mobile = this.isPhone;
        if (!o.title) {
            o.title = ''; // 'Mission Bay Real Estate Association';
        }
        decaf.each(o.navigation, function( value, key ) {
            o.navigation[key] = (uri === key) ? ' class="active"' : '';
        });

        var pages = [];
        decaf.each(db.pages.find({ navigation : true }).toArray(), function( page ) {
            pages.push({
                name  : page.name,
                title : page.title
            });
        });
        o.pages = pages;
        this.res.send(viewManager[tpl].render(o, viewManager));
//        this.req.session.save();
    }
});

module.exports = Page;

[mschwartz]

A HoganJS template to render an Event Detail page.

{{! created by mschwartz at 6/25/14 }}
{{!
    Event Details Template
}}
{{> common/header }}
{{#paymentOptions}}
    <script>
        var stripeApiKey = '{{apiKey}}',
            paymentTypes = {{{paymentTypesEncoded}}},
            paymentTitle = '{{{paymentTitle}}}';
    </script>
    {{> common/payment-dialog }}
{{/paymentOptions}}
<div class="content-block">
    {{#document}}
        <div class="page-header">
            <h2 id="title">{{{title}}}</h2>
            {{#isAdmin}}
                <div class="pull-right">
                    <button id="edit-event" class="btn btn-xs btn-primary" data-slug="{{slug}}">Edit</button>
                    <button id="delete-event" class="btn btn-xs btn-danger" data-slug="{{slug}}">Delete</button>
                </div>
            {{/isAdmin}}
            <div><b>When:</b> {{startDate}} to {{endDate}}</b></div>
            {{#location}}
                <div><b>Where:</b> {{location}}</div>
            {{/location}}

            {{#speakers}}
                <div><b>Speakers: </b> {{speakers}}</div>
            {{/speakers}}
        </div>
        <div class="row">
            <div class="col-md-9" style="overflow-x: auto;">
                {{{body}}}
                <div>
                    Posted by {{creator}}, {{created}}
                </div>
                {{#edited}}
                    <small>Edited by {{editor}}, {{edited}}</small>
                {{/edited}}
            </div>
            <div class="col-md-3">
                <div class="panel panel-default">
                    <div class="panel-heading">
                        <h2 class="panel-title">Event Details</h2>
                    </div>
                    <div class="panel-body">
                        <div><b>Start: </b>{{startDate}}</div>
                        <div><b>End: </b>{{endDate}}</div>
                    </div>
                </div>
                {{#paymentOptions}}
                    <div id="payment-options" class="panel panel-default">
                        <div class="panel-heading">
                            <h3 class="panel-title">Payment Options</h3>
                        </div>
                        {{#user}}
                            <div class="panel-body">
                                Your payments will appear on your profile page. These payments are not shown to anyone but you and MBREA staff.
                            </div>
                        {{/user}}
                        {{^user}}
                            <div class="panel-body">
                                If you are a MBREA member and sign in, your payments will be shown to you on your profile page.
                                These payments are not shown to anyone but you and MBREA staff.
                            </div>
                        {{/user}}
                        <div class="list-group">
                            {{#paymentTypes}}
                                <div class="list-group-item">
                                    <h4>{{title}}
                                        <small>${{amount}}</small>
                                    </h4>
                                    <div style="padding-bottom: 15px;">
                                        <small>{{description}}</small>
                                    </div>
                                    <button class="btn btn-sm btn-block btn-primary btn-payment" data-ndx="{{ndx}}">Pay Now</button>
                                </div>
                            {{/paymentTypes}}
                        </div>
                    </div>
                {{/paymentOptions}}
            </div>
        </div>
    {{/document}}
</div>
{{> common/footer }}

[mschwartz]

Let me know if you have questions :)

[mschwartz]

Events list in action: http://mbrea.net/events

Event detail in action: http://mbrea.net/events/view/562cf7ace4b0f62620862bdc

[mm765]

It is interesting to see, how things get done so differently - but it's probably mostly because i'm rendering on the client - the structure as such is kind of similar. i have a page baseclass from which all other pages inherit, a json-layout file similar to your hogan.js template . What you call a controller is a pageview in my app and my controller handles events (user-actions). When i will be further than the login-page, maybe i can show it to you one day :)

[mschwartz]

Check out the page load times on that mbrea site. I get sub 1 second. 1/2 second for most, or less.

That's doing the db queries for authentication and documents, and rendering via hoganjs templates.

SEO friendly.

Anyhow, there are examples of how I translate mongo documents into ones that are usable.

Conversions of date to string or milliseconds, etc.

On Tuesday, February 2, 2016, mm765 notifications@github.com wrote:

It is interesting to see, how things get done so differently - but it's probably mostly because i'm rendering on the client - the structure as such is kind of similar. i have a page baseclass from which all other pages inherit, a json-layout file similar to your hogan.js template . What you call a controller is a pageview in my app and my controller handles events (user-actions). When i will be further than the login-page, maybe i can show it to you one day :)

— Reply to this email directly or view it on GitHub https://github.com/decafjs/decaf/issues/16#issuecomment-178974065.

[mm765]

I am getting around 1.4 - 2.3 seconds on that page (from germany) need to get some sleep now (4 am here) .

[mschwartz]

The server is in NY, I am in California.

On Tuesday, February 2, 2016, mm765 notifications@github.com wrote:

I am getting around 1.4 - 2.3 seconds on that page (from germany) need to get some sleep now (4 am here) .

— Reply to this email directly or view it on GitHub https://github.com/decafjs/decaf/issues/16#issuecomment-178976932.