SMBios table gets very large with NeDB

[MordyT]

Updated from 0.4.5-a to 0.4.5-k through web interface. Platform: Windows Server 2012R2 (2 xeon core, 4GB Ram, 75GB SSD) Node version: Originally v12.13.0-x64, upgraded to v13.3.0-x64 just in case.

Error in log: C:\Program Files\Open Source\MeshCentral\node_modules\nedb\lib\data store.js:77 if (err) { throw err; } ^ Error: Cannot create a string longer than 0x3fffffe7 characters at Object.slice (buffer.js:586:37) at Buffer.toString (buffer.js:783:14) at FSReqCallback.readFileAfterClose [as oncomplete] (internal/fs/read_file_context.js:58:23) { code: 'ERR_STRING_TOO_LONG' }

[MordyT]

I have 26 PCs in MC... (Windows and Linux) Using the built in DB (Not Mongo)

I tried rolling a full image backup from yesterday and last week to no avail.

I completely removed MC and fresh install, stopped service, restored backup files, same issue.

Googling around I find there is some sort of 256MB limit in nedb and might be the cause of my issue?

I exported the database to json, installed MongoDB, imported the database, updated the config file, and MC came back. Agents are reconnecting now and all appears to be working.

I guess all that I would want to find out is why my db is so large with 26 PCs in since MC docs state I should be good for 1000's of PCs.

I have a full backup of the system before converting to MongoDB as well. If Ylianst or another dev wants to look into this, great. Otherwise I guess we can close the issue.

[Ylianst]

This is certainly not normal. You can look at all of the .db files in "meshcentral-data" and report back on the size of each of them? My plan for this is to go to the office later today and create a "--dbstats" command that will show a bunch of stats of the records in the DB. Once run, we will know what is going on and I will be able to make you a cleanup command and add code to prevent this from happening again. The build-in NeDB should be able to handle a 100+ devices perfectly fine.

[MordyT]

See attached. the smbios is over a gig....

(P.S. this thread is what led me to find out about the memory limit: https://github.com/louischatriot/nedb/issues/389)

[Ylianst]

Oh my!!! Just rename the meshcentral-smbios.db and restart the server. It will create a new meshcentral-smbios.db automatically. Let me know if that works, I will take a look later today at fixing the problem. No way that .db file should ever get that big.

[MailYouLater]

Might be worth looking into LinvoDB, which seems to be a based on NeDB, but able to be used for larger databases. You'd still want to encourage people to switch to MongoDB if their database is going to be huge, but at least it wouldn't fail to launch because one of the databases is over 256 MB.

[MordyT]

I can confirm that removing meshcentral-smbios.db does fix the issue.

Perhaps MC could check the size of all DBs once a day and compact/prune/something when they hit a certain limit? Or like MailYouLater said, perhaps nedb might not be the best?

Page 16/17 in manual talks about migrating to mongodb - how do I migrate back to nedb?

I assume like so: node meshcentral --mongodb mongodb://127.0.0.1:27017/meshcentral --dbexport node meshcentral --dbimport

Remove the 2 settings from the config file.

[MailYouLater]

NeDB has a speed advantage over LinvoDB and MongoDB because it loads the whole database into RAM, however once the database gets above a certain size it loses that speed advantage because it has to scan the whole database every time it stores or reads something in the database, and when the database exceeds 256MB it runs into an issue caused by the fact that it loads the whole database as a single string, and V8 (and therefore Node) can't handle a single string larger than 256MB. LinvoDB doesn't keep the whole database in RAM, so it's slower than NeDB at smaller sizes, but keeping an index of the database allows it to be faster than NeDB when the database gets larger, and since it doesn't load the whole database as a single string, the 256 MB limit doesn't effect it. It's kind of somewhere in between NeDB and MongoDB. If it was possible to easily migrate your data from NeDB to LinvoDB or MongoDB when your database starts getting large, that would be great, but at the very least, there should be a warning when your database size is approaching the limits of what NeDB is capable of.

[Ylianst]

What i don't understand is that the SMBios table should contain around 2k to 8k per device at most and only keeps the latest version of the SMBios table for a given device. Even with a lot of devices, that table should be pretty small. I don't understand why it would ever get this big.

In the new version I published today, I added additional checking and improved how I store the SMBios data. Hopefully that will fix it. If it happens again, someone will have to send me their smbios.db for me to analyses.

[Ylianst]

If you have a small installation, there is no need to move to MongoDB. This is a bug that I probably fixed, no need to a work around.

@MordyT Your correct, to migrate back you just do the opposite like you wrote.

[MordyT]

@Ylianst would you like a copy of my 1.5+ GB smbios file to see what happened? Otherwise I will move back to nedb at some point soon and report if it happens again.

[Ylianst]

I am very curious to see what that file contains. Can you compress it can see if it drops in size a lot? If it does and you have a way to send it over, I would like to look at it. My contract info is here. Thanks.

[MordyT]

Zipped it's about 140MB. I'll send you a download link now.

[Ylianst]

Thanks! Working on getting the new MeshAgent & Server published and will take a look at that just after.