Open livejake opened 5 years ago
Looking through the this using --log4j2-file log4j2.xml, I see the following error preceding the Fatal error.
com.torodb.backend.exceptions.BackendException: On context ADD_COLUMN with sqlState 54011: ERROR: tables can have at most 1600 columns
What is the best approach here? I see you can filter databases and collections, but is there a way to exclude part of a collection that is generating tons of columns? Or should I try to copy the mongo database and remove part of the collection?
Hi @livejake, you actually hit a ToroDB Stampede limit, I will leave this bug open but we are out of resources to fix it any time soon.
Probably the best option to workaround the limitation would be to transform the document in the source database so that the structure is not plain. A trivial example would be something like:
db.main.jobs.find().forEach(function (i) {
var o={};
for (var key in i) {
if (key != "_id") {
if (!o[key.substring(0,1)]) {
o[key.substring(0,1)] = {};
};
o[key.substring(0,1)][key] = i[key];
} else {
o[key] = i[key];
}
};
db.main.mapped.jobs.insert(o);
});
The problem with this approach is that replication will not work as is since collection with mapped copy of original collection will become stale.
I'm seeing the following error after going through the QuickStart setup and using my own data.