search

status-im / infra-office-legacy

Infrastructure for cloud office services
0 stars 0 forks source link

Wekan boards very slow UI #15

Closed jakubgs closed 2 years ago

jakubgs commented 2 years ago

We've been seeing more frequent issues with Wekan being very slow, Specifically this can be seen in the XHR requests it's making:

image

The most affected board appears to be:

https://boards.status.im/b/79zLjA2QdoAWRTbSy/new-status-ambassadors

Related:

jakubgs commented 2 years ago

We can see that after every restart the CPU usage keeps growing:

image

And so does memory usage, but it's less of a problem since the host has 8 GB of RAM:

image

jakubgs commented 2 years ago

What's more weird is the endlessly growing traffic after every restart:

image

And a growing number of allocated TCP sockets:

image

Not sure what to make of this but seems like some kind of bug.

jakubgs commented 2 years ago

Possibly related issues:

I don't think there is much hope for this project.

jakubgs commented 2 years ago

While looking for some metrics I realized that the code is full of commented out lines, like here: https://github.com/wekan/wekan/blob/master/models/server/metrics.js

And also the metrics are useless and unrelated to performance:

admin@node-01.do-ams3.wekan.office:~ % c 0:8280/metrics       
# Number of connected users
connectedUsers 1
# Number of registered users
registeredUsers 409
# Number of registered boards
registeredboards 81
# Number of registered boards by registered users
registeredboardsBysRegisteredUsers 0.1980440097799511
# Number of registered boards
registeredboardsWithOnlyOneMember 48
# Number of users with last connection dated 5 days ago
usersWithLastConnectionDated5DaysAgo 16
# Number of users with last connection dated 10 days ago
usersWithLastConnectionDated10DaysAgo 18
# Number of users with last connection dated 20 days ago
usersWithLastConnectionDated20DaysAgo 18
# Number of users with last connection dated 30 days ago
usersWithLastConnectionDated30DaysAgo 18
jakubgs commented 2 years ago

It does make me wonder tho, why is this number so high:

registeredboardsWithOnlyOneMember 48
jakubgs commented 2 years ago

The number of boards in the DB appears to be quite high actually:

> db.boards.find().count()
512

But a lot of them appear to be some kind of templates:

> db.boards.find({title:'Templates'}).count()
408
jakubgs commented 2 years ago

Not sure what all of those templates are for:

> db.boards.find({title:'Templates'}, {title:1,slug:1,archived:1})
{ "_id" : "ba63igqjHgSLSDkLb", "title" : "Templates", "slug" : "templates", "archived" : false }
{ "_id" : "Bbxxp8issyNae4xFP", "title" : "Templates", "slug" : "templates", "archived" : false }
{ "_id" : "b8d5RCtbrmvu7GNKS", "title" : "Templates", "slug" : "templates", "archived" : false }
{ "_id" : "qrCmnSTaLHeGKFvQw", "title" : "Templates", "slug" : "templates", "archived" : false }
{ "_id" : "BBFC8GfM9PGN7A28Y", "title" : "Templates", "slug" : "templates", "archived" : false }
{ "_id" : "TCK4H7vT7ptDEvnsE", "title" : "Templates", "slug" : "templates", "archived" : false }
{ "_id" : "xJwnP6hxmfwCXL8N9", "title" : "Templates", "slug" : "templates", "archived" : false }
{ "_id" : "i3J8mkkL8T6vkdjNz", "title" : "Templates", "slug" : "templates", "archived" : false }
{ "_id" : "7YzdHMsmXXhxRTydE", "title" : "Templates", "slug" : "templates", "archived" : false }
{ "_id" : "vzZJgmPCRt8SsmxCt", "title" : "Templates", "slug" : "templates", "archived" : false }
{ "_id" : "jd55f66XBmHZxqY7k", "title" : "Templates", "slug" : "templates", "archived" : false }
{ "_id" : "g5uTQPT2qGL7ieE6r", "title" : "Templates", "slug" : "templates", "archived" : false }
{ "_id" : "p7zpqywZDuwk4QrBd", "title" : "Templates", "slug" : "templates", "archived" : false }
{ "_id" : "EcZmhWQPr8wcmEk7i", "title" : "Templates", "slug" : "templates", "archived" : false }
{ "_id" : "5WA93eSPEX99prmaM", "title" : "Templates", "slug" : "templates", "archived" : false }
{ "_id" : "oyee69Xfkor4AKRtw", "title" : "Templates", "slug" : "templates", "archived" : false }
{ "_id" : "gyyDY2bCiiyeiooGi", "title" : "Templates", "slug" : "templates", "archived" : false }
{ "_id" : "K3aiDmzfTZZ43TsDX", "title" : "Templates", "slug" : "templates", "archived" : false }
{ "_id" : "PpEWzcQbSnjeuKv6L", "title" : "Templates", "slug" : "templates", "archived" : false }
{ "_id" : "YT6tJzExjarpL4Prq", "title" : "Templates", "slug" : "templates", "archived" : false }
Type "it" for more

But I will remove them. We have backups anyway.

jakubgs commented 2 years ago

I deleted all of them:

> db.boards.deleteMany({title:'Templates'})
{ "acknowledged" : true, "deletedCount" : 408 }

And the site still appears to function.

jakubgs commented 2 years ago

There's a lot of boards that appear to be just tests:

> db.boards.find({archived:true}, {title:1,archived:1})
{ "_id" : "tKuQtmGk29gr7Woa2", "title" : "Social Content Calendar (Example)", "archived" : true }
{ "_id" : "uK2YykAbA6KepgLyp", "title" : "^8-14 OCT 2018 | SOCIAL CONTENT^", "archived" : true }
{ "_id" : "JNusMDKRfDewku6Cd", "title" : "Social Content Calendar | 8-14 OCT 2018", "archived" : true }
{ "_id" : "DMkwzfYopiQYLcGHg", "title" : "Social Content Calendar | 15-21 OCT 2018", "archived" : true }
{ "_id" : "d7kxpNxo6SryLQSC2", "title" : "Social Content Calendar | 22-28 OCT 2018", "archived" : true }
{ "_id" : "MC5BEizH3QBwDmbpG", "title" : "Staus Dev Channel Consent Calendar", "archived" : true }
{ "_id" : "BHvKQJvupwTXaF4Ek", "title" : "Ambassador Working Groups", "archived" : true }
{ "_id" : "dcQNxopqX9jSMa6hk", "title" : "^V1 Launch Comms Production List^", "archived" : true }
{ "_id" : "3Ge6zoCNTQyqKtXxM", "title" : "Bucket List", "archived" : true }
{ "_id" : "vybnv2twPpNHSRyC7", "title" : "Hello World", "archived" : true }
{ "_id" : "JG6ug585yHymzvSRY", "title" : "Asif", "archived" : true }
{ "_id" : "MbF3kDZK9EWugFPu7", "title" : "test", "archived" : true }
{ "_id" : "WdLBuy8zkdaHJhLzs", "title" : "sdvds", "archived" : true }
{ "_id" : "JcQLd9FSpnJJiNfhx", "title" : "Test", "archived" : true }
{ "_id" : "GJPJbFdSY7i3JZmvw", "title" : "Translate a new blog post into Bengali", "archived" : true }
{ "_id" : "2QTZic3S2Fp8ByNBP", "title" : "Bengali Translation", "archived" : true }
{ "_id" : "wf9cztPTduYcRcw5H", "title" : "vvv", "archived" : true }
{ "_id" : "mBicwT4TA2sAwXupu", "title" : "Bengali translation", "archived" : true }
{ "_id" : "bTMpdWjuZ4wEdoA9f", "title" : "^Status Marketing Kanban^", "archived" : true }

And some are literally garbage:

> db.boards.find({_id:'3DcgEDhmzyGms7Tz3'}, {title:1,archived:1})
{ "_id" : "3DcgEDhmzyGms7Tz3", "title" : "\"><img src=x>", "archived" : false }
> db.boards.find({_id:'wr8rPrATEABM3Xnsc'}, {title:1,archived:1})
{ "_id" : "wr8rPrATEABM3Xnsc", "title" : "<h1>hello</h1>", "archived" : false }
> db.boards.find({_id:'iYvrZ2JMjxqZHZevE'}, {title:1,archived:1})
{ "_id" : "iYvrZ2JMjxqZHZevE", "title" : "Status ETH 2.0 আমানত চুক্তি সমর্থন করে\n\nEth2 ফেজ 0 এর বহুল প্রত্যাশিত আসন্ন প্রবর্তনের সাথে, জেনেসিস প্রক্রিয়াটি শুরু করার জন্য জনসাধারণের পক্ষে একটি প্রচলিত আমানত চুক্তির ঠিকানার উপর সিদ্ধান্ত নেওয়া অন্যতম গুরুত্বপূর্ণ পদক্ষেপ। বাইটকোডের জমা দেওয়ার চুক্তির একটি উদাহরণ বর্তমানে বেনামে মেইনেটের 0x00000000219ab540356cbb839cbe05303d7705fa ঠিকানায় স্থাপন করা হয়েছে। ইথেরিয়াম সম্প্রদায়ের সদস্যরা এই চুক্তির ঠিকানাটি Eth2 পর্যায়ের এগিয়ে যাওয়ার জন্য মূল ঠিকানা হিসাবে সামাজিক কমত্যে পৌঁছানোর চেষ্টা করছেন। আমাদের সেরা জ্ঞানের জন্য, ইথেরিয়াম মেইনেটে বর্তমানে আমানত চুক্তির বাইটকোডের এটি প্রথম এবং একমাত্র উদাহরণ। \n\nএই https://nimbus.team/  নিবন্ধটি Eth2 এর জন্য ন্যাশনাল আমানত চুক্তি হতে উপরের ঠিকানায় মোতায়েন করা চুক্তির জন্য সমর্থন এবং সিম্বলস দলগুলোর স্থিতাবস্থা এবং একটি ঘোষণার বিষয়। আমরা স্বাধীনভাবে যাচাইয়ের পরে এই সিদ্ধান্তে পৌঁছেছি যে চুক্তির ঠিকানায় স্থাপন করা বাইটকোড হ'ল স্থানীয় বিল্ডে চুক্তির সলিডিটি সোর্স কোডটি সংকলন করে যা আমরা পুনরুউৎপাদন করতে পারি ঠিক তার মতোই।আমরা জেনেসিস প্রক্রিয়া এবং তার বাইরেও এই ETH 2 ডিপোজিট চুক্তির ঠিকানার ব্যবহার সমর্থন করার জন্য ইথেরিয়াম সম্প্রদায়ের অন্যান্য সদস্যদের সাথে যোগ দিচ্ছি তা জানাতে আমরা এই গুরুত্বপূর্ণ ঘোষণাটি প্রকাশ করছি।\n\nIndependent verification (স্বতন্ত্র যাচাইকরণ)\n\nউপরের ঠিকানায় চুক্তিটি টাইমস্ট্যাম্পে তৈরি হয়েছিল Oct-14-2020 09:22:52 AM +UTC যেমনটি https://etherscan.io/tx/0xe75fb554e433e03763a1560646ee22dcb74e5274b34c5ad644e7c0f619a7e1d0 হ্যাশ দিয়ে লেনদেন দ্বারা \n0xe75fb554e433e03763a1560646ee22dcb74e5274b34c5ad644e7c0f619a7e1d0\nচুক্তির উৎস কোডটি যাচাই করা হয়েছে https://etherscan.io/address/0x00000000219ab540356cbb839cbe05303d7705fa#code\nযেখানে এটি নোট করে যে চুক্তিটি সলিডিটি সংকলক সংস্করণ v0.6.11 + কমিট 5.5660b1 এবং 5000000 রান সহ অনুকূলিতকরণের সাহায্যে সংকলিত হয়েছিল।\nনিম্নলিখিত পদক্ষেপগুলি আমাদের যাচাই প্রক্রিয়াটির রূপরেখা দেয়:\n\n১. আমরা ইথেরিয়াম ব্লকচেইন ব্যবহার করে eth_getCode এই ঠিকানায় বাইটকোড পেয়েছি। (via Infura)\n\n2. আমরা eth2.0-specs থেকে https://github.com/ethereum/eth2.0-specs সংগ্রহস্থলটিকে ক্লোন করেছি এবং আমানতের চুক্তির জন্য সংকলনের নির্দেশাবলী অনুসরণ করেছি।\n৩. আমরা solc-static-linux.bin এখান থেকে লিনাক্সের জন্য সলিডিটি সংকলক সংস্করণ version v0.6.11+commit.5ef660b1 ডাউনলোড করেছি।\n\n৪. আমরা এই কমান্ডটি দিয়ে eth2.0-specs সংগ্রহস্থলে জমা _contract.sol সঙ্কলন করেছি: solc --optimize --optimize- রান 5000000 --metadata- literal --bin-runtime deposit_contract.সোল যা আমাদের এই চুক্তির জন্য রানটাইম বাইটকোড দিয়েছে। \n\n৫. আমরা ধাপ (1) https://notes.status.im/onchain_bytecode?view এ প্রাপ্ত বাইটকোডকে ধাপে (4) https://notes.status.im/compiled_bytecode?view এর সাথে তুলনা করেছি এবং সেগুলি দেখতে পেয়েছি একইরকম ..\n\nConclusion (উপসংহার) \n\nসমালোচনামূলক Eth2 আমানত চুক্তি (যা মূলত ভাইপরে লেখা হয়েছিল) সলিডিতে পুনরায় লিখিত হয়েছিল, সলিডিটি বিশেষজ্ঞরা https://github.com/runtimeverification/deposit-contract-verification/blob/master/deposit-contract-verification.pdf পর্যালোচনা করেছিলেন এবং তারপরে ইথেরিয়াম ফাউন্ডেশনের এই https://blog.ethereum.org/2020/06/23/eth2-quick-update-no-12/  ব্লগ পোস্টে ব্যাখ্যা করা হিসাবে রানটাইম যাচাইকরণ দ্বারা আনুষ্ঠানিকভাবে যাচাই করা হয়েছিল। এটি এখন ঠিকানার মেইনেটে স্থাপন করা হয়েছে\n0x00000000219ab540356cbb839cbe05303d7705fa\n\nStatus এবং nimbus দল Eth2 এর জন্য ন্যায্যক আমানতের চুক্তি হতে এই ঠিকানায় মোতায়েন করা চুক্তিকে সমর্থন করে .", "archived" : false }

We should probably limit who can create boards... if that's even possible in this garbage software.

jakubgs commented 2 years ago

So much garbage:

> db.boards.find({slug:{$regex:'less'}}, {title:1,members:1})
{ "_id" : "RNw7oJHhDSjSj4GvD", "title" : "<marquee>hello world</marquee>", "members" : [ { "userId" : "JsZPT8TRkkr77kiMk", "isAdmin" : true, "isActive" : true, "isNoComments" : false, "isCommentOnly" : false, "isWorker" : false } ] }
{ "_id" : "oNATBtAQ4tGEXSXL4", "title" : "<input>", "members" : [ { "userId" : "tQThYy6ixkkikCoj6", "isAdmin" : true, "isActive" : true, "isNoComments" : false, "isCommentOnly" : false, "isWorker" : false } ] }
{ "_id" : "wqLAjjnMhmYMaHzfk", "members" : [ { "userId" : "tQThYy6ixkkikCoj6", "isActive" : true, "isAdmin" : true, "isNoComments" : false, "isCommentOnly" : false } ], "title" : "<iframe src=\"https://google.com\">" }
{ "_id" : "mD5tNneARHxwZReAq", "members" : [ { "userId" : "tQThYy6ixkkikCoj6", "isActive" : true, "isAdmin" : true, "isNoComments" : false, "isCommentOnly" : false } ], "title" : "<iframe src=\"https://google.com\">" }
{ "_id" : "XrBiWPQEmk9RsYLAT", "members" : [ { "userId" : "tQThYy6ixkkikCoj6", "isActive" : true, "isAdmin" : true, "isNoComments" : false, "isCommentOnly" : false } ], "title" : "<iframe src=\"https://google.com\">" }
{ "_id" : "3DcgEDhmzyGms7Tz3", "title" : "\"><img src=x>", "members" : [ { "userId" : "tQThYy6ixkkikCoj6", "isAdmin" : true, "isActive" : true, "isNoComments" : false, "isCommentOnly" : false, "isWorker" : false } ] }

> db.users.find({_id:'tQThYy6ixkkikCoj6'}, {username:1,emails:1})
{ "_id" : "tQThYy6ixkkikCoj6", "username" : "cjlegacion24", "emails" : [ { "address" : "legacion_c@yahoo.com", "verified" : false } ] }
jakubgs commented 2 years ago

Users probably need pruning too:

> db.users.find().count()
409

I just found out that we had open registrations enabled:

image

Though I'm pretty sure I disabled it in the past...

jakubgs commented 2 years ago

There were all created by Jinho:

> db.boards.find({members:{$elemMatch:{isAdmin:true,userId:'Sjq8EQ3fNpjHwqa32'}}},{title:1,slug:1})
{ "_id" : "fchXE6x3kCqbs7RHL", "title" : "Status Ambassador Tasks", "slug" : "status-ambassador-tasks" }
{ "_id" : "79zLjA2QdoAWRTbSy", "title" : "[New] Status Ambassadors", "slug" : "new-status-ambassadors" }
{ "_id" : "u5FWB7hq39F4SERT5", "title" : "^[New] Status Ambassadors^", "slug" : "new-status-ambassadors" }
{ "_id" : "4g26Y42ehz8ahjQYs", "title" : "^[New] Status Ambassadors^", "slug" : "new-status-ambassadors" }
{ "_id" : "zCuPGXmycHkQMeAvT", "title" : "Status Marketing Kanban", "slug" : "status-marketing-kanban" }
{ "_id" : "A3i7MNm6JbQf8LMNb", "title" : "^[New] Status Ambassadors^", "slug" : "new-status-ambassadors" }
{ "_id" : "NPYh3hebS8H48meho", "title" : "^[New] Status Ambassadors^", "slug" : "new-status-ambassadors" }
{ "_id" : "jm9sQzZspaATp8fWB", "title" : "^[New] Status Ambassadors^", "slug" : "new-status-ambassadors" }
{ "_id" : "Scfxz8yssEgEjjNHr", "title" : "^[New] Status Ambassadors^", "slug" : "new-status-ambassadors" }
{ "_id" : "zfEmZCpd9ksLP3ATA", "title" : "^[New] Status Ambassadors^", "slug" : "new-status-ambassadors" }
{ "_id" : "jNXZ4m9jBzL7wtcWv", "title" : "^[New] Status Ambassadors^", "slug" : "new-status-ambassadors" }
{ "_id" : "aGnnhMTQuALgrMoSw", "title" : "^[New] Status Ambassadors^", "slug" : "new-status-ambassadors" }
{ "_id" : "Jdt2HPpkMtRyWz7W4", "title" : "^[New] Status Ambassadors^", "slug" : "new-status-ambassadors" }

But he has no recollection of doing this, just archiving them. Might be a side effect of some bug.

jakubgs commented 2 years ago

There's more of those starting and ending with ^:

> db.boards.find({title:{$regex:'\\^'}},{title:1,slug:1})
{ "_id" : "uK2YykAbA6KepgLyp", "title" : "^8-14 OCT 2018 | SOCIAL CONTENT^", "slug" : "8-14-oct-2018-or-social-content" }
{ "_id" : "dcQNxopqX9jSMa6hk", "title" : "^V1 Launch Comms Production List^", "slug" : "v1-launch-comms-production-list" }
{ "_id" : "FJhaBf9ZbKfGDmNiG", "title" : "^Status Brasil - Canais de comunicação e interação^", "slug" : "status-brasil-canais-de-comunicacao-e-interacao" }
{ "_id" : "u5FWB7hq39F4SERT5", "title" : "^[New] Status Ambassadors^", "slug" : "new-status-ambassadors" }
{ "_id" : "4g26Y42ehz8ahjQYs", "title" : "^[New] Status Ambassadors^", "slug" : "new-status-ambassadors" }
{ "_id" : "bTMpdWjuZ4wEdoA9f", "title" : "^Status Marketing Kanban^", "slug" : "status-marketing-kanban" }
{ "_id" : "zGs3mvKqeQyAhWxWp", "title" : "^Status Marketing Kanban^", "slug" : "status-marketing-kanban" }
{ "_id" : "A3i7MNm6JbQf8LMNb", "title" : "^[New] Status Ambassadors^", "slug" : "new-status-ambassadors" }
{ "_id" : "NPYh3hebS8H48meho", "title" : "^[New] Status Ambassadors^", "slug" : "new-status-ambassadors" }
{ "_id" : "jm9sQzZspaATp8fWB", "title" : "^[New] Status Ambassadors^", "slug" : "new-status-ambassadors" }
{ "_id" : "Scfxz8yssEgEjjNHr", "title" : "^[New] Status Ambassadors^", "slug" : "new-status-ambassadors" }
{ "_id" : "zfEmZCpd9ksLP3ATA", "title" : "^[New] Status Ambassadors^", "slug" : "new-status-ambassadors" }
{ "_id" : "jNXZ4m9jBzL7wtcWv", "title" : "^[New] Status Ambassadors^", "slug" : "new-status-ambassadors" }
{ "_id" : "aGnnhMTQuALgrMoSw", "title" : "^[New] Status Ambassadors^", "slug" : "new-status-ambassadors" }
{ "_id" : "Jdt2HPpkMtRyWz7W4", "title" : "^[New] Status Ambassadors^", "slug" : "new-status-ambassadors" }

Deleted:

> db.boards.deleteMany({title:{$regex:'\\^'}})
{ "acknowledged" : true, "deletedCount" : 15 }
jakubgs commented 2 years ago

That's a lot of garbage:

> db.cards.find().count()
3498
jakubgs commented 2 years ago

Apparently you can't restrict things like boards creation yet:

This software is killing me.

jakubgs commented 2 years ago

More garbage:

> db.boards.find({_id:{$in: badIds}}, {title:1,slug:1})
{ "_id" : "6rAQafgkTN5DmJYv3", "title" : "creating a local meetup to introduce some new member on board.", "slug" : "creating-a-local-meetup-to-introduce-some-new-member-on-board" }
{ "_id" : "7hGNLZvJptme7KTfE", "title" : "test", "slug" : "test" }
{ "_id" : "93AZ6LtpxqEbuaZiP", "title" : "argserg", "slug" : "argserg" }
{ "_id" : "JG6ug585yHymzvSRY", "title" : "Asif", "slug" : "asif" }
{ "_id" : "JcQLd9FSpnJJiNfhx", "title" : "Test", "slug" : "test" }
{ "_id" : "MbF3kDZK9EWugFPu7", "title" : "test", "slug" : "test" }
{ "_id" : "WA7GwpYAqtBboMZ7D", "title" : "ff", "slug" : "ff" }
{ "_id" : "WdLBuy8zkdaHJhLzs", "title" : "sdvds", "slug" : "sdvds" }
{ "_id" : "sAZyaHoqMFDcarFeK", "title" : "asda", "slug" : "tanginamo" }
{ "_id" : "vybnv2twPpNHSRyC7", "title" : "Hello World", "slug" : "hello-world" }
{ "_id" : "wf9cztPTduYcRcw5H", "title" : "vvv", "slug" : "vvv" }
jakubgs commented 2 years ago

Some more cleanup of archives boards:

> db.boards.find({archived:true}, {title:1,slug:1,archived:1})
{ "_id" : "tKuQtmGk29gr7Woa2", "title" : "Social Content Calendar (Example)", "slug" : "weekly-content-calendars", "archived" : true }
{ "_id" : "JNusMDKRfDewku6Cd", "title" : "Social Content Calendar | 8-14 OCT 2018", "slug" : "social-content-calendar-or-8-14-oct-2018", "archived" : true }
{ "_id" : "DMkwzfYopiQYLcGHg", "title" : "Social Content Calendar | 15-21 OCT 2018", "slug" : "social-content-calendar-or-15-21-oct-2018", "archived" : true }
{ "_id" : "d7kxpNxo6SryLQSC2", "title" : "Social Content Calendar | 22-28 OCT 2018", "slug" : "social-content-calendar-or-22-28-oct-2018", "archived" : true }
{ "_id" : "MC5BEizH3QBwDmbpG", "title" : "Staus Dev Channel Consent Calendar", "slug" : "staus-dev-channel-consent-calendar", "archived" : true }
{ "_id" : "BHvKQJvupwTXaF4Ek", "title" : "Ambassador Working Groups", "slug" : "ambassador-working-groups", "archived" : true }
{ "_id" : "3Ge6zoCNTQyqKtXxM", "title" : "Bucket List", "slug" : "bucket-list", "archived" : true }
{ "_id" : "GJPJbFdSY7i3JZmvw", "title" : "Translate a new blog post into Bengali", "slug" : "translate-a-new-blog-post-into-bengali", "archived" : true }
{ "_id" : "2QTZic3S2Fp8ByNBP", "title" : "Bengali Translation", "slug" : "bengali-translation", "archived" : true }
{ "_id" : "mBicwT4TA2sAwXupu", "title" : "Bengali translation", "slug" : "bengali-translation", "archived" : true }
{ "_id" : "KdyjhW8xP34WtPbPi", "title" : "Public Board", "slug" : "public-board", "archived" : true }

> db.boards.deleteMany({archived:true})
{ "acknowledged" : true, "deletedCount" : 11 }
jakubgs commented 2 years ago

Here's a simple query to remove cards without a board:

> existingBoards = db.boards.find({}, {_id:1}).map(x => x._id)

> db.cards.find({boardId:{$not:{$in:existingBoards}}}).count()
436

> db.cards.count()
3499
jakubgs commented 2 years ago

With $size I can find boards with just one member:

> db.boards.find({members:{$size:1}}, {title:1,slug:1,members:1}).limit(5)
{ "_id" : "q9QfDbXcXmJayvjje", "title" : "Status Social Content Calendar", "slug" : "status-social-content-calendar", "members" : [ { "userId" : "RTu7CRAnEbt4xpyqB", "isAdmin" : true, "isActive" : true, "isNoComments" : false, "isCommentOnly" : false } ] }
{ "_id" : "bEXmsTFuZX34W5Sed", "title" : "Layer0", "slug" : "layer0", "members" : [ { "userId" : "kYqudaTHKeFKBPJbM", "isAdmin" : true, "isActive" : true, "isNoComments" : false, "isCommentOnly" : false } ] }
{ "_id" : "JyRRNb42t2haBwAj8", "title" : "Keycard", "slug" : "keycard", "members" : [ { "userId" : "gkL647fumyyiRDgzc", "isAdmin" : true, "isActive" : true, "isNoComments" : false, "isCommentOnly" : false } ] }
{ "_id" : "K9Kgwisrc2cT8TvHa", "title" : "Payment Network", "slug" : "payment-network", "members" : [ { "userId" : "dqYnp9eDP2JteqKu6", "isAdmin" : true, "isActive" : true, "isNoComments" : false, "isCommentOnly" : false } ] }
{ "_id" : "bNP42FzgpCP2gyGjd", "title" : "Turkish", "slug" : "turkish", "members" : [ { "userId" : "Jq6mQp9m96YHcHnAM", "isAdmin" : true, "isActive" : true, "isNoComments" : false, "isCommentOnly" : false } ] }
jakubgs commented 2 years ago

And there we go:

> existingBoards = db.boards.find({}, {_id:1}).map(x => x._id)
> db.cards.deleteMany({boardId:{$not:{$in:existingBoards}}})
{ "acknowledged" : true, "deletedCount" : 789 }

The final count is:

> db.cards.count()
2703
> db.boards.count()
21

I think Wekan got noticeable faster.

jakubgs commented 2 years ago

Also upgraded Wekan to 6.52 and migrated to our own SSL proxy:

Now it uses proxy.infra.status.im instead of CloudFlare:

 > dig +short boards.status.im
proxy.infra.status.im.
143.198.248.188
178.128.136.222
jakubgs commented 2 years ago

We have 122 users that have not had any activity since 2021-01-01:

> db.users.find({'emails.0.address':{$not:{$regex:'.*@status.im'}},modifiedAt:{$lt:ISODate("2021-01-01")}}).count()
122

> db.users.find({modifiedAt:{$lt:ISODate("2021-01-01")}}).count()
190

> db.users.count()
394

It's 190 if we include @status.im addresses.

jakubgs commented 2 years ago

I've disabled all accounts with no activity after 2021-01-01:

> db.users.updateMany({modifiedAt:{$lt:ISODate("2021-01-01")}}, {$set:{loginDisabled:true}})
{ "acknowledged" : true, "matchedCount" : 188, "modifiedCount" : 186 }

If anyone complains we can always re-enable. If not we can delete them later.

jakubgs commented 2 years ago

Haven't heard complaints in a while.