Open Martii opened 5 years ago
Had a friend in Wisconson (say cheese ;) try it and it's not loading for him either.
Here's my traceroute:
$ time traceroute openuserjs.org
traceroute to openuserjs.org (104.236.255.50), 30 hops max, 60 byte packets
1 *(Intranet)* 0.614 ms 0.646 ms 0.672 ms
2 *(ISP)* 10.865 ms 10.872 ms 16.676 ms
3 *(ISP hop)* 20.595 ms 20.588 ms 20.592 ms
4 *(ISP hop)* 17.291 ms 17.301 ms 17.815 ms
5 *(ISP hop) 16.543 ms 17.109 ms 17.116 ms
6 *(ISP City hop)*.Level3.net 17.123 ms 16.745 ms 16.750 ms
7 ae-2-3602.ear4.Newark1.Level3.net (4.69.211.181) 64.112 ms 59.858 ms 59.861 ms
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
real 0m25.048s *(Time to site way to long for these requests)*
user 0m0.003s
sys 0m0.004s
Of the sites monitored by https://livemap.pingdom.com/ it's up to ~18,000 now but the Level3 map url is more specific to opening this issue.
I don't believe this is MongoLabs or AWS connection issue as dev is working at normal speed. Thinking the internet backbone is damaged via Level3 (owned by CenturyLink now... probably answers the why it is broken perhaps).
Anyhow... no new status updates. Spent a wonderful time with my ISP confirming they deny anything is problematic with their service (and cross referenced with a different provider out of State).
Temporarily downgraded node... not the issue and tested roll back to about a85b989 ... no change.
Some summary info our VPS provider had me run with some tests:
I've triple checked the firewall. Functional and okay.
Still awaiting any further suggestions including some preferred results... until then it's the waiting game. Ughh.
Still sitting around 1% to 9% with process usage (few spikes when downloading certain items but nominal is occurring). Removed letsencrypt package and it seemed to improve that from 3% (or just more ppl trying). Google Public DNS still hammering UDP (plus I'm trying it myself instead of my ISPs... no perceptual difference than native). I'd block it but then anyone using it wouldn't be able to reach us. Heh.
When it rains it pours... yet another issue bleh. (cascading is my guess)
Should rule out node by using nvm atm. e.g. gcc
issues with system compilied node. Using precompiled node. gcc (Debian 4.9.2-10+deb8u2) 4.9.2
is different than back in March and I do recall an extra notice within the last few months that said the distro was adding this for some compatibility check.
Passed all tests back then and even now... however since this issue appeared out of "thin air" I'm lead to believe we need a VPS migration to a newer distro. It's near the holidays and I'm super busy but I'll try to squeeze something in after @sizzlemctwizzle responds. Will try project in a VM first though as I've had that prepped since March. Until then this is BLOCKING which means sizzle has to unblock it when migration occurs.
Found that async dep is lagging on .parallel
and .series
. Test point console
dropped in here and it was definitely lagging there. Tried async@next mentioned from caolan/async#1589 and ended up with the same lagging issue... so possibly not a dep issue.
Facing 2-3 minutes of load on every page on a VPN secured connection through Iceland from Pakistan. Am on a 20Mbps U/D fiber connection from a Tier 2 ISP.
Tracert:
1 189 ms 189 ms 189 ms 10.8.8.1
2 189 ms 192 ms 191 ms 185.159.158.252
3 225 ms 228 ms 226 ms be-2-ver.peer1tc2.ams.nl.is1net.net [31.15.113.5]
4 224 ms 224 ms 256 ms ix-xe-4-1-1-0.tcore1.av2-amsterdam.as6453.net [195.219.194.109]
5 294 ms 295 ms 292 ms if-ae-2-2.tcore2.av2-amsterdam.as6453.net [195.219.194.6]
6 296 ms 295 ms 299 ms if-ae-14-2.tcore2.l78-london.as6453.net [80.231.131.160]
7 302 ms 300 ms * if-ae-4-2.tcore2.n0v-new-york.as6453.net [80.231.131.5]
8 296 ms 295 ms 294 ms if-ae-2-2.tcore1.n0v-new-york.as6453.net [216.6.90.21]
9 300 ms 300 ms 299 ms if-ae-7-2.tcore1.nto-new-york.as6453.net [63.243.128.25]
10 298 ms 302 ms 547 ms if-ae-9-2.tcore1.n75-new-york.as6453.net [63.243.128.122]
11 605 ms * 360 ms 66.110.96.22
12 * * * Request timed out.
13 * * * Request timed out.
14 306 ms 322 ms 306 ms 104.236.255.50
@brazenvoid Appreciate that report. Cleaning up my test VM before I create a new VPS with the next distro version. Want to make sure I'm not wasting money or too much time. So far in the VM it's not lagging but that's only using dev DBs/system. Local pro has to be setup and is quite lengthy to do it right. The real, final, test will have to be on production unfortunately. The VM I created mostly mirrors what we have on production.
Thanks for the continued patience. This is going to be interesting getting done in between the holiday stuff I have planned because I'll be AFK for quite a bit of it... so please continue patience. This might take a couple of weeks since I don't have full access. Sorry... but I'm trying. :) Until then just try to use the site as is... the best recommendation at the moment.
Hmmm new distro via MongoDB only has MongoDB 4... that can present a problem/delay with express-brute-mongo.
OOOH.. not good... local pro delay with next version of distro in VM. Guess I'll need to add that async test back in a local branch. It is about peak internet time so that could be a little bit of it too. Will do some more thorough testing.
Okay... pro was dragging AWS and MongoLabs down in local pro. Killed pro and local pro is at top speed (my usual perusing the site) atm. Still have the lurking Level3 issue in my outbound network for a possible additional reason.
Restored pro to "online-expected".
@sizzlemctwizzle
I'm still recommending the VPS upgrade to a new one esp. to (hopefully) resolve this issue. Adding the extra security will take a bit more time on the VPS but once it's all in place I can't move the DNS. That's something you'll need to do. Please stick with IPv4 for now. Some of our deps don't currently do well at IPv6. Plus this will cost some extra during the setup to migration from them. I'm building a list of what needs to be done. We really need the next level up to with more vCores which is more mulah per month.
I'm about beat from lack of sleep so I'll await your response(s) please.
Hmmm looks like I have DNS access change... we'll know for sure in ~24 to 48 hours.
So before the DNS propagated to me it was fine using the direct IP. Granted no one else had that IP. There's very few things left that I can think of.
The server was recreated to the same data center but better cpu/mem/ssd stats. I can try to migrate it to another data center but have to do a backup first (and another IP change and some other details too)
I'm also needing a break after 12 hours of this.
P.S. It's still lagging but lagging quicker heh. Note pingdom url is up to > 21000 atm
Misc notes:
devDependencies
... no differenceLocal pro test (this means I'm not using our VPS providers hosting for those new here but actual pro(duction) is running atm) with this diff and async dep at this code block point which has already contacted MongoLabs and succeeded in the callback but somehow messes up async now. (This code block point hasn't changed in quite some time):
diff --git a/controllers/user.js b/controllers/user.js
index 37ec725..3fa8f27 100644
--- a/controllers/user.js#b914392
+++ b/controllers/user.js
@@ -390,6 +390,9 @@ exports.userListPage = function (aReq, aRes, aNext) {
async.parallel([
function (aCallback) {
+
+ console.timeEnd('userListPage()');
+
if (!!!options.isFlagged || !options.isAdmin) { // NOTE: Watchpoint
aCallback();
return;
@@ -440,6 +443,9 @@ exports.userListPage = function (aReq, aRes, aNext) {
tasks.push(execQueryTask(userListQuery, options, 'userList'));
//---
+
+ console.time('userListPage()');
+
async.parallel(tasks, asyncComplete);
};
@@ -462,6 +468,9 @@ exports.view = function (aReq, aRes, aNext) {
async.parallel([
function (aCallback) {
+
+ console.timeEnd('view() ' + username);
+
if (!options.isAdmin) { // NOTE: Watchpoint
aCallback();
return;
@@ -525,6 +534,9 @@ exports.view = function (aReq, aRes, aNext) {
tasks = tasks.concat(stats.getSummaryTasks(options));
//---
+
+ console.time('view() ' + username);
+
async.parallel(tasks, asyncComplete);
});
};
... produces this output on some random clicks of users and user list:
view() -JesperJod: 332.085ms
view() -hoverboard: 69.237ms
view() -_ArmandLevas: 63.867ms
view() -lavienrose: 64.703ms
view() 00000H: 67.742ms
view() 0097gvk: 71.954ms
view() 04MR17: 75.263ms
view() 01018575475: 69.633ms
view() 007: 70.182ms
view() -mg-: 67.090ms
userListPage(): 405.968ms
view() 1544cman2000gmail.com: 70.499ms
view() 1solutions: 32389.779ms
view() 160004000: 25381.381ms
view() 1solutions: 64.616ms
... Some are quick... some are realllllllllllllly slow.
Same test with some more:
userListPage(): 134.631ms
userListPage(): 139.506ms
userListPage(): 109.424ms
userListPage(): 148.130ms
userListPage(): 147.365ms
userListPage(): 132.894ms
userListPage(): 136.901ms
userListPage(): 125.249ms
userListPage(): 129.335ms
userListPage(): 127.833ms
userListPage(): 119.766ms
userListPage(): 135.394ms
userListPage(): 132.066ms
userListPage(): 234.387ms
userListPage(): 769.262ms
userListPage(): 121.794ms
userListPage(): 111.775ms
userListPage(): 122.818ms
userListPage(): 108.172ms
userListPage(): 388.980ms
userListPage(): 352.293ms
userListPage(): 149.513ms
userListPage(): 172.532ms
userListPage(): 150.295ms
userListPage(): 594.304ms
userListPage(): 221.400ms
userListPage(): 195.818ms
userListPage(): 174.826ms
userListPage(): 198.745ms
userListPage(): 204.019ms
userListPage(): 135.008ms
userListPage(): 105.093ms
userListPage(): 112.189ms
userListPage(): 93.537ms
userListPage(): 111.668ms
userListPage(): 235.212ms
userListPage(): 259.961ms
userListPage(): 304.103ms
userListPage(): 318.409ms
userListPage(): 489.174ms
userListPage(): 582.791ms
userListPage(): 139.654ms
userListPage(): 122.359ms
userListPage(): 771.824ms
userListPage(): 601.540ms
userListPage(): 199.029ms
view() 93Akkord: 19528.984ms
view() 99aintenough: 9438.322ms
view() 9tfall: 111.546ms
view() 9kopb: 69.801ms
view() AAK: 66.806ms
view() ADRENALINE1234: 71.058ms
view() AJMansfield: 24289.024ms
... last 10 or so userListPage()
are actually different pages... the ones above are the same exact page over and over (page refresh like a human could do).
Finally!!! Found evidence and confirmation of network issue on production (from VPS to MongoLabs):
2018-12-18 13:30:17.100 +00:00: Group rating NOT updated aErr := MongoNetworkError: connection 10 to *clipped*.mongolab.com:*portclipped* timed out aGroup := undefined
I've temporarily audited url usage (on pro) this morning for about 4 minutes on a single thread and most of the requests are to .meta.js and .user.js which is AWS and not MongoLabs.
The main site is still sluggish for me (several minutes to load)... but the requests for .user.js (AWS not Mongo?) is completely unresponsive. Nothing but HTTP code 429 (too many requests) 95% of the time, and the other 5% return HTTP code 444 (unknown).
Is the AWS instance rate limited like this? Also, no "Retry-After" header in the response.
@Joeviocoe
AWS not Mongo?
See https://github.com/OpenUserJS/OpenUserJS.org/issues/1548#issuecomment-447667327 for the breakdown of which one is which. Both IPs that we connect to are Amazon IPs though which I didn't know until that comment.
Is the AWS instance rate limited like this? Also, no "Retry-After" header in the response.
See #944 (Hopefully this isn't a repeat of that as well... when it rains it pours is the axiom... as I said above this came out of "thin air" not even during our standard update cycle) ... in short yes and correct.
but the requests for .user.js (AWS not Mongo?) is completely unresponsive.
I will test that next. Thanks for the reminder. See also https://openuserjs.org/about with site messages about // @updateURL
if you are using OUJS .user.js . This could be an issue externally, with GH for example.
Tests in SM latest with GM Port latest:
Would be helpful to have a sample traceroute
(Linux/macOS) or tracert
(Windows) to our site of openuserjs.org
posted too. This is a very large issue that has cascaded across dependencies which makes it harder to debug. So any extra info is quite helpful. Our VPS should be watching this issue but they also are investigating it on their end.
Some of our routines silently handle any kind of error too... some don't... which I've never really approved of and in some cases can be fatal... see #430 ... working on that now but I'm almost out of time to do this as the last several days I've ignored real life duties. There are a few logs to review and comment about disconnections to the DB's.
Most recent, and only on this server restart, log message:
2018-12-19 05:07:22.573 +00:00: { MongoNetworkError: connection 2 to *clipped*.mongolab.com:43493 timed out
at Socket.<anonymous> (~/OpenUserJS.org2/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:259:7)
at Object.onceWrapper (events.js:273:13)
at Socket.emit (events.js:182:13)
at Socket.EventEmitter.emit (domain.js:441:20)
at Socket._onTimeout (net.js:453:8)
at ontimeout (timers.js:436:11)
at tryOnTimeout (timers.js:300:5)
at listOnTimeout (timers.js:263:5)
at Timer.processTimers (timers.js:223:10)
name: 'MongoNetworkError',
errorLabels: [ 'TransientTransactionError' ],
[Symbol(mongoErrorContextSymbol)]: {} }
@Joeviocoe
With my edited test results up there the one with no content is fatal to end users. Until this is resolved the site is offline. I've put the parking page on it just so you know our server is still there. You, and everyone else, may see it if I do any pro tests... but mostly I'll be doing this in local pro.
@sizzlemctwizzle we need you here please. Also I would like access to MongoLabs and AWS (if they have teams just add me somehow). We have got to figure this out.
Well here's local pro, at current project HEAD, on how I see it. I'm the only user connected to the DB's right now and it is full speed. Doing this image to show that at least writes to the DB seem to be happening on actual pro. If it were to fail with no content on write to them that would mean a loss of script source which is why it is parked atm. I can possibly turn on script storage read only and see how that behaves on actual pro.
We're in RO script storage mode. See https://openuserjs.org/about for site status messages. Will see when this starts to lag for me.
Already lagging... That was quick.
> db['bruteforce-store'].count()
2727
That's since yesterday of unique IPs... so it's well below the threshhold when we were encountering a .user.js engine malfunction in #944. Those addresses are either behind a proxy or have a faulty .user.js engine that isn't paying attention to the cache header tags (this includes the last GM 3.x line.. but not GM Port last release). The stats on it are low though.
Manually changed poolSize
to 2
since my last comment and get these:
stderr:
2018-12-19 10:29:30.828 +00:00: MongoDB connection is disconnected
2018-12-19 10:29:30.831 +00:00: MongoDB connection is disconnected
2018-12-19 10:29:30.832 +00:00: MongoDB connection is disconnected
2018-12-19 10:29:30.832 +00:00: MongoDB connection is disconnected
2018-12-19 10:36:46.687 +00:00: MongoDB connection is reconnected
2018-12-19 10:37:21.554 +00:00: MongoDB connection is reconnected
2018-12-19 10:45:34.813 +00:00: MongoNetworkError: connection 51 to *clipped*.mongolab.com:*portclipped* timed out
at Socket.<anonymous> (~/OpenUserJS.org2/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:259:7)
at Object.onceWrapper (events.js:273:13)
at Socket.emit (events.js:182:13)
at Socket.EventEmitter.emit (domain.js:441:20)
at Socket._onTimeout (net.js:453:8)
at ontimeout (timers.js:436:11)
at tryOnTimeout (timers.js:300:5)
at listOnTimeout (timers.js:263:5)
at Timer.processTimers (timers.js:223:10)
... interesting... all four connections (processes) and only two came back after about 5 minutes.
stdout (not sure this should be there):
2018-12-19 10:45:34.769 +00:00: { MongoNetworkError: connection 51 to *clipped*.mongolab.com:*clipped* timed out
at Socket.<anonymous> (~/OpenUserJS.org2/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:259:7)
at Object.onceWrapper (events.js:273:13)
at Socket.emit (events.js:182:13)
at Socket.EventEmitter.emit (domain.js:441:20)
at Socket._onTimeout (net.js:453:8)
at ontimeout (timers.js:436:11)
at tryOnTimeout (timers.js:300:5)
at listOnTimeout (timers.js:263:5)
at Timer.processTimers (timers.js:223:10)
name: 'MongoNetworkError',
errorLabels: [ 'TransientTransactionError' ],
[Symbol(mongoErrorContextSymbol)]: {} }
... TransientTransactionError
???
Refs:
TransientTransactionError
)
reconnectTries
- If you're connected to a single server or mongos proxy (as opposed to a replica set), the MongoDB driver will try to reconnect every reconnectInterval milliseconds for reconnectTries times, and give up afterward. When the driver gives up, the mongoose connection emits a reconnectFailed event. This option does nothing for replica set connections.reconnectInterval
- See reconnectTries
Change in site response... pending...
And change back... had about 4 minutes of sheer bliss where it was full speed on pro... however back to lagging... UGGH.
Need some ppl to login and peruse the site some more... right now it's just me because I was running some tests with a theory.
Typically it's about 4 ppl during all of this testing... one helps but need more. I can't click in two accounts... I have my Moderator account also logged in but I can't interact with it since I only have two hands on the same machine. LOL
We're still in RO so you won't be able to save scripts but try installing something even if it's your own scripts. Try a search, etc. just spend a few minutes here and there.
Login page is fast now. Landing is still slow at 3.5s. Script pages take ~2s but are good enough.
Installing script ends up in a white page with url: https://openuserjs.org/install/brazenvoid/PH_-_Search_UI_Tweaks.user.js#bypass=true
Failed request
{
"log": {
"version": "1.1",
"creator": {
"name": "Firefox",
"version": "64.0"
},
"browser": {
"name": "Firefox",
"version": "64.0"
},
"pages": [
{
"startedDateTime": "2018-12-19T21:54:19.786+05:00",
"id": "page_1",
"title": "brazenvoid | Users | OpenUserJS",
"pageTimings": {
"onContentLoad": 516,
"onLoad": 620
}
}
],
"entries": [
{
"pageref": "page_1",
"startedDateTime": "2018-12-19T21:54:19.786+05:00",
"request": {
"bodySize": 0,
"method": "GET",
"url": "https://openuserjs.org/install/brazenvoid/PH_-_Search_UI_Tweaks.user.js#bypass=true",
"httpVersion": "HTTP/1.1",
"headers": [
{
"name": "Host",
"value": "openuserjs.org"
},
{
"name": "User-Agent",
"value": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0"
},
{
"name": "Accept",
"value": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
},
{
"name": "Accept-Language",
"value": "en-US,en;q=0.5"
},
{
"name": "Accept-Encoding",
"value": "gzip, deflate, br"
},
{
"name": "DNT",
"value": "1"
},
{
"name": "Connection",
"value": "keep-alive"
},
{
"name": "Cookie",
"value": "connect.sid=s%3A5lNPrMidd818RR4qa287nK5S0ZjKVX5N.E1kbDuDqfgblg2V3Di%2BL0R7je3WBwv6pFTeSKGvA7Og"
},
{
"name": "Upgrade-Insecure-Requests",
"value": "1"
}
],
"cookies": [
{
"name": "connect.sid",
"value": "s:5lNPrMidd818RR4qa287nK5S0ZjKVX5N.E1kbDuDqfgblg2V3Di+L0R7je3WBwv6pFTeSKGvA7Og"
}
],
"queryString": [],
"headersSize": 488
},
"response": {
"status": 444,
"statusText": "unknown",
"httpVersion": "HTTP/1.1",
"headers": [
{
"name": "X-Powered-By",
"value": "Express"
},
{
"name": "Strict-Transport-Security",
"value": "max-age=31536000000; includeSubDomains"
},
{
"name": "Warning",
"value": "199 openuserjs.org Invalid @updateURL in lockdown"
},
{
"name": "set-cookie",
"value": "connect.sid=s%3A5lNPrMidd818RR4qa287nK5S0ZjKVX5N.E1kbDuDqfgblg2V3Di%2BL0R7je3WBwv6pFTeSKGvA7Og; Path=/; Expires=Wed, 19 Dec 2018 22:54:18 GMT; HttpOnly; Secure; SameSite=Strict"
},
{
"name": "Date",
"value": "Wed, 19 Dec 2018 16:54:20 GMT"
},
{
"name": "Connection",
"value": "keep-alive"
},
{
"name": "Transfer-Encoding",
"value": "chunked"
}
],
"cookies": [
{
"name": "connect.sid",
"value": "s:5lNPrMidd818RR4qa287nK5S0ZjKVX5N.E1kbDuDqfgblg2V3Di+L0R7je3WBwv6pFTeSKGvA7Og"
}
],
"content": {
"mimeType": "application/x-javascript",
"size": 0,
"text": ""
},
"redirectURL": "",
"headersSize": 453,
"bodySize": 453
},
"cache": {},
"timings": {
"blocked": 1,
"dns": 0,
"connect": 0,
"ssl": 0,
"send": 0,
"wait": 393,
"receive": 0
},
"time": 394,
"_securityState": "secure",
"serverIPAddress": "68.183.63.30",
"connection": "443"
},
{
"pageref": "page_1",
"startedDateTime": "2018-12-19T21:54:20.333+05:00",
"request": {
"bodySize": 0,
"method": "GET",
"url": "https://openuserjs.org/favicon.ico",
"httpVersion": "HTTP/1.1",
"headers": [
{
"name": "Host",
"value": "openuserjs.org"
},
{
"name": "User-Agent",
"value": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0"
},
{
"name": "Accept",
"value": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
},
{
"name": "Accept-Language",
"value": "en-US,en;q=0.5"
},
{
"name": "Accept-Encoding",
"value": "gzip, deflate, br"
},
{
"name": "DNT",
"value": "1"
},
{
"name": "Connection",
"value": "keep-alive"
},
{
"name": "Cookie",
"value": "connect.sid=s%3A5lNPrMidd818RR4qa287nK5S0ZjKVX5N.E1kbDuDqfgblg2V3Di%2BL0R7je3WBwv6pFTeSKGvA7Og"
}
],
"cookies": [
{
"name": "connect.sid",
"value": "s:5lNPrMidd818RR4qa287nK5S0ZjKVX5N.E1kbDuDqfgblg2V3Di+L0R7je3WBwv6pFTeSKGvA7Og"
}
],
"queryString": [],
"headersSize": 421
},
"response": {
"status": 200,
"statusText": "OK",
"httpVersion": "HTTP/1.1",
"headers": [
{
"name": "X-Powered-By",
"value": "Express"
},
{
"name": "Strict-Transport-Security",
"value": "max-age=31536000000; includeSubDomains"
},
{
"name": "Cache-Control",
"value": "public, max-age=31536000"
},
{
"name": "ETag",
"value": "\"3326-Z+bMgTx7Wv8xsjloebss3Z8wHQs\""
},
{
"name": "Content-Type",
"value": "image/x-icon"
},
{
"name": "set-cookie",
"value": "connect.sid=s%3A5lNPrMidd818RR4qa287nK5S0ZjKVX5N.E1kbDuDqfgblg2V3Di%2BL0R7je3WBwv6pFTeSKGvA7Og; Path=/; Expires=Wed, 19 Dec 2018 22:54:18 GMT; HttpOnly; Secure; SameSite=Strict"
},
{
"name": "Vary",
"value": "Accept-Encoding"
},
{
"name": "Content-Encoding",
"value": "gzip"
},
{
"name": "Date",
"value": "Wed, 19 Dec 2018 16:54:20 GMT"
},
{
"name": "Connection",
"value": "keep-alive"
},
{
"name": "Transfer-Encoding",
"value": "chunked"
}
],
"cookies": [
{
"name": "connect.sid",
"value": "s:5lNPrMidd818RR4qa287nK5S0ZjKVX5N.E1kbDuDqfgblg2V3Di+L0R7je3WBwv6pFTeSKGvA7Og"
}
],
"content": {
"mimeType": "image/x-icon",
"size": 13094,
"encoding": "base64",
"text": "AAABAAQAEBAAAAEACABoBQAARgAAACAgAAABAAgAqAgAAK4FAAAwMAAAAQAIAKgOAABWDgAAQEAAAAEACAAoFgAA/hwAACgAAAAQAAAAIAAAAAEACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABuTQUAcE8KAHJRDgBzUhAAel8hAHtgIgB/ZRgAfmMmAH9kJwCNdzoAjng7AJJ8PwCTfUEAlH5CAMC8pADBvp4A2t/VANzh1wDd4tgA3uPZAN/k2gDg5dsA3ejiAOHs5QAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFRUVFRUVFRUVFRUVFRUVFRUVFRUVExYXFxYTFRUVFRUVFRUUFg8MBAQLDxYUFRUVFRUUFQkAAQICAQAJFRQVFRUVFgkDBg8QEA8HAwkWFRUVEw8ACBETFBQTEgcADxMVFRYMAQ4TFRUVFRMPAQsWFRUXBQIQFBUVFRUUEAIEFxUVFwUCEBQVFRUVFBACBBcVFRYNAA8TFRUVFRMPAQwWFRUTDwAHERMUFBMSBwAPExUVFRYKAwcOEBAPBwMJFhUVFRUUFQoAAQICAQAJFRQVFRUVFRQWDwwFBQwPFhQVFRUVFRUVFRMWFxcWExUVFRUVFRUVFRUVFRUVFRUVFRUVFQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAoAAAAIAAAAEAAAAABAAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAcE8KAHJRDgB1VgAAc1IQAHZXAAB3WAAAdFMSAHVUEwB4WQIAdlUVAHxcDAB4XR4Ae2AiAH5jJgB/ZCcAgWYpAIlvJgCJbTMAhnAyAI13OgCOeDsAj3k8AJF7PgCSfD8AmIdRAJmIUgCvoXoAsKJ7AKqifwCxo3wAq6OAAKykgQC0q4gAtayJALm2lgDAwqYAxsGpAMDBrADHwqoAwcKtAMLDrgDIybUAy8y3AM/QuwDR18YA0tnHANfc0wDZ4M8A2t/VANvg1gDc4dcA3uPZAN/k2gDg5dsA2uXeAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NDU1NjU1NjU1NDU1NTU1NTU1NTU1NTU1NTU1NTU1NTUvIxoYExMYGiUwNTU1NTU1NTU1NTU1NTU1NTU1NTQuIBEJAQAAAAAACRAgLjU1NTU1NTU1NTU1NTU1NTU0KhYGAAAAAAAAAAAAAAMWKTQ1NTU1NTU1NTU1NTU1NCcOAQAAAAcLDAwLBwAAAAEOJzQ1NTU1NTU1NTU1NTQqDgEAAAUUHyMrKyUfFQUBAAEOKTU1NTU1NTU1NTU1LhYBAAENHiwzNTU1NTMtHw0BAAEWLjU1NTU1NTU1NTQgBgAADSIxNDU1NTU1NTUyIg4BAAYgNTU1NTU1NTU0LhIAAAUdMTU1NTU1NTU1NTUyHwUAABEvNDU1NTU1NTUmBAAAFCw0NTU1NTU1NTU1NTQsFQAACSU1NTU1NTU1NRoBAAcfMzU1NTU1NTU1NTU1NTMfBwAAGjU1NTU1NTU1GQAACiQ0NTU1NTU1NTU1NTU1NSUKAAAYNjU1NTU1NTYTAAAMKzQ1NTU1NTU1NTU1NTU1KwwAABM1NTU1NTU1NRQBAAwrNDU1NTU1NTU1NTU1NTUrDAAAEzU1NTU1NTU1GAAACiQ0NTU1NTU1NTU1NTU1NSMLAAAYNjU1NTU1NTUcAQAHHzM1NTU1NTU1NTU1NTUzHwcAABo1NTU1NTU1NScFAAAULDU1NTU1NTU1NTU1NCwVAAAJJTU1NTU1NTU0LhIAAAUeMTU1NTU1NTU1NTUyHggAABEvNDU1NTU1NTU0IQYAAA0iMTQ1NTU1NTU0MSINAAAGITU1NTU1NTU1NTUuFwAAAQ0eLDM1NDQ1MyweDQEAARYuNTU1NTU1NTU1NTUpDwAAAAUUHyQrKyQfFQUAAAEOKjQ1NTU1NTU1NTU1NTQoDwEAAAECCgwMCgcAAAABDic0NTU1NTU1NTU1NTU1NTQpFwcAAAAAAAAAAAAAAxYqNDU1NTU1NTU1NTU1NTU1NTQuIRIFAAEBAQEABRIgLjQ1NTU1NTU1NTU1NTU1NTU1NTU0LiYbGBQUGBonLjU1NTU1NTU1NTU1NTU1NTU1NTU1NTU0NTU1NTU1NTU0NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACgAAAAwAAAAYAAAAAEACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABwTwoAclEOAHVVAAB1VgAAc1IQAHRTEgB1VBMAeFkCAHZVFQB7WwkAfFwMAH1dDgB3XB0AeF0eAHpfIQB8YSMAfWIkAH9kJwCAZSgAgWYpAIZsIgCHbSQAiG4lAIdrMACJbyYAinAoAIVvMQCKdDcAi3U4AJF7PgCSfD8Ak31BAJeBRQCTgkwAl4ZQAJmIUgCgj1oAoZBbAKORXACdkWEAopZmAKKVbQCjl2cApJdvAKWYcACooH0AqaJ3AK+hegCpoX4AsKJ7AKujgACtpYIArqaDALCnhACxqIUAtKuIALi1lQC5tpYAvLmZAL26mgC+u5sAv7ycAMK/nwDDwKAAxL+nAMDCpgDGwakAxMWxAMXGsgDGx7MAx8i0AMjJtQDMzbgAzc65ANHXxgDS2ccA09rIANTbyQDX3NMA197NANjfzgDZ4M8A2t/VANvg1gDc4dcA3eLYAN7j2QDf5NoA4OXbAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFdYV1hYWFhXWFdYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhXV1ZVVFJSUlJUVVZXV1hYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhUSkU4MSgkJCgxOERKVFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFdXUUY2IhYPBwYBAQYHDxYiNUZSV1dYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYV1VJLxwKBQAAAAAAAAAAAAAFDBsvSFVYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhXUDshDgQAAAAAAAAAAAAAAAAAAAQKIDtPV1hYWFhYWFhYWFhYWFhYWFhYWFhYWFdMNBQGAAAAAAAAAQEBAQEBAAAAAAAABhQzTFdYWFhYWFhYWFhYWFhYWFhYWFhYV0wvEQEAAAABBAYNEhUZGRUSDQYEAQAAAAERL0xXWFhYWFhYWFhYWFhYWFhYWFhXUDQRBQAAAAAIER4qNzxBQT03Kh4SCAAAAAAFETNPWFhYWFhYWFhYWFhYWFhYWFdVOxQBAAAABAsfM0NLTlVWVlVOS0M0HgsEAAAAARQ6VVdYWFhYWFhYWFhYWFhYWFdJIQYAAAACEidATVZYWFhYWFhYWFZNPyUSBgAAAAYgSFdYWFhYWFhYWFhYWFhYWFEvDgAAAAQSK0dSVlhYWFhYWFhYWFhWUkcsEgQAAAAKL1JYWFhYWFhYWFhYWFhYWEccBQAAAAsmR1RXWFhYWFhYWFhYWFhYV1VHJgsAAAAFHEZYWFhYWFhYWFhYWFhXVDYLAAABCB9AUldYWFhYWFhYWFhYWFhYWFdSPx4IAQAACjZUV1hYWFhYWFhYWFhXSiIGAAAEETNNVlhYWFhYWFhYWFhYWFhYWFhWTTQSBAAABSJKV1hYWFhYWFhYWFhWRRgAAAAGHkNWWFhYWFhYWFhYWFhYWFhYWFhYVkMeBgAAABZEVldYWFhYWFhYWFdWOBAAAAANKkpYWFhYWFhYWFhYWFhYWFhYWFhYWEsqDQAAAA84VVhYWFhYWFhYWFdUMgcAAAASN05YWFhYWFhYWFhYWFhYWFhYWFhYWE43EgEAAAcxVFdYWFhYWFhYWFdTKAMAAAAVPFRYWFhYWFhYWFhYWFhYWFhYWFhYWFU9FQEAAAYoUlhYWFhYWFhYWFhSJAQAAAAZQVZYWFhYWFhYWFhYWFhYWFhYWFhYWFZBGQAAAAEkUlhYWFhYWFhYWFhSJAQAAAAZQVZYWFhYWFhYWFhYWFhYWFhYWFhYWFZBGQAAAAEkUlhYWFhYWFhYWFdTKQMAAAAVPFRYWFhYWFhYWFhYWFhYWFhYWFhYWFU8FwEAAAYoUlhYWFhYWFhYWFdUMgcAAAASN05YWFhYWFhYWFhYWFhYWFhYWFhYWE43EgEAAAcxVFdYWFhYWFhYWFdWORAAAAANKkpYWFhYWFhYWFhYWFhYWFhYWFhYWEsqDQAAAA84VVhYWFhYWFhYWFhWRRgAAAAGHkNVWFhYWFhYWFhYWFhYWFhYWFhYVkMeBgAAABZEVldYWFhYWFhYWFhXSiMDAAABETNNVlhYWFhYWFhYWFhYWFhYWFhWTTMRBAAABSJKV1hYWFhYWFhYWFhXVDYNAAABCB4+UldYWFhYWFhYWFhYWFhYWFdTPh8IAQAACjZUV1hYWFhYWFhYWFhYWEccBQAAAAonR1RXWFhYWFhYWFhYWFhYWFRHJwsAAAAFHEZYWFhYWFhYWFhYWFhYWFEuDgAAAAQSK0dSVlhYWFhYWFhYWFhWUkcsEgQAAAAKL1FYWFhYWFhYWFhYWFhYWFdJIQYAAAACEiU+TVZYWFhYWFhYWFZNPicSAgAAAAYgSVdYWFhYWFhYWFhYWFhYWFhVOxUEAAAABAofM0NLTlVWVlVOS0MzHwoEAAAAARQ7VFdYWFhYWFhYWFhYWFhYWFhXUDQTBQAAAAAIER0qNzxCQjw3Kh4RCAAAAAAFETRQV1hYWFhYWFhYWFhYWFhYWFhYV0wwEwEAAAABBAYNEhUZGRUSDQYEAQAAAAERLUxXWFhYWFhYWFhYWFhYWFhYWFhYWFdMNBUGAQAAAAAAAAAAAAAAAAAAAAAABhQ0TFdYWFhYWFhYWFhYWFhYWFhYWFhYWFhXUDshDgUAAAAAAAAAAAAAAAAAAAUKITtQV1hYWFhYWFhYWFhYWFhYWFhYWFhYWFhYV1VJLhwNAwAAAAAAAAAAAAADDRwvSFVXWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhXUUc2IxoPCQMEBAMJDxgjNkdRV1dYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhUSkU5MikkJCkxOEVKVFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhXV1ZWVFNSUlNUVlZXV1hYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhXV1dYWFdXV1hYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAKAAAAEAAAACAAAAAAQAIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHBPCgByUQ4AdVYAAHNSEAB2VwAAd1gAAHRTEgB1VBMAeFkCAHZVFQB7WwkAfV0OAHdcHQB4XR4Ael8hAHtgIgB8YSMAfWIkAH5jJgB/ZCcAgWYpAIRqHwCCZysAhWkuAIdtJACGai8AiG4lAIlvJgCHcTMAiHI0AIlzNgCLdTgAjHY5AI13OgCOeDsAj3k8AJB6PQCSfD8Ak31BAJeBRQCVhE4AloVPAJeGUACci1YAoI9aAKGQWwCjkVwAnZFhAJ6SYgCfk2MAopZmAKSYaACnmnIAqp1tAKuddQCsnnYAr6F6AKmhfgCtpYIAs6qHALSriAC1rIkAtq2KALeviwC4sIwAubGNALe0lAC4tZUAubaWALu4mAC9upoAvrubAL+8nADBvp4AxsGpAMfCqgDCw64AxcayAMfItADIybUAzs+6AM/QuwDQ0bwA0tO/ANPUwADO1cQA0NbFANHXxgDS2ccA1NvJANXcygDX3NMA197NANngzwDa39UA2+DWANzh1wDd4tgA3uPZAN/k2gDg5dsAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGNiX11cWVlZWVxdX2JjZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRjYVxVT0lAOjY0NDY6QElOVVxhY2RkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGJdVkw9MikhGhQODQ0OFBohKDI9TFZdYmRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkY19USTQnGAsIAgMBAAAAAAEDAgULGCc0SVNfY2RkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkYllLNCQRBQMBAAAAAAAAAAAAAAAAAQMFESM0SlliZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRjYFI/KRQFAQAAAAAAAAAAAAAAAAAAAAAAAAEEEyg/Ul9jZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRjXU00HQwDAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMMHTRNXWNkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRjXUwwFgQBAAAAAAAAAQMHAgkEBAkCBwMBAAAAAAAAAQQULkxbY2RkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRjXUwsEwcAAAAAAAEDBwgRFxwfIiIfHBcRCAcDAQAAAAAABxMsTF1jZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkYE0wEwcAAAAAAAEJDhgmLzg9REZGRD44LyYYDgkBAAAAAAAHEy5NX2RkZGRkZGRkZGRkZGRkZGRkZGRkZGRkYlI0FgcAAAAAAQMKGCo4RU1SWFxbW1xYUk1FOSoYCgYBAAAAAAcWNFFiZGRkZGRkZGRkZGRkZGRkZGRkZGRkY1k/HQQAAAAAAQIPIzRIUFxgYmJjZGRjYmJgXFFJNCMPAgEAAAAABB0/WWNkZGRkZGRkZGRkZGRkZGRkZGRkZF9LKQwBAAAAAQQUKTtOWmFjZGRkZGRkZGRkZGNhWk88KRQEAQAAAAEMKEpfZGRkZGRkZGRkZGRkZGRkZGRkZGJUNBQDAAAAAQIUK0NSXmJkZGRkZGRkZGRkZGRkZGJeU0MrFAIBAAAAAxM0U2JkZGRkZGRkZGRkZGRkZGRkZGRdSSQFAAAAAAMPKkNUYGNkZGRkZGRkZGRkZGRkZGRkY2BVQyoQBgAAAAAFJEldZGRkZGRkZGRkZGRkZGRkZGRjVjUSAwAAAAEKIzxSYGNkZGRkZGRkZGRkZGRkZGRkZGRjYFM8IwoBAAAAARE0VmNkZGRkZGRkZGRkZGRkZGRkYUwnCAAAAAEJGDROXmNkZGRkZGRkZGRkZGRkZGRkZGRkZGNeTzQYCQEAAAAFJ0xhZGRkZGRkZGRkZGRkZGRkZFw9GAcAAAADDipIWmJkZGRkZGRkZGRkZGRkZGRkZGRkZGRkYlpIKg4DAAAAAxg9XGRkZGRkZGRkZGRkZGRkZGNVMw4BAAAABxg4UGFkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRhUTkYBwAAAAELMlVjZGRkZGRkZGRkZGRkZGRiTykIAAAAAQgmRVpiZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkY1xFJggBAAAACClOYmRkZGRkZGRkZGRkZGRkYEkiBAAAAAMRLU1fZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRgTS8RAwAAAAIhSV9kZGRkZGRkZGRkZGRkZF1BGwcAAAAGFzhSYmRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkYlI4FwcAAAAHGkBdZGRkZGRkZGRkZGRkZGRcOhQDAAAABxw9V2JkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGJYPhwCAAAAARQ6XGRkZGRkZGRkZGRkZGRkWjYPAQAAAAIfRFxjZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRjXEQgCQAAAAEONllkZGRkZGRkZGRkZGRkZFk0DQEAAAAJIkZbY2RkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZFtGIgkAAAAADTRZZGRkZGRkZGRkZGRkZGRZNA0BAAAACSJGW2NkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRbRiIJAAAAAA00WWRkZGRkZGRkZGRkZGRkWjcPAQAAAAIfRFxjZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRjXEQgCQAAAAEPNllkZGRkZGRkZGRkZGRkZFw6FAMAAAAHHD1XYmRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkYlg9HAIAAAABFDpcZGRkZGRkZGRkZGRkZGRdQRsHAAAABhc4UmJkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGJSOBcHAAAABxpAXWRkZGRkZGRkZGRkZGRkYEkiBAAAAAMRLU1fZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRgTS8RAwAAAAIiSV9kZGRkZGRkZGRkZGRkZGJPKQgAAAABCCZFWmJkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRjXEUmCAEAAAAIKU5iZGRkZGRkZGRkZGRkZGRjVTMOAQAAAAcYOFBgZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkYVA5GAcAAAABCzJVY2RkZGRkZGRkZGRkZGRkZFw+GgcAAAABDipIWmJkZGRkZGRkZGRkZGRkZGRkZGRkZGRkYlpIKg4DAAAAAxg9XGRkZGRkZGRkZGRkZGRkZGRhTCcIAAAAAQkZNE5eY2RkZGRkZGRkZGRkZGRkZGRkZGRkY15ONBgJAQAAAAUnTGFkZGRkZGRkZGRkZGRkZGRkY1Y1EgMAAAABCCM7UmBjZGRkZGRkZGRkZGRkZGRkZGRkY2BTPCMKAQAAAAERNFZjZGRkZGRkZGRkZGRkZGRkZGRdSSQFAAAAAAMPKUJUYGNkZGRkZGRkZGRkZGRkZGRkY2BUQykPAwAAAAAFJEldZGRkZGRkZGRkZGRkZGRkZGRkYlQ1FAMAAAABAhQrQ1JeYmRkZGRkZGRkZGRkZGRkYl5SQysUAgEAAAADEzRUYmRkZGRkZGRkZGRkZGRkZGRkZGRfSykMAQAAAAEEFCk7TlphYmRkZGRkZGRkZGRiYVpOOykUBAEAAAABDChLX2RkZGRkZGRkZGRkZGRkZGRkZGRkY1lAHgQAAAAAAQIPIjRHUFpgYmJjY2NjYmJgXFBINCMPAgEAAAAABB0/WWNkZGRkZGRkZGRkZGRkZGRkZGRkZGRiUjQVAgAAAAABAwgYKjhFTVJXXFtbXFdSTUU5KhgIAwEAAAAABxY0UmJkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGBNMRQHAAAAAAABCQ4ZJS04PURGRkQ9OC8mGA4JAQAAAAAABxMwTWBkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRjXkwvFAcAAAAAAAEDBwgRFxwfIiIfHBcRCAcDAQAAAAAABxMsTF5jZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGNdTDEVBQEAAAAAAAABAwYHAgkJAgcGAwEAAAAAAAABBBYwTF1jZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkY15NNB4MBwAAAAAAAAAAAAAAAAAAAAAAAAAAAAADDB00TV1jZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRjYFJAKRQFAwAAAAAAAAAAAAAAAAAAAAAAAAMFEyk/UmBjZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRiWUs1JBIIBwEAAAAAAAAAAAAAAAABBwgSJDRKWWJkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGNfVEk1JxoOCAIHAwEBAQEDBwIIDhonNUlUX2NkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGJdVkw+MyoiGxQPCwsPFBsiKTM+TFZdYmRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGNhXFVPSUE6NzQ0NzpASU9VXGFjZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRjYmBdXFpZWVpcXWBiY2RkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="
},
"redirectURL": "",
"headersSize": 546,
"bodySize": 3475
},
"cache": {},
"timings": {
"blocked": 1,
"dns": 0,
"connect": 0,
"ssl": 0,
"send": 0,
"wait": 300,
"receive": 518
},
"time": 819,
"_securityState": "secure",
"serverIPAddress": "68.183.63.30",
"connection": "443"
}
]
}
}
Login page is fast now.
I de async'd that page... the real test will be when more ppl try all at the same time. I'm getting script source here but I also booted 3 ppl off while I was testing... apologies but was necessary to do this to make a clean break.
Well it's lagging with another party attempting to do something... don't know what though... not url monitoring.
Refreshed the script info page and it has again gone into a wait loop
Ok got loaded in 2.09 minutes. Tried again to install, the request failed with 444 after 992ms.
Well I've got confirmation that async fails when AWS/MongoLabs has issues... they'll be issued after mongoose responds to my issue hopefully. I'll keep the login page de async'd for the time being manually.
Up to 3 users doing something (that includes you @brazenvoid ;)
I'm also on the website doing random stuff and it's been really fast for the last minute or so.
I closed my session. I just wanted to update my scripts to newest versions essentially. But as that's not up yet so I'll help around, if I am able.
Tried a very rough transition (and I do mean rough) to #847 with connect-mongodb-session instead of connect-mongo and it had the same issue... so not that deps issue.
@brazenvoid A mindless activity is to load a bunch of the User homepages from the Users... don't have to be logged in. Oh and thank you. :)
On a script page, the source code tab works fine, but the raw page ends in "ERR_INVALID_RESPONSE" weirdly.
@Azkellas Hmmm that's why we're in RO mode. I'd hate to have this AWS failure write that to that DB. Discussions for me are going super slow at the moment... just random clicking. Thanks for the test result too.
@Azkellas
Btw the source code page is AWS along with the raw source... everything else is MongoLabs.
I'm also testing, in a mobile device... Hit and miss... But user.js still always fails. Is that parked, or set up for Read Only?
@Joeviocoe
RO only... parked would be a 503 page and nothing works. You can still click around though but you'll always get a 503 on every route (url address)
Interesting...tried to delete a script and now it's lagging 100% in every tab. (this is a reason I keep so many disposable test scripts on the site :)
And back to full speed... 3 minute time out... one more time to confirm maybe?
Nope. Not confirmed.
Btw if you are lucky enough to hit a clear point RO only applies to script sources... e.g. one can comment. I seem to remember I did that so if there's a script issue one might be able to comment about it.
Perhaps a permanent banner/announcement can be added on the site for unsuspecting users as to what they can do.
@brazenvoid You mean this? https://openuserjs.org/about that page has no async (and no DB) and always tells you what's up. Not going to do it across the site atm. These situations aren't typical so I'd rather not cause a complete panic.
Perhaps you can change/duplicate the privacy announcement for that purpose and also make it permanent.
So I've been contacting all kinds of people in the last 24 hours with no clear resolution on why sometimes the site is super fast vs super slow.
When I know anything more I'll let everyone know however there isn't much to be done as everything has been triple checked on our end (hence the dep updates a couple of times in the last few days [that's not typical for my updating], server restarts, and we did an unannounced backup since it's about that time.).
Anyhow... just letting everyone know we're on top of what we can do. Apologies for what we can't do.
P.S. When it's sitting idle in the browser (spinner spinning) it's doing nothing in the network management tools e.g. your request isn't reaching us every time atm from the test points we tried and our VPS (real person) confirmed occasionally it's taking an excessive period from their testing.
Refs: