Connection timeout and error: Server temporarily overloaded and cannot process your request

elenadimitrova commented 5 years ago

Our last few verify runs fail with Server temporarily overloaded and cannot process your request; please try again later. followed by a Failed in retrieving analysis response, HTTP status code: 401. UUID: 8de33ad2-3226-48f2-8c16-7dfa0880826f

Examples from the last 2 Circle CI builds below:

https://circleci.com/gh/JoinColony/colonyNetwork/8229

ColonyNetworkDataTypes: Server temporarily overloaded and cannot process your request; please try again later.
IRecovery: Server temporarily overloaded and cannot process your request; please try again later.
Data: Server temporarily overloaded and cannot process your request; please try again later.
ColonyFunding: Server temporarily overloaded and cannot process your request; please try again later.
OldRolesFactory: Failed in retrieving analysis response, HTTP status code: 401. UUID: 8de33ad2-3226-48f2-8c16-7dfa0880826f
OneTxPayment: IncomingMessage {
  _readableState:
   ReadableState {
     objectMode: false,
     highWaterMark: 16384,
     buffer: BufferList { head: null, tail: null, length: 0 },
     length: 0,
     pipes: null,
     pipesCount: 0,
     flowing: true,
     ended: true,
     endEmitted: true,
     reading: false,
     sync: true,
     needReadable: false,
     emittedReadable: false,
     readableListening: false,
     resumeScheduled: false,
     emitClose: true,
     destroyed: false,
     defaultEncoding: 'utf8',
     awaitDrain: 0,
     readingMore: true,
     decoder: null,
     encoding: null },
  readable: false,
  _events:
   { end: [ [Function: responseOnEnd], [Function] ],
     close: [ [Function], [Function] ],
     data: [Function],
     error: [Function] },
  _eventsCount: 4,
  _maxListeners: undefined,

https://circleci.com/gh/JoinColony/colonyNetwork/8224

NoLimitSubdomains
Internal MythX errors encountered:
ColonyFunding: Server temporarily overloaded and cannot process your request; please try again later.
ContractRecovery: Failed in submitting a JWT refresh request: Bad Request (HTTP status code: 400)
IPatriciaTree: Failed in retrieving analysis response, HTTP status code: undefined. UUID: 1edebdc6-dbbe-406e-b3a3-3130f0daea53
NoLimitSubdomains: IncomingMessage {
  _readableState:
   ReadableState {
     objectMode: false,
     highWaterMark: 16384,
     buffer: BufferList { head: null, tail: null, length: 0 },
     length: 0,
     pipes: null,
     pipesCount: 0,
     flowing: true,
     ended: true,
     endEmitted: true,
     reading: false,
     sync: true,
     needReadable: false,
     emittedReadable: false,
     readableListening: false,
     resumeScheduled: false,
     emitClose: true,
     destroyed: false,
     defaultEncoding: 'utf8',
     awaitDrain: 0,
     readingMore: true,
     decoder: null,
     encoding: null },
  readable: false,

nbanmp commented 5 years ago

We have been having some spotty backend issues recently, and are nearing stability again. I will transfer this to the mythx-developer-support repo, as it is an issue with the backend.

nbanmp commented 5 years ago

I believe backend issues have been for the most part resolved, so I will close this issue. If it is still a problem, please reopen it and explain what is happening.

elenadimitrova commented 5 years ago

@nbanmp please reopen the issue as it still presents itself. Just now when running locally I got:

Internal MythX errors encountered:
ColonyAuthority: Server temporarily overloaded and cannot process your request; please try again later.

nbanmp commented 5 years ago

Sorry for the delay. Reopening. Could you run one more time to see if the temporary overload has let up?

elenadimitrova commented 5 years ago

I just did three consecutive runs and all failed, each with the respective error message below:

1st fail:

Internal MythX errors encountered:
ColonyNetworkAuthority: Server temporarily overloaded and cannot process your request; please try again later.
ColonyNetworkStorage: Server temporarily overloaded and cannot process your request; please try again later.

2nd fail

Internal MythX errors encountered:
CommonStorage: Server temporarily overloaded and cannot process your request; please try again later.
PatriciaTree: Server temporarily overloaded and cannot process your request; please try again later.

3rd fail

Internal MythX errors encountered:
ColonyNetworkMining: request to https://api.mythx.io/v1/analyses/e10ef050-a555-4e0d-aa66-a9fb4c5f57fc/issues failed, reason: connect ETIMEDOUT 52.170.118.198:443
ColonyPayment: Server temporarily overloaded and cannot process your request; please try again later.
ColonyStorage: connect ETIMEDOUT 52.170.118.198:443
ReputationMiningCycleStorage: Server temporarily overloaded and cannot process your request; please try again later.
Resolver: Server temporarily overloaded and cannot process your request; please try again later.
UpdatedColonyNetwork: Server temporarily overloaded and cannot process your request; please try again later.

elenadimitrova commented 5 years ago

Also this is an example from 4 days ago in CI https://circleci.com/gh/JoinColony/colonyNetwork/8713

elenadimitrova commented 5 years ago

And right now not all files analysed are failing with Too Many Requests (HTTP status code: 429) :( https://circleci.com/gh/JoinColony/colonyNetwork/8769

muellerberndt commented 5 years ago

I think the issue should be resolved as we replaced the whole architecture backend in the meantime. Closing this issue :)

Consensys / mythx-developer-support

Connection timeout and error: Server temporarily overloaded and cannot process your request #21