Hashcat wrapper doesn't tolerate network failures

[bzekanovic]

Noticed that my job would just kill the EC2 instance and after looking at output.log file below log showed up.

Any idea why this would happen?

Error sending status update to API Gateway Error: getaddrinfo EAI_AGAIN api.npk.DOMAIN.com at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:72:26) { errno: -3001,

[bzekanovic]

Additional info from output log file.

9.04% finished @ 6,081,428,737H/s Error sending status update to API Gateway Error: getaddrinfo EAI_AGAIN api.npk.DOMAIN.com at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:72:26) { errno: -3001, code: 'EAI_AGAIN', syscall: 'getaddrinfo', hostname: 'api.npk.DOMAIN.com', config: { url: 'https://api.npk.DOMAIN.com/v1/statusreport/us-east-2%REDACTED/performance', method: 'post', data: '{"startTime":1649306980,"estimatedEndTime":1649307283,"hashRate":6081428737,"progress":"9.04","hashes":1,"recoveredHashes":0,"recoveredPercentage":"0.00","rejectedPercentage":"5.74","performance":{"1":6081428737}}', headers: { Accept: 'application/json', 'Content-Type': 'application/json', 'x-amz-date': '20220407T045007Z', Authorization: 'AWS4-HMAC-SHA256 Credential=REDACTED/REDACTED/us-east-2/execute-api/aws4_request, SignedHeaders=accept;content-type;host;x-amz-date, Signature=REDACTED', 'x-amz-security-token': 'REDACTED', 'User-Agent': 'axios/0.21.4', 'Content-Length': 213 }, transformRequest: [ [Function: transformRequest] ], transformResponse: [ [Function: transformResponse] ], timeout: 0, adapter: [Function: httpAdapter], xsrfCookieName: 'XSRF-TOKEN', xsrfHeaderName: 'X-XSRF-TOKEN', maxContentLength: -1, maxBodyLength: -1, validateStatus: [Function: validateStatus], transitional: { silentJSONParsing: true, forcedJSONParsing: true, clarifyTimeoutError: false } }, request: <ref *1> Writable { _writableState: WritableState { objectMode: false, highWaterMark: 16384, finalCalled: false, needDrain: false, ending: false, ended: false, finished: false, destroyed: false, decodeStrings: true, defaultEncoding: 'utf8', length: 0, writing: false, corked: 0, sync: true, bufferProcessing: false, onwrite: [Function: bound onwrite], writecb: null, writelen: 0, afterWriteTickInfo: null, buffered: [], bufferedIndex: 0, allBuffers: true, allNoop: true, pendingcb: 0, constructed: true, prefinished: false, errorEmitted: false, emitClose: true, autoDestroy: true, errored: null, closed: false, closeEmitted: false,

},
_events: [Object: null prototype] {
  response: [Function: handleResponse],
  error: [Function: handleRequestError]
},
_eventsCount: 2,
_maxListeners: undefined,
_options: {
  maxRedirects: 21,
  maxBodyLength: 10485760,
  protocol: 'https:',
  path: '/v1/statusreport/us-east-REDACTED',
  method: 'POST',
  headers: [Object],
  agent: undefined,
  agents: [Object],
  auth: undefined,
  hostname: 'api.npk.DOMAIN.com',
  port: null,
  nativeProtocols: [Object],
  pathname: '/v1/statusreport/us-east-REDACTED/performance'
},
_ended: false,
_ending: true,
_redirectCount: 0,
_redirects: [],
_requestBodyLength: 213,
_requestBodyBuffers: [ [Object] ],
_onNativeResponse: [Function (anonymous)],
_currentRequest: ClientRequest {
  _events: [Object: null prototype],
  _eventsCount: 7,
  _maxListeners: undefined,
  outputData: [],
  outputSize: 0,
  writable: true,
  destroyed: false,
  _last: true,
  chunkedEncoding: false,
  shouldKeepAlive: false,
  maxRequestsOnConnectionReached: false,
  _defaultKeepAlive: true,
  useChunkedEncodingByDefault: true,
  sendDate: false,
  _removedConnection: false,
  _removedContLen: false,
  _removedTE: false,
  _contentLength: null,
  _hasBody: true,
  _trailer: '',
  finished: false,
  _headerSent: true,
  _closed: false,
  socket: [TLSSocket],
  _header: 'POST /v1/statusreport/us-east-2%REDACTED/performance HTTP/1.1\r\n' +
    'Accept: application/json\r\n' +
    'Content-Type: application/json\r\n' +
    'x-amz-date: 20220407T045007Z\r\n' +
    'Authorization: AWS4-HMAC-SHA256 Credential=REDACTED' +
    'User-Agent: axios/0.21.4\r\n' +
    'Content-Length: 213\r\n' +
    'Host: api.npk.DOMAIN.com\r\n' +
    'Connection: close\r\n' +
    '\r\n',
  _keepAliveTimeout: 0,
  _onPendingData: [Function: nop],
  agent: [Agent],
  socketPath: undefined,
  method: 'POST',
  maxHeaderSize: undefined,
  insecureHTTPParser: undefined,
  path: '/v1/statusreport/us-east-2%REDACTED/performance',
  _ended: false,
  res: null,
  aborted: false,
  timeoutCb: null,
  upgradeOrConnect: false,
  parser: null,
  maxHeadersCount: null,
  reusedSocket: false,
  host: 'api.npk.DOMAIN.com',
  protocol: 'https:',
  _redirectable: [Circular *1],
  [Symbol(kCapture)]: false,
  [Symbol(kNeedDrain)]: false,
  [Symbol(corked)]: 0,
  [Symbol(kOutHeaders)]: [Object: null prototype]
},
_currentUrl: 'https://api.npk.DOMAIN.com/v1/statusreport/us-east-2%REDACTED/performance',
[Symbol(kCapture)]: false

}, response: undefined, isAxiosError: true, toJSON: [Function: toJSON] } node:internal/process/promises:246 triggerUncaughtException(err, true / fromPromise /); ^

[UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "false".] { code: 'ERR_UNHANDLED_REJECTION' }

Node.js v17.0.1 [*] Hashcat wrapper finished with status code 1

[c6fc]

Looks like a DNS resolution failure in your VPC. This was probably a blip in AWS networking, but if it's reproducible let me know.

[bzekanovic]

I can still reproduce it with some, but not other instances. Not sure which instances work as its pain to test each one individually. Specifically, I ran campaign with G4DN instance in us-east-1.

Anything that can be done to prevent this with Hashcat wrapper?

[bzekanovic]

I'm also able to reproduce this in my staging environment.

[c6fc]

is there anything else you can tell me about the campaign you're running? Are you using custom or community-provided wordlists or rules? Are you doing a mask attack? Can you provide the hashcat parameters from the log file?

Does it always send successful status reports first? Does it error at the same percentage or after roughly the same duration?

[c6fc]

Also, what is your primaryRegion?

[bzekanovic]

This one has me going in circles and not sure why it works sometimes, but sometimes its consistently failing.

Scenario 1 - Staging Org - 1400 hash type - us-west-2 - M60 G3S instance - rockyou wordlist - OneRule... rule file Scenario 1 Results - this time its successful and no failures in output log file. I got the cracked_hashes file and everything looked good. Scenario 1 other results - ran through couple of more with custom wordlists and no issues. (this was failing before).

Scenario 2 - Prod Org - 1400 hash type - us-west-2 - M60 G3S instance - rockyou wordlist - OneRule... rule file Scenario 2 Results - on this one I did see the error message in output logs, but my cracked_hashes file still showed up.

Basically, I have no idea why sometimes its failing and others its fine.

I hope this helps and let me know if you have any other recommendations regarding this.

I'm going to run through couple of more tests in prod org and see how it goes.

[bzekanovic]

I just did another test with actual prod hash that is not as easy to crack and below are results.

Scenario 1 - Staging Org - 13100 hash type - us-west-2 - M60 G3S instance - rockyou wordlist - OneRule... rule file Scenario 1 Results - NPK ran the job without any issues and actually returned cracked hashes file and kept sending status metrics.

Scenario 2 - Prod Org - 13100 hash type - us-west-2 - M60 G3S instance - rockyou wordlist - OneRule... rule file Scenario 2 Results - NPK did return cracked hashes file, but it never sent status metrics to NPK due to same error message as above. The job ran for a while until it just terminated and never showed status bar.

[bzekanovic]

I may have found the issue and I'm just testing if thats the cause.

[bzekanovic]

Yeah, I feel pretty damn dumb. Basically, I did not realize that I had left over DNS records for api.npk.domain.com pointing to different NS servers and thats is exactly why it was failing (as you stated above). https://isitdns.com/