search

ulixee / hero

The web browser built for scraping
MIT License
696 stars 33 forks source link

TLS re-negotiate ciphers (edited title) #163

Open bratao opened 2 years ago

bratao commented 2 years ago

I´m trying to access a specific website ('https://esaj.tjsp.jus.br/cposg/search.do'), that works good on all major browsers. However on secret-agent I get an error. I already tired to use the process.env.MITM_ALLOW_INSECURE=true; flag, but it still rejects the connection.

I´m using Windows 10.

2021-09-17T03:11:47.295Z STATS [E:\Projetos\secret-python\node_modules\@secret-agent\core\lib\FrameNavigations] emit:status-change {
  id: 1,
  url: 'https://esaj.tjsp.jus.br/cposg/search.do',
  stateChanges: {
    HttpRequested: '2021-09-17T03:11:46.906Z',
    HttpResponded: '2021-09-17T03:11:47.175Z',
    DomContentLoaded: '2021-09-17T03:11:47.221Z',
    Load: '2021-09-17T03:11:47.221Z',
    ContentPaint: '2021-09-17T03:11:47.288Z'
  },
  newStatus: 'ContentPaint',
  context: { sessionId: 'fd0403f0-1764-11ec-8229-432b2c56418c', frameId: 1 }
}
2021-09-17T03:11:47.343Z INFO [E:\Projetos\secret-python\node_modules\@secret-agent\mitm-socket\lib\MitmSocketSession] MitmSocketSession.onMessage {
  error: 'remote error: tls: handshake failure',
  'error-step': 'emulateTls',
  id: 4,
  status: 'error',
  context: { sessionId: 'fd0403f0-1764-11ec-8229-432b2c56418c' }
}
2021-09-17T03:11:47.344Z INFO [E:\Projetos\secret-python\node_modules\@secret-agent\mitm-socket\index] MitmSocket.error {
  message: 'remote error: tls: handshake failure',
  host: '200.142.86.246',
  context: { sessionId: 'fd0403f0-1764-11ec-8229-432b2c56418c' }
}
2021-09-17T03:11:47.345Z INFO [E:\Projetos\secret-python\node_modules\@secret-agent\mitm-socket\index] MitmSocket.Closing { context: { sessionId: 'fd0403f0-1764-11ec-8229-432b2c56418c' } }
2021-09-17T03:11:47.346Z STATS [E:\Projetos\secret-python\node_modules\@secret-agent\mitm-socket\index] MitmSocket.Closed { context: { sessionId: 'fd0403f0-1764-11ec-8229-432b2c56418c' } }
2021-09-17T03:11:47.346Z INFO [E:\Projetos\secret-python\node_modules\@secret-agent\mitm\handlers\HttpRequestHandler] MitmHttpRequest.ProxyToServer.RequestHandlerError {
  request: 'GET: https://esaj.tjsp.jus.br/favicon.ico',
  error: 'Error: Failed to connect to esaj.tjsp.jus.br',
  context: {}
} SocketConnectError: Failed to connect to esaj.tjsp.jus.br
    at buildConnectError (E:\Projetos\secret-python\node_modules\@secret-agent\mitm-socket\index.js:195:17)
    at MitmSocket.close (E:\Projetos\secret-python\node_modules\@secret-agent\mitm-socket\index.js:67:87)
    at MitmSocket.onError (E:\Projetos\secret-python\node_modules\@secret-agent\mitm-socket\index.js:160:14)
    at MitmSocket.onMessage (E:\Projetos\secret-python\node_modules\@secret-agent\mitm-socket\index.js:119:18)
    at MitmSocketSession.onMessage (E:\Projetos\secret-python\node_modules\@secret-agent\mitm-socket\lib\MitmSocketSession.js:45:93)
    at MitmSocketSession.onIpcData (E:\Projetos\secret-python\node_modules\@secret-agent\mitm-socket\lib\BaseIpcHandler.js:129:14)
    at Socket.emit (events.js:400:28)
    at addChunk (internal/streams/readable.js:290:12)
    at readableAddChunk (internal/streams/readable.js:265:9)
    at Socket.Readable.push (internal/streams/readable.js:204:10)
------DIAL----------------------------------------
    Error
    at new MitmSocket (E:\Projetos\secret-python\node_modules\@secret-agent\mitm-socket\index.js:27:26)
    at MitmRequestAgent.createSocketConnection (E:\Projetos\secret-python\node_modules\@secret-agent\mitm\lib\MitmRequestAgent.js:120:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async Object.cb (E:\Projetos\secret-python\node_modules\@secret-agent\mitm\lib\SocketPool.js:57:32)
    at async Queue.next (E:\Projetos\secret-python\node_modules\@secret-agent\commons\Queue.js:68:25)
------SOCKET TO ORIGIN----------------------------
  at Queue.run (E:\Projetos\secret-python\node_modules\@secret-agent\commons\Queue.js:28:25)
    at SocketPool.getSocket (E:\Projetos\secret-python\node_modules\@secret-agent\mitm\lib\SocketPool.js:42:27)
    at MitmRequestAgent.assignSocket (E:\Projetos\secret-python\node_modules\@secret-agent\mitm\lib\MitmRequestAgent.js:112:39)
    at MitmRequestAgent.request (E:\Projetos\secret-python\node_modules\@secret-agent\mitm\lib\MitmRequestAgent.js:52:20)
    at HttpRequestHandler.createProxyToServerRequest (E:\Projetos\secret-python\node_modules\@secret-agent\mitm\handlers\BaseHttpHandler.js:38:56)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async HttpRequestHandler.onRequest (E:\Projetos\secret-python\node_modules\@secret-agent\mitm\handlers\HttpRequestHandler.js:25:42)
    at async Function.onRequest (E:\Projetos\secret-python\node_modules\@secret-agent\mitm\handlers\HttpRequestHandler.js:203:9)
    at async MitmProxy.onHttpRequest (E:\Projetos\secret-python\node_modules\@secret-agent\mitm\lib\MitmProxy.js:163:13)
2021-09-17T03:11:47.947Z STATS [E:\Projetos\secret-python\node_modules\@secret-agent\puppet-chrome\lib\Frame] emit:frame-lifecycle {
  frame: {
    id: 'AB090297237523D8525A17E1F02B34F9',
    parentId: undefined,
    activeLoaderId: 'F50ED56CD507AF55E69FA5D54B36400D',
    name: '',
    url: 'https://esaj.tjsp.jus.br/cposg/search.do',
    navigationReason: undefined,
    disposition: undefined,
    isLoaderResolved: true,
    lifecycle: {
      init: '2021-09-17T03:11:47.176Z',
      load: '2021-09-17T03:11:47.229Z',
      DOMContentLoaded: '2021-09-17T03:11:47.241Z',
      firstPaint: '2021-09-17T03:11:47.288Z',
      firstContentfulPaint: '2021-09-17T03:11:47.290Z',
      firstMeaningfulPaintCandidate: '2021-09-17T03:11:47.293Z',
      networkAlmostIdle: '2021-09-17T03:11:47.947Z'
    }
  },
  name: 'networkAlmostIdle',
  loaderId: 'F50ED56CD507AF55E69FA5D54B36400D',
  context: {
    sessionId: 'fd0403f0-1764-11ec-8229-432b2c56418c',
    browserContextId: 'F86E226B9380A95D7855D1FE8125B17D',
    targetId: 'AB090297237523D8525A17E1F02B34F9'
  }
}
blakebyrnes commented 2 years ago

Thanks for the report. It looks like TLS fails in regular Chrome too, but regular Chrome re-proposes a ClientHello with cipher suite "TLS_RSA_WITH_3DES_EDE_CBC_SHA". We'll need to add fallback tls handshakes into the MITM.

blakebyrnes commented 1 year ago

To fix this, I think we need to create a Double Agent test to measure re-proposed TLS specs. Then we'll need to modify unblocked-web/agent/mitm-socket/go code to handle the retry