Discrete mysteries/goals/areas where logging could help more.

[gbinal]

• There are 2400 URLs whose primary scans do not complete, but that still have a page title. Why doesn't the http page status field populate? Could it?
  • Same but with final URL field.
  • Same but with final_url_media_type.
  • Alternatively, if it's a matter that failed scans overwrite the relevant fields and so that scan errors result in empty fields, it looks like maybe this doesn't take place with Page Title but doesn't for the other fields, so could we make the other fields resilient like Page title is?
• There are 3000 URLs where the primary scans timeout. Is there any way that we can generate more data about what is going on when this happens?
• For each browser error message listed here (e.g. ETIMEDOUT, ERR_CONNECTION_RESET, ERR_TOO_MANY_REDIRECTS), is it possible to generate more metadata around the error occurrence?

Questions:

• Does puppeteer ever fail to start correctly, e.g. 'checked browser health'= False; 'disconnected'=true; etc.
• Do any sites ever have GB sized memory logs? (eg rss, heapTotal, heapUsed, external, arrayBuffers)
• Is clarifyTimeoutError anything with potential?
• What are the numbers at the end of each error file location, e.g. the below. Specifically, can they tell us anything?
  • Answer: Yes, they are lines of code:character of code. But this is less useful since it's referring to compiled end code, not necessarily what we see in our repos.

at AxiosError.from (/path/to/site-scanning-engine/node_modules/axios/dist/node/axios.cjs:836:14)\n    
at RedirectableRequest.handleRequestError (/path/to/site-scanning-engine/node_modules/axios/dist/node/axios.cjs:3086:25)\n    
at RedirectableRequest.emit (node:events:518:28)\n    
at eventHandlers.<computed> (/path/to/site-scanning-engine/node_modules/follow-redirects/index.js:38:24)\n    
at ClientRequest.emit (node:events:518:28)\n    
at TLSSocket.socketErrorListener (node:_http_client:495:9)\n    
at TLSSocket.emit (node:events:518:28)\n    at emitErrorNT (node:internal/streams/destroy:169:8)\n    
at emitErrorCloseNT (node:internal/streams/destroy:128:3)\n  

[luke-at-flexion]

Q: Does puppeteer ever fail to start correctly, e.g. 'checked browser health'= False; 'disconnected'=true; etc.

A: Apparently, no.

Valid is never false. https://logs.fr.cloud.gov/goto/b0345ca1ac3f0860e60be4a95635b93b

Disconnected is never true. https://logs.fr.cloud.gov/goto/9f2b4e7a66d92a16e7fce455942c40e9

[luke-at-flexion]

Q: Do any sites ever have GB sized memory logs? (eg rss, heapTotal, heapUsed, external, arrayBuffers)

A: That's a great question. Reworded, do processes (such as scans) ever run out of memory and exit?

First, we can see that the answer to your direct question is "yes", heapTotal does exceed 1GB in some cases: https://logs.fr.cloud.gov/goto/e2a13ed049894a8cbcfb1dfee239325c

Second, we can see things crashing because they ran out of memory: https://logs.fr.cloud.gov/goto/a845704fdb5eee6cb714bd28b9f82aa5

... and finally, we can see container metrics that were emitted right before processes exceeded their memory quota: https://logs.fr.cloud.gov/goto/9ced552cc2f9b2d7afb702da0446c465

This definitely warrants further investigation because it's likely the cause of (at least) some of our failures.