Open davemfish opened 8 months ago
Doug and I talked about possibly dis-allowing Taskgraph's DEBUG logging to reach the Workbench at all, only every allowing it to go the file handler. But that might involve some other tough decisions. Some options to accomplish this,
DEBUG
logging for Taskgraph for all invest
CLI users (the Workbench being one such user)DEBUG
menu option from the Workbench UI so a user can never select it. But this menu controls the level for both the StreamHandler
and the FileHandler
, so these would need to be de-coupled if we still want DEBUG
logging available for the file. And I don't much like the idea of adding more UI widgets to control all this, as most Workbench users do not know/care about all these options.So far I'm inclined to leave the Workbench logging as-is, assuming we can make it perform adequately.
Here's the ultimate crash from the Workbench. Out-of-memory.
[1] Error sending from webFrameMain: Error: Render frame was disposed before WebFrameMain could be accessed
[1] at s.send (node:electron/js2c/browser_init:2:93158)
[1] at _.send (node:electron/js2c/browser_init:2:78814)
[1] at C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\electronApi.js:205:23
[1] at Array.forEach (<anonymous>)
[1] at sendIpcToRenderer (C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\electronApi.js:203:42)
[1] at Object.sendIpc (C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\electronApi.js:185:5)
[1] at transport (C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\transports\ipc.js:40:17)
[1] at runTransport (C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\log.js:44:5)
[1] at runTransports (C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\log.js:27:7)
[1] at log (C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\log.js:21:3)
[1] [10:27:51.012] [undefined] { reason: 'oom', exitCode: -536870904 }
[1] Error sending from webFrameMain: Error: Render frame was disposed before WebFrameMain could be accessed
[1] at s.send (node:electron/js2c/browser_init:2:93158)
[1] at _.send (node:electron/js2c/browser_init:2:78814)
[1] at C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\electronApi.js:205:23
[1] at Array.forEach (<anonymous>)
[1] at sendIpcToRenderer (C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\electronApi.js:203:42)
[1] at Object.sendIpc (C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\electronApi.js:185:5)
[1] at transport (C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\transports\ipc.js:40:17)
[1] at runTransport (C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\log.js:44:5)
[1] at runTransports (C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\log.js:27:7)
[1] at log (C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\log.js:21:3)
I saw memory usage get up to 8 or 9 GB which seems strange when the entire logfile itself is only 159MB. So perhaps there is a memory leak here.
The renderer's memory use is also very high when displaying a pre-existing logfile, such as from a recently completed invest run. For example, a logfile that is 25MB on disk causes the renderer to use ~1GB of memory when displaying the log. Closing the model tab does not immediately free the memory, though it does eventually. And then baseline memory use for the renderer is about 100 MB.
Rob gave us a datastack that includes a large
wind_data_path
CSV input. It's about 5MB, which is 2-3x larger than our sample data CSV. WhenTaskgraph
logging level is set toDEBUG
, a single line that is more than 18 million characters long is logged. This causes the Workbench renderer to consume an unreasonable amount of memory and ultimately crash.The line begins,
That
other_argument
is thewind_data
argument in,For some reason, instead of passing the CSV as an argument to
_compute_density_harvested_fields
, we first read the CSV into a dict and pass that, sowind_data
here is a dict. I don't see any reason why we need to do this, so passing the CSV into the function would avoid this logging problem.Should the workbench more gracefully handle logging lines that are this long? Perhaps.