DFE-Digital / polis-whitelabel

:milky_way: Open Source AI for large scale open ended feedback
https://pol.is
GNU Affero General Public License v3.0
3 stars 0 forks source link

Math container filling up with errconv.*.edn files #58

Closed brendanarnold closed 1 year ago

brendanarnold commented 1 year ago

Over time the math container fills up with what appears to be debugging files - this can cause the disk to become full and shut down the math node. Investigate a way to prevent this happening.

patcon commented 1 year ago

Maybe this is too naive, but math worker has always had challenges with periodic crashing and so needing to be restarted regularly. I'm reluctant to suggest it might be something as simple as an ephemeral file system filling up, but still... wondering if this is just with the white label fork, or if you suspect it's also upstream 🤔

brendanarnold commented 1 year ago

@patcon I guess the problem for this ticket is that if the math module starts erroring - for whatever reason - it can choke itself by filling up the disk so I wanted to put in some mitigation.

I'll make a separate ticket to address the actual error once I get chance to look into the error files.