[6:53 PM] Mat Richmond
Does the faucet do the write-to-disk thing like the address-book does?
[6:56 PM]
Philippe Camacho added Jeb Bearer to the chat and shared all chat history.
[6:56 PM] Philippe Camacho
Mat RichmondDoes the faucet do the write-to-disk thing like the address-book does?
Any idea Jeb Bearer?
[6:58 PM] Jeb Bearer
In the healthcheck you mean? I don't believe it does
[6:59 PM] Mat Richmond
we should do that I think.
[6:59 PM] Mat Richmond
At the very least, it should (hopefully) cause healthchecks to fail and the service to restart.
[7:00 PM] Jeb Bearer
I'm not sure that's it, though. The faucet writes to disk anyways every time there is a request, and in the background threads when it generates a transaction
[7:00 PM] Jeb Bearer
I guess at least it would help us catch I/O errors
[7:00 PM] Jeb Bearer
I think the root problem more likely has to do with either running out of space or exceeding some inode or dirent limit because of how many small files it is creating
[7:05 PM] Philippe Camacho
Sorry for the naive question: why does the faucet need to create many files?
[7:07 PM] Mat Richmond
So for a service design, fail fast should be the preferred option. Right now we have failures but the health check continues to be good, which hopefully that would prevent.
like 1
[7:08 PM] Mat Richmond
If the health check fails, then the service gets restarted and the symptoms are gone, even if the problem is still there.
[7:08 PM] Mat Richmond
Which preserves service availability.
[7:09 PM] Jeb Bearer
Yep definitely support adding it to healthcheck for that reason
[7:10 PM] Jeb Bearer
Philippe Camacho Sorry for the naive question: why does the faucet need to create many files?
It does not. It uses an implementation of RollingLog from atomicstore that is supposed to keep snapshots of the most recent N states, but the current implementation never actually prunes old states. I think this might be fixed on the latest atomicstore? But not completely sure, NY should know
[7:10 PM] Jeb Bearer
If that is the case, I don't think upgrading CAPE to the latest atomicstore would be too hard
heart 1 like 1
[7:10 PM] Jeb Bearer
CAPE + Seahorse, that is
[7:11 PM] Philippe Camacho
Cool, thanks Jeb Bearer
[6:53 PM] Mat Richmond Does the faucet do the write-to-disk thing like the address-book does?
[6:56 PM] Philippe Camacho added Jeb Bearer to the chat and shared all chat history.
[6:56 PM] Philippe Camacho Mat RichmondDoes the faucet do the write-to-disk thing like the address-book does? Any idea Jeb Bearer?
[6:58 PM] Jeb Bearer In the healthcheck you mean? I don't believe it does
[6:59 PM] Mat Richmond we should do that I think.
[6:59 PM] Mat Richmond At the very least, it should (hopefully) cause healthchecks to fail and the service to restart.
[7:00 PM] Jeb Bearer I'm not sure that's it, though. The faucet writes to disk anyways every time there is a request, and in the background threads when it generates a transaction
[7:00 PM] Jeb Bearer I guess at least it would help us catch I/O errors
[7:00 PM] Jeb Bearer I think the root problem more likely has to do with either running out of space or exceeding some inode or dirent limit because of how many small files it is creating
[7:05 PM] Philippe Camacho Sorry for the naive question: why does the faucet need to create many files?
[7:07 PM] Mat Richmond So for a service design, fail fast should be the preferred option. Right now we have failures but the health check continues to be good, which hopefully that would prevent. like 1
[7:08 PM] Mat Richmond If the health check fails, then the service gets restarted and the symptoms are gone, even if the problem is still there.
[7:08 PM] Mat Richmond Which preserves service availability.
[7:09 PM] Jeb Bearer Yep definitely support adding it to healthcheck for that reason
[7:10 PM] Jeb Bearer Philippe Camacho Sorry for the naive question: why does the faucet need to create many files? It does not. It uses an implementation of RollingLog from atomicstore that is supposed to keep snapshots of the most recent N states, but the current implementation never actually prunes old states. I think this might be fixed on the latest atomicstore? But not completely sure, NY should know
[7:10 PM] Jeb Bearer If that is the case, I don't think upgrading CAPE to the latest atomicstore would be too hard heart 1 like 1
[7:10 PM] Jeb Bearer CAPE + Seahorse, that is
[7:11 PM] Philippe Camacho Cool, thanks Jeb Bearer