Open amoeba opened 3 years ago
Regarding 1, the memory usage graphs show ~40% average memory use on cn-ucsb-1 over the last month with the current 64 GB allocated. We can easily go to 128 GB, higher is also an option, with max going to be ~700 GB.
Thanks @nickatnceas, that's great to know.
We noticed an out-of-sync state between the production CN and
urn:node:ARCTIC
the other day and found the CN thought it was completely in sync when it wasn't. In this particular case, the CN had failed to pick up tens of System Metadata updates fromurn:node:ARCTIC
we were expecting to see and the CN may have missed many more. I messaged @taojing2002 for help and we found that sync had crashed due to being OOM. Our fix was to set the last harvest timestamp back a day and allow processing to run. My immediate thoughts are:We talked about possible next steps on our dev call this week and came up with:
cn-ucsb-1
.ARCTIC
,ESS-DIVE
,RW
)For now, @taojing2002 is going to look into this and coordinate with @datadavev and we can go from there.
[Note: This might on the wrong repo since I can't see our logs on cn-ucsb-1 to see what actually crashed. Feel free to move.]