search

NOAA-GSL / VxIngest

Other
2 stars 0 forks source link

Increase Capella RAM per node to 32 GB #431

Open gopa-noaa opened 3 days ago

gopa-noaa commented 3 days ago

Capella is giving low resident ratio errors, need to discuss if we can bump up RAM per node to 32 GB. This will increase our credit burn rate, need to estimate the new burn rate and see how long the current credis will take us with the increased burn rate.

ian-noaa commented 3 days ago

A couple of questions for discussion:

  1. What cluster resources was our original estimate based on? If the resource increase is below that, I think we can just go ahead and apply it.
  2. Since we're learning about the system, it'd be good to know what's causing the increased need for memory.
  3. It'd be good to know why we didn't get alerts for the low resident ratio errors.
gopa-noaa commented 3 days ago

Our estimate and recommended configuration was 4 CPU, 32 GB RAM, 3 TB per node. So since we have been burning at a lower rate so far, it would be fine to bump up to 32 GB. Per discussion today I will create another issue to address Capella TTLs, since we already have about 16 million documents in Capella now.

ian-noaa commented 3 days ago

For Time-To-Live, we had some prior discussion in #131. So we could reuse that or summarize the conclusions & open questions from there into a new implementation-focused issue.

gopa-noaa commented 3 days ago

Forgot about that one, thanks, let me read up and see which would be a better approach ...