search

storacha / project-tracking

🐾 Used as central/default repo for project management, backlog, etc.
0 stars 0 forks source link

aRc to Station #9

Closed alanshaw closed 6 months ago

reidlw commented 7 months ago

Travis working on this; isn't going to take 100% of time for upcoming sprint.

travis commented 7 months ago

I believe this is just waiting on guidance from @reidlw and @mroth on the PR and node operator relations elements of rolling it out - I don't believe we're blocked on the technical side.

reidlw commented 7 months ago

@travis is there a document with the context/open questions for us to page into? If needed, can you grab time with me, @heyjay44, @hannahhoward and @prodalex next week to walk through what we need to consider for rolling this out?

travis commented 7 months ago

We met yesterday to chat about this and came up with the following launch plan for the aRc->Voyager transition:

  1. we've asked the station folks to ramp up requests to max capacity (1M requests/hour) - they may not be able to to this until they have finished integrating Voyager into Meridian, the Station payment system but they are increasing the number of requests 6x compared to the testing period - after that change it looks like we are getting ~350-450 thousand requests per hour from Station (though it does seem to be pretty variable right now - it spiked up to about 1.2M per hour and has been dropping over the last few hours)
  2. we'd like to start ramping down traffic over the course of a month. I'd propose the following (very open to different percentages):
    1. move aRc traffic down to 90% of peak in first week
    2. 70% next week
    3. 50% next week
    4. 30% next week
    5. 0% following week

I believe we are clear to move to 90% of peak immediately - @hannahhoward offered to operate the levers on this - scale at will @hannahhoward !

reidlw commented 7 months ago

Update to traffic ramp down plan...

@hannahhoward to drop Arc to 50% of peak load on Thursday April 11th @hannahhoward to drop Arc to 25% of peak load on Tuesday April 16th @hannahhoward to turn all Arc traffic off on Monday, April 22nd.

hannahhoward commented 6 months ago

we are at 50% -- 4/15/2024

reidlw commented 6 months ago

This is like 99.9% done.

travis commented 6 months ago

PR to go to 0: https://github.com/Arc-io/core/pull/410