[EVENT] OpenScapes at AGU 2021

choldgraf commented 2 years ago

Summary

OpenScapes is running a session at AGU 2021, and will need a hub scale-up similar to the one that we ran in November.

Event Info

Community Representative: @jules32
Event begin: December 12th
Event end: December 12th
Active times:
Number of attendees: 18
Hub Events Calendar

Hub info

Hub URL:
Hub decommisioned after event?: No

Task List

Before the event

[x] Dates confirmed with the community representative and added to Hub Events Calendar
[x] One week before event Hub is running.

[x] Confirm with Community Representative that their workflows function as expected.

👉Template message to send to community representative

Hey {{ COMMUNITY REPRESENTATIVE }}, the date of your event is getting close!

Could you please confirm that your hub environment is ready-to-go, and matches your hub's infrastructure setup, by ensuring the following things:
- [ ] Confirm that the "Event Info" above is correct
- [ ] On your hub: log-in and authentication works as-expected
- [ ] `nbgitpuller` links you intend to use resolve properly
- [ ] Your notebooks and content run as-expected

[x] 1 day before event, either a separate nodegroup is provisioned for the event or the cluster is scaled up.

During and after event

[x] Confirm event is finished.
[x] Nodegroup created for the hub is decommissioned / cluster is scaled down.
~[ ] Hub decommissioned (if needed).~
[x] Debrief with community representative.
- 👉Template debrief to send to community representative
``` Hey {{ COMMUNITY REPRESENTATIVE }}, your event appears to be over 🎉 We hope that your hub worked out well for you! We are trying to understand where we can improve our hub infrastructure and setup around events, and would love any feedback that you're willing to give. Would you mind answering the following questions? If not, just let us know and that is no problem! - Did the infrastructure behave as expected? - Anything that was confusing or could be improved? - Any extra functionality you wish you would have had? - Could you share a story about how you used the hub? - Any other feedback that you'd like to share? ```

choldgraf commented 2 years ago

Update: giving `eksctl` a shot

In the team meeting today we discussed a few things relevant to this issue:

This is the first time (?) that we've supported a hub over a weekend. Usually when an event begins there are a few fires to put out, but we should generally try not to put ourselves on a critical path during weekend use.
In addition, our current kops setup is a bit special-cased, because the team does not have equal knowledge about how kops works. We have plans to move to eksctl, but this hasn't happened yet.
@yuvipanda has offered to try a time-boxed migration to eksctl in the hopes that we can make this transfer quickly, and thus the hub will be more maintainable by multiple people rather than just him and @damianavila

So our next steps are to wait for @yuvipanda to try the eksctl migration, and if this is straightforward, come up with a plan tomorrow for how we'd like to support the extra users on the hub.

I'm gonna assign @yuvipanda , @damianavila , and myself on this so we don't lose track of it, but we should narrow this down once we decide who will take the scale up / down actions.

damianavila commented 2 years ago

@yuvipanda confirmed he succeeded in the migration and there is now a PR to review at https://github.com/2i2c-org/infrastructure/pull/885.

betolink commented 2 years ago

Quick question, did this migration also affected https://staging.openscapes.2i2c.cloud/? this is the hub we are planning in using this Sunday.

choldgraf commented 2 years ago

I suspect that the staging hub has also moved to eksctl as well. Though can I ask why you intend on using that hub? In general staging hubs are not meant for end-users, but they are instead there to test out changes, make sure things are running properly, etc.

yuvipanda commented 2 years ago

Yeah, that was moved too. But I highly recommend using openscapes.2i2c.cloud instead - as @choldgraf says, staging hubs are meant to test changes, and not for production use.

betolink commented 2 years ago

I guess we can use production but technically that's being used for the cloud hackathon participants, maybe we can add the AGU participants to prod and just remove their github handles after a week. What do you think @jules32 @erinmr?

jules32 commented 2 years ago

Hi All, Yes we can use the production and remove their github handles afterwards.

damianavila commented 2 years ago

@betolink @jules32, a few more questions for you:

Are you planning to use the small instances (the same you used in the previous hackathon), right? I am asking this question to know which pool of nodes we should be warming before the event.
Can you confirm the exact times for the event?

erinmr commented 2 years ago

Hi @damianavila - we are planning to use small instances. The event starts at 9am ET/6am PT and ends at 1pm ET/10am PT.

damianavila commented 2 years ago

@erinmr @jules32 @betolink,

I have scaled-up the cluster a few minutes ago and 80 small instances are available for you to use. If you need more than that during the event, the autoscaler should handle it automatically since I have just raised the minimum amount of nodes for the cluster.

Since the event is finishing at 1 pm ET, my plan is to cold down the cluster by 2 pm ET. The cold down process should not kick out anyone actively using their servers (if they want to keep working on stuff), it will just reduce the minimum number of nodes to zero, and then, the culling process on unused servers + the autoscaler will reduce the nodes count until we achieve "normal" levels.

If you have any general questions (such as the ones we have been discussing above), it is OK to post them here.

But if you have specific questions or issues during the event, please reach out to the support email so we can properly prioritize them and act on them.

Thanks!

jules32 commented 2 years ago

Thank you @damianavila, this is perfect.

damianavila commented 2 years ago

FYI, the cluster was already scaled down. I hope you had a great event!!

jules32 commented 2 years ago

Thanks Damian!

On Sun, Dec 12, 2021, 12:02 Damian Avila @.***> wrote:

FYI, the cluster was already scaled down. I hope you had a great event!!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/2i2c-org/infrastructure/issues/870#issuecomment-991961362, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABM6ORPOSAZ4KWN52WT5EMDUQT5WPANCNFSM5JKZO3DA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

damianavila commented 2 years ago

@erinmr @jules32 @betolink, usual set of questions to gather some feedback from you. If you are inclined to answer, it would be appreciated!

We hope that your hub worked out well for you! We are trying to understand where we can improve our hub infrastructure and setup around events, and would love any feedback that you're willing to give. Would you mind answering the following questions? If not, just let us know and that is no problem!

Did the infrastructure behave as expected?
Anything that was confusing or could be improved?
Any extra functionality you wish you would have had?
Could you share a story about how you used the hub?
Any other feedback that you'd like to share?

erinmr commented 2 years ago

Hi @damianavila -

Things went smoothly and as expected and nothing confusing or for improvement. We had 18 instances running during the workshop. We are curious about usage statistics mentioned earlier last week. This time used nbgitpuller to simplify the participant workflow by removing the github cloning step. I am going to work on a few stories by end of the year to share back on our experience with 2i2c infrastructure. Thanks so much!

damianavila commented 2 years ago

Thanks for the quick reply, @erinmr! I am going to close this one now but feel free to share whatever you want at the best time for you. Have a great end of the year!!

choldgraf commented 2 years ago

Thanks everybody for taking another step forward for 2i2c and OpenScapes :-)

jules32 commented 2 years ago

Yes, I just wanted to echo our thanks; we appreciate collaborating with you!

On Mon, Dec 13, 2021 at 2:48 PM Chris Holdgraf @.***> wrote:

Thanks everybody for taking another step forward for 2i2c and OpenScapes :-)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/2i2c-org/infrastructure/issues/870#issuecomment-992981819, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABM6ORLWT4MP6ZOW572QZQTUQZZ3JANCNFSM5JKZO3DA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

--

Julia Stewart Lowndes, PhD Openscapes Co-Director National Center for Ecological Analysis and Synthesis (NCEAS https://www.nceas.ucsb.edu/) University of California, Santa Barbara (UCSB) Openscapes https://openscapes.org • Ocean Health Index http://ohi-science.org/ • Mozilla Fellow website http://jules32.github.io/ • github https://github.com/jules32 • twitter https://twitter.com/juliesquid

2i2c-org / infrastructure