Demo day 2022-10-27

[mrocklin]

We'll be organizing a demo day (see https://github.com/dask/community/issues/280) Thursday next week at 8am US Pacific, 5pm Central European Time.

Context

I'd like to solicit 5-10 minute demos that show off ongoing or lesser known work. I hope to have 3-5 of these during the meeting. Meetings will be recorded and advertised on social. Hopefully this helps to educate folks on some of the great work people are up to.

If you're interested please respond to this issue with a brief (a couple sentences) description. If you have colleagues who you think should be interested please let them know.

Agenda

1. "Matplotlib is popular": @mrocklin
2. "Task Queuing / Constant Memory Scheduling" : @fjetter or @gjoseph92
3. "Kubernetes stuff" : @jacobtomlinson
4. "Apache Beam on Dask": @alxmrs
5. "Prometheus" : @ntabris or @dchudz
6. ... (room for one more!)

If there is time at the end I'll show some simple work with the arXiv dataset, which was fun

Meeting Invite

Matthew Rocklin is inviting you to a scheduled Zoom meeting.

Topic: Dask Demo Day Time: Oct 27, 2022 10:00 AM Central Time (US and Canada)

Join Zoom Meeting https://us06web.zoom.us/j/85409976995?pwd=L1ZrbG00OVkzZ2RvV0R3bTdFQlRDdz09

Meeting ID: 854 0997 6995 Passcode: 130413 One tap mobile +13462487799,,85409976995# US (Houston) +16694449171,,85409976995# US

Dial by your location +1 346 248 7799 US (Houston) +1 669 444 9171 US +1 669 900 6833 US (San Jose) +1 719 359 4580 US +1 253 215 8782 US (Tacoma) +1 564 217 2000 US +1 646 931 3860 US +1 929 436 2866 US (New York) +1 301 715 8592 US (Washington DC) +1 309 205 3325 US +1 312 626 6799 US (Chicago) +1 360 209 5623 US +1 386 347 5053 US Meeting ID: 854 0997 6995 Find your local number: https://us06web.zoom.us/u/kcshPwvXdS

[dchudz]

Folks from Coiled could demo some fairly basic (boring?) Coiled functionality, e.g.:

• a setup CLI we have that gets a user's AWS or GCP account prepared for Coiled to work properly
• a "cluster details" page showing some of the infrastructure, and the history of cluster-related events (launching instances, etc).

For something that's not yet in the Coiled product, maybe we could demo some stuff we've been playing around with collecting metrics from the Prometheus endpoint and plotting them in Graphana.

[jrbourbeau]

@dchudz I suspect there will be folks who haven't used Coiled before. Something like a "Getting started with Coiled in 5 minutes" demo could be informative for folks to see. We could walk through (1) using the coiled CLI to connect to a cloud provider account, (2) spin up a Coiled cluster (using local environment syncing), (3) walk through several of the niceties one gets with Coiled (e.g. logs, package details, analytics). Thoughts?

@jacobtomlinson, you've usually always got interesting things to show, is there something you're working on that you'd like to demo?

[jrbourbeau]

Whoops, seeing @jacobtomlinson proposed demo-ing the Dask Kubernetes Operator over in https://github.com/dask/community/issues/280#issuecomment-1277220590

[fjetter]

Task queuing / Constant memory scheduling

See also https://github.com/dask/distributed/discussions/7128

A new configuration parameter / feature toggle was introduced that significantly impacts scheduling behavior and is typically expected to deliver constant memory scheduling and therefore reduce spilling, increase stability and in some cases even drastically improves runtime.

During the demo we can briefly discuss tradeoffs between show how to enable this and discuss a couple of example workloads where this matters

cc @gjoseph92

[dchudz]

"Getting started with Coiled in 5 minutes" demo ... Thoughts?

Sounds great. I'd be happy to either do it, or invite someone else from our Platform team to give the demo.

[jacobtomlinson]

I'm in no rush if there are other topics that folks also want to demo 🙂

[mrocklin]

Folks should list things. If there is a surplus then I'll arbitrarily select a set and push others to next month.

[jacobtomlinson]

The Dask Kubernetes Operator would be a nice thing to show.

[mrocklin]

We're still low on user-focused demos. Does anyone know anyone who would be good here?

[alxmrs]

I'd be happy to demo https://github.com/apache/beam/pull/22421 when it's ready (fairly soon).

[TomAugspurger]

I'd be happy to do something around "Scalable geospatial data analysis with Dask". That'd be user-focused (albeit on a specific user community).

[gjoseph92]

@TomAugspurger "scalable geospatial analysis with dask" might play nicely with constant-memory scheduling https://github.com/dask/distributed/discussions/7128 if you can update the PC images to the latest version.

[mrocklin]

@alxmrs are you good to go for this Thursday? WIP work is still quite welcome.

[mrocklin]

I've placed a tentative agenda and Zoom invite in the opening comment. I'll tweet this out later today.

@TomAugspurger I think that you're still the guardian of the calendar. Thoughts on how to make this show up more easily in folks' calendars?

[alxmrs]

Yes! I should be good to go -- I'm happy to present WIP work.

[TomAugspurger]

There should be an event for Thursday the 27th at 10:00 US/Central. I see it on the embedded calendar at https://docs.dask.org/en/latest/support.html. Did it show up for others who are already subscribed.

I have conflicts during that time so I won't be able to attend. I can update the agenda on the calendar as it solidifies though.

[gjoseph92]

I also won't be able to attend that time. Maybe someone else can talk about queuing?

[jrbourbeau]

Did it show up for others who are already subscribed

I see the event 👍

[mrocklin]

I also won't be able to attend that time. Maybe someone else can talk about queuing?

@fjetter ?

[mrocklin]

This has been advertised on Twitter: https://twitter.com/dask_dev/status/1584935939213705216

[ljstrnadiii]

Exciting! I am hoping to quietly sit in on this.

[TomAugspurger]

Just to note, since I'm listed as presenting, I unfortunately have another commitment that overlaps with the demo day time. I'm looking forward to the recording and will join in on the next one! I'll edit the original post to remove myself.

[mrocklin]

Alas! Oh well.

See you next month?

[mmccarty]

Sorry I missed it. Had a conflict. Is there a recording?

[gtmaskall]

I'm always up for some scalable geoscience, so will have to look out for your demo @TomAugspurger .

[mrocklin]

Finished processing, but I should trim things off of the beginning and end. Hopefully we'll get the recording out tomorrow.

[mrocklin]

https://youtu.be/VlTgcLqb1DQ

[ntabris]

https://youtu.be/VlTgcLqb1DQ

YouTube tells me the video is private (so I can't see it).

[mrocklin]

Sorry! It should be open now.

[mrocklin]

Thanks all who participating for making this a success. Closing.