search

mrocklin / dask-tutorial

BSD 3-Clause "New" or "Revised" License
17 stars 9 forks source link

To package_sync or not to package_sync #1

Open mrocklin opened 1 year ago

mrocklin commented 1 year ago

@dchudz says

If we don't absolutely need it, I'd recommend (and Nat concurs) not using package sync for the tutorial.

Maybe we can mention it as a new option people can try. But we want the basic stuff to just work reliably, which package sync won't.

mrocklin commented 1 year ago

@dchudz you tried launching 100 clusters, yes? Did that work?

Regardless, I'm going to want to playtest things before this goes in front of users. I hope to do this tomorrow after content settles a little bit today.

dchudz commented 1 year ago

I did try launching 100 clusters. (Not w/ package sync.) It worked.

In [18]: async def return_cluster(i):
    ...:     try:
    ...:         return await coiled.Cluster(n_workers=25, asynchronous=True, show_widget=False)
    ...:     except Exception as e:
    ...:         import logging
    ...:         logging.exception("hi")
    ...:

In [19]: results = await asyncio.gather(*[return_cluster(i) for i in range(100)])

In [20]: results
Out[20]:
[ClusterBeta(coiled-training-940f3976-2, 'tls://44.195.41.173:8786', workers=25, threads=100, memory=373.29 GiB),
 ClusterBeta(coiled-training-b2422b06-8, 'tls://18.215.15.74:8786', workers=25, threads=100, memory=373.22 GiB),
 ClusterBeta(coiled-training-0274e073-5, 'tls://3.227.21.126:8786', workers=25, threads=100, memory=373.25 GiB),
 ClusterBeta(coiled-training-95fe27d2-6, 'tls://18.215.62.120:8786', workers=25, threads=100, memory=373.25 GiB),
 ClusterBeta(coiled-training-6d064056-c, 'tls://3.234.178.80:8786', workers=25, threads=100, memory=373.24 GiB),
 ClusterBeta(coiled-training-802dad73-6, 'tls://3.208.92.51:8786', workers=25, threads=100, memory=373.23 GiB),
 ClusterBeta(coiled-training-82257be1-f, 'tls://18.207.106.112:8786', workers=25, threads=100, memory=373.30 GiB),
 ClusterBeta(coiled-training-e472ff77-1, 'tls://3.239.183.208:8786', workers=25, threads=100, memory=373.25 GiB),
 ClusterBeta(coiled-training-51aa7bee-b, 'tls://34.200.244.222:8786', workers=25, threads=100, memory=373.27 GiB),
 ClusterBeta(coiled-training-c238d22a-1, 'tls://3.218.250.148:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-d5374e6a-c, 'tls://44.196.58.18:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-8d0e1762-8, 'tls://3.238.154.194:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-2eee1c5d-2, 'tls://35.170.72.194:8786', workers=25, threads=100, memory=373.27 GiB),
 ClusterBeta(coiled-training-0f915dba-9, 'tls://3.237.196.176:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-4806213b-f, 'tls://35.175.116.169:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-d0c2deb6-e, 'tls://3.228.2.193:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-9f48b101-b, 'tls://3.239.80.255:8786', workers=24, threads=96, memory=358.31 GiB),
 ClusterBeta(coiled-training-7fc10b17-b, 'tls://3.83.36.251:8786', workers=25, threads=100, memory=373.25 GiB),
 ClusterBeta(coiled-training-b0cdc773-1, 'tls://44.199.236.185:8786', workers=25, threads=100, memory=373.23 GiB),
 ClusterBeta(coiled-training-f1d6b3ee-2, 'tls://54.85.13.83:8786', workers=25, threads=100, memory=373.25 GiB),
 ClusterBeta(coiled-training-114235ed-1, 'tls://3.239.123.83:8786', workers=25, threads=100, memory=373.27 GiB),
 ClusterBeta(coiled-training-c19c4267-6, 'tls://44.192.7.72:8786', workers=25, threads=100, memory=373.24 GiB),
 ClusterBeta(coiled-training-465b3ef9-b, 'tls://44.204.199.149:8786', workers=25, threads=100, memory=373.24 GiB),
 ClusterBeta(coiled-training-746bce64-d, 'tls://3.237.186.79:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-f4b7d549-a, 'tls://18.207.111.32:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-71ce4e9f-c, 'tls://3.239.228.17:8786', workers=25, threads=100, memory=373.19 GiB),
 ClusterBeta(coiled-training-38644b58-c, 'tls://18.205.2.109:8786', workers=25, threads=100, memory=373.28 GiB),
 ClusterBeta(coiled-training-10837707-1, 'tls://44.204.100.239:8786', workers=25, threads=100, memory=373.27 GiB),
 ClusterBeta(coiled-training-ca972491-a, 'tls://3.237.185.110:8786', workers=25, threads=100, memory=373.27 GiB),
 ClusterBeta(coiled-training-cef1e2b4-5, 'tls://44.198.190.36:8786', workers=25, threads=100, memory=373.22 GiB),
 ClusterBeta(coiled-training-ae638fa5-e, 'tls://34.200.247.118:8786', workers=25, threads=100, memory=373.25 GiB),
 ClusterBeta(coiled-training-7631d59f-8, 'tls://3.234.178.254:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-6f12b3a2-0, 'tls://3.231.161.110:8786', workers=25, threads=100, memory=373.24 GiB),
 ClusterBeta(coiled-training-62a620b4-c, 'tls://3.238.162.34:8786', workers=25, threads=100, memory=373.23 GiB),
 ClusterBeta(coiled-training-258bc0ab-f, 'tls://44.203.238.76:8786', workers=25, threads=100, memory=373.27 GiB),
 ClusterBeta(coiled-training-d294a236-8, 'tls://44.203.223.61:8786', workers=25, threads=100, memory=373.25 GiB),
 ClusterBeta(coiled-training-9ff5bdfc-d, 'tls://44.204.245.5:8786', workers=25, threads=100, memory=373.22 GiB),
 ClusterBeta(coiled-training-622b1300-b, 'tls://44.211.59.32:8786', workers=25, threads=100, memory=373.28 GiB),
 ClusterBeta(coiled-training-c004ea06-0, 'tls://3.215.185.19:8786', workers=25, threads=100, memory=373.24 GiB),
 ClusterBeta(coiled-training-6c175e27-1, 'tls://34.200.254.145:8786', workers=25, threads=100, memory=373.27 GiB),
 ClusterBeta(coiled-training-c096daa8-b, 'tls://44.195.34.93:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-505167a5-f, 'tls://44.193.17.254:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-297a78f4-f, 'tls://3.238.189.78:8786', workers=25, threads=100, memory=373.23 GiB),
 ClusterBeta(coiled-training-58d33778-6, 'tls://3.238.145.6:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-ae1135a6-a, 'tls://44.212.49.175:8786', workers=0, threads=0, memory=0 B),
 ClusterBeta(coiled-training-b70468aa-6, 'tls://44.200.54.224:8786', workers=25, threads=100, memory=373.27 GiB),
 ClusterBeta(coiled-training-7dce29df-5, 'tls://35.172.233.171:8786', workers=25, threads=100, memory=373.21 GiB),
 ClusterBeta(coiled-training-7b2fccb3-a, 'tls://3.221.170.84:8786', workers=25, threads=100, memory=373.30 GiB),
 ClusterBeta(coiled-training-d2ba2e75-3, 'tls://44.200.218.62:8786', workers=25, threads=100, memory=373.28 GiB),
 ClusterBeta(coiled-training-68968f54-f, 'tls://44.195.80.143:8786', workers=25, threads=100, memory=373.30 GiB),
 ClusterBeta(coiled-training-abbad373-8, 'tls://44.195.69.45:8786', workers=25, threads=100, memory=373.25 GiB),
 ClusterBeta(coiled-training-8815b650-a, 'tls://3.89.81.19:8786', workers=25, threads=100, memory=373.21 GiB),
 ClusterBeta(coiled-training-989d87a3-9, 'tls://3.220.230.110:8786', workers=25, threads=100, memory=373.27 GiB),
 ClusterBeta(coiled-training-de63d16a-c, 'tls://44.201.12.138:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-acef94b5-9, 'tls://44.203.6.92:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-07fefe41-5, 'tls://3.223.141.164:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-d04ef261-1, 'tls://3.238.159.6:8786', workers=25, threads=100, memory=373.28 GiB),
 ClusterBeta(coiled-training-4be8d05e-b, 'tls://44.199.214.9:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-7b55743d-f, 'tls://3.218.252.164:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-8d726e17-e, 'tls://44.199.235.119:8786', workers=25, threads=100, memory=373.23 GiB),
 ClusterBeta(coiled-training-4d317179-5, 'tls://3.237.78.145:8786', workers=25, threads=100, memory=373.29 GiB),
 ClusterBeta(coiled-training-f0d5dc57-3, 'tls://3.239.163.16:8786', workers=25, threads=100, memory=373.24 GiB),
 ClusterBeta(coiled-training-d2b1d287-6, 'tls://44.192.56.210:8786', workers=25, threads=100, memory=373.28 GiB),
 ClusterBeta(coiled-training-8d1fa655-5, 'tls://3.230.163.200:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-68e384d2-6, 'tls://44.192.118.7:8786', workers=25, threads=100, memory=373.21 GiB),
 ClusterBeta(coiled-training-50b1a801-8, 'tls://3.236.164.85:8786', workers=25, threads=100, memory=373.29 GiB),
 ClusterBeta(coiled-training-006fb1cd-7, 'tls://54.236.59.226:8786', workers=25, threads=100, memory=373.25 GiB),
 ClusterBeta(coiled-training-c0826815-e, 'tls://3.228.22.3:8786', workers=25, threads=100, memory=373.23 GiB),
 ClusterBeta(coiled-training-124ee8dc-c, 'tls://34.200.246.240:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-f7216c9c-c, 'tls://34.237.176.200:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-cd79d5bc-8, 'tls://3.94.150.133:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-df8dec38-a, 'tls://44.204.133.239:8786', workers=25, threads=100, memory=373.27 GiB),
 ClusterBeta(coiled-training-cbcd98b2-8, 'tls://3.230.0.189:8786', workers=25, threads=100, memory=373.27 GiB),
 ClusterBeta(coiled-training-edcaa47d-0, 'tls://44.193.24.172:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-b6bca3d2-2, 'tls://54.236.35.63:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-2677b25a-7, 'tls://44.202.252.57:8786', workers=25, threads=100, memory=373.22 GiB),
 ClusterBeta(coiled-training-58acc46d-9, 'tls://3.239.228.47:8786', workers=25, threads=100, memory=373.23 GiB),
 ClusterBeta(coiled-training-6b0636b0-2, 'tls://44.201.25.185:8786', workers=25, threads=100, memory=373.28 GiB),
 ClusterBeta(coiled-training-3432c188-1, 'tls://44.193.0.136:8786', workers=25, threads=100, memory=373.27 GiB),
 ClusterBeta(coiled-training-e68d395d-6, 'tls://44.199.202.150:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-9ad4a941-a, 'tls://35.170.58.51:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-c9ac00d3-b, 'tls://44.200.156.253:8786', workers=25, threads=100, memory=373.24 GiB),
 ClusterBeta(coiled-training-9c8bc3be-2, 'tls://44.197.176.111:8786', workers=25, threads=100, memory=373.23 GiB),
 ClusterBeta(coiled-training-933f8e50-9, 'tls://3.237.77.139:8786', workers=25, threads=100, memory=373.24 GiB),
 ClusterBeta(coiled-training-991d21fd-0, 'tls://44.212.52.127:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-f16e896d-1, 'tls://44.198.159.40:8786', workers=25, threads=100, memory=373.22 GiB),
 ClusterBeta(coiled-training-ac920b5c-5, 'tls://34.200.248.97:8786', workers=25, threads=100, memory=373.25 GiB),
 ClusterBeta(coiled-training-f9b6493a-e, 'tls://3.235.136.163:8786', workers=25, threads=100, memory=373.27 GiB),
 ClusterBeta(coiled-training-6bf01298-3, 'tls://3.236.164.6:8786', workers=25, threads=100, memory=373.22 GiB),
 ClusterBeta(coiled-training-2b6c8756-e, 'tls://34.234.170.49:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-c55ed040-5, 'tls://44.201.34.98:8786', workers=25, threads=100, memory=373.27 GiB),
 ClusterBeta(coiled-training-9e0415f3-8, 'tls://44.211.120.132:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-01c51732-6, 'tls://3.228.217.71:8786', workers=25, threads=100, memory=373.26 GiB),
 ClusterBeta(coiled-training-49fc64df-0, 'tls://3.235.7.169:8786', workers=25, threads=100, memory=373.30 GiB),
 ClusterBeta(coiled-training-da3c98c3-4, 'tls://44.198.50.147:8786', workers=25, threads=100, memory=373.24 GiB),
 ClusterBeta(coiled-training-f0e2bd2f-3, 'tls://3.237.237.32:8786', workers=25, threads=100, memory=373.23 GiB),
 ClusterBeta(coiled-training-ce23fb6f-0, 'tls://44.201.32.58:8786', workers=25, threads=100, memory=373.27 GiB),
 ClusterBeta(coiled-training-4d7ed89a-c, 'tls://44.200.156.40:8786', workers=25, threads=100, memory=373.21 GiB),
 ClusterBeta(coiled-training-0b27fe0b-3, 'tls://54.160.3.208:8786', workers=25, threads=100, memory=373.23 GiB),
 ClusterBeta(coiled-training-b1b68911-2, 'tls://3.238.4.85:8786', workers=25, threads=100, memory=373.24 GiB)]

In [21]: len(results)
Out[21]: 100

In [22]: for c in results: await c.close()
dchudz commented 1 year ago

This is in the coiled-training account. I'll make a new user for the attendees, but for now the members of that account are:

Happy to add anyone else.