search

skypilot-org / skypilot

SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 12+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.
https://skypilot.readthedocs.io
Apache License 2.0
6.82k stars 513 forks source link

[Core] Submitting 1000 jobs to a cluster #4263

Closed Michaelvll closed 2 weeks ago

Michaelvll commented 2 weeks ago

sky launch -c test-many-jobs --cloud aws --cpus 16 --region us-east-1 and submit jobs:

import subprocess
from multiprocessing.pool import ThreadPool

def run_task(task):
    print(f'Running task {task}')
    subprocess.run(
        f'sky exec test-many-jobs -d "echo hi {task}; sleep 60"', shell=True
    )

with ThreadPool(8) as pool:
    pool.map(run_task, range(1000))

There are some pending jobs directly in FAILED status:

sky job queue ``` 397 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-36-653878 396 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-36-241842 395 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-35-938395 394 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-35-242534 393 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-34-388982 392 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-33-484291 391 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-33-343155 390 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-30-894991 389 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-30-271563 388 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-29-496423 387 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-29-210465 386 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-29-456516 385 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-28-390933 384 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-27-188816 383 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-27-098285 382 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-24-541945 381 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-24-229395 380 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-23-402925 379 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-23-071930 378 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-23-145018 377 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-22-539985 376 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-21-025510 375 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-20-997171 374 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-18-671169 373 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-17-896233 372 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-17-097363 371 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-16-872947 370 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-17-040533 369 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-16-321652 368 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-14-689782 367 sky-cmd 47 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-14-612180 366 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-12-561881 365 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-11-489980 364 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-11-074665 363 sky-cmd 48 mins ago - - 1x[CPU:1+] FAILED ~/sky_logs/sky-2024-11-05-08-02-10-755273 362 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-10-400760 361 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-10-216905 360 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-08-709074 359 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-08-474316 358 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-06-822676 357 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-04-858041 356 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-04-921609 355 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-04-662728 354 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-03-636612 353 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-03-776459 352 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-01-931069 351 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-01-599564 350 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-02-00-654291 349 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-57-863715 348 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-57-476013 347 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-57-198596 346 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-56-759002 345 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-56-728631 344 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-55-006414 343 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-55-220696 342 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-54-064863 341 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-52-218586 340 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-51-040224 339 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-50-935741 338 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-50-288325 337 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-50-478773 336 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-49-099053 335 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-48-550194 334 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-48-217828 333 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-46-255106 332 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-44-558566 331 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-44-467170 330 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-43-892493 329 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-43-716273 328 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-42-848585 327 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-41-999266 326 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-42-235626 325 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-40-082665 324 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-38-614665 323 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-38-487463 322 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-38-032037 321 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-37-467258 320 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-36-202257 319 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-35-780035 318 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-35-966845 317 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-34-124504 316 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-32-308747 315 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-32-096878 314 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-31-733177 313 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-31-403574 312 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-30-537735 311 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-29-387918 310 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-29-582632 309 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-28-219573 308 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-26-278180 307 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-25-500778 306 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-25-306906 305 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-25-386088 304 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-24-443275 303 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-23-497353 302 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-23-165759 301 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-21-960793 300 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-20-041373 299 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-19-224309 298 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-19-441115 297 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-18-773858 296 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-18-322541 295 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-17-440484 294 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-16-785337 293 sky-cmd 48 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-15-770487 292 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-13-659089 291 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-13-018024 290 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-12-633196 289 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-12-947045 288 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-12-198609 287 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-11-250759 286 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-11-006672 285 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-09-793409 284 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-07-327897 283 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-06-646837 282 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-05-628923 281 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-05-771241 280 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-05-473983 279 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-04-536272 278 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-04-354728 277 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-03-428077 276 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-01-00-486087 275 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-59-334567 274 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-59-151814 273 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-58-950464 272 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-58-310687 271 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-57-206339 270 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-56-941714 269 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-56-535521 268 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-54-329514 267 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-53-071108 266 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-52-941368 265 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-52-881440 264 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-52-200498 263 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-50-945052 262 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-50-920789 261 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-50-449763 260 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-48-121061 259 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-46-976192 258 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-46-598611 257 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-46-553419 256 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-46-181387 255 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-45-163721 254 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-44-567216 253 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-44-087470 252 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-42-091929 251 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-41-113196 250 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-40-579688 249 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-40-087820 248 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-40-248555 247 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-38-861945 246 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-38-294149 245 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-38-258045 244 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-35-599548 243 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-34-534367 242 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-34-526143 241 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-34-189236 240 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-33-647800 239 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-32-618731 238 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-31-954582 237 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-31-920088 236 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-29-534488 235 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-28-269088 234 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-27-661199 233 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-27-713331 232 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-27-701794 231 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-26-492482 230 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-25-519140 229 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-25-408865 228 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-23-518706 227 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-22-218402 226 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-21-805968 225 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-21-519246 224 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-21-255874 223 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-20-310804 222 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-19-435243 221 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-19-274018 220 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-17-704161 219 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-16-072318 218 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-15-431482 217 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-15-276972 216 sky-cmd 49 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-14-791237 215 sky-cmd 50 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-14-053555 214 sky-cmd 50 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-13-367657 213 sky-cmd 50 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-13-333595 212 sky-cmd 50 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-11-804236 211 sky-cmd 50 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-09-733314 210 sky-cmd 50 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-09-401792 209 sky-cmd 50 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-09-036496 208 sky-cmd 50 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-08-437905 207 sky-cmd 50 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-07-792564 206 sky-cmd 50 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-07-012027 205 sky-cmd 50 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-06-712460 204 sky-cmd 50 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-05-829213 203 sky-cmd 50 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-02-724472 202 sky-cmd 50 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-01-926897 201 sky-cmd 50 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-01-949462 200 sky-cmd 50 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-01-155412 199 sky-cmd 50 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-08-00-00-635174 198 sky-cmd 50 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-07-59-59-770938 197 sky-cmd 50 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-07-59-59-791625 196 sky-cmd 50 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-07-59-58-527490 195 sky-cmd 50 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-07-59-56-311420 194 sky-cmd 50 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-07-59-55-741381 193 sky-cmd 50 mins ago - - 1x[CPU:1+] PENDING ~/sky_logs/sky-2024-11-05-07-59-55-649851 192 sky-cmd 50 mins ago 1 min ago 1m 3s 1x[CPU:1+] RUNNING ~/sky_logs/sky-2024-11-05-07-59-54-725529 191 sky-cmd 50 mins ago 1 min ago 1m 6s 1x[CPU:1+] RUNNING ~/sky_logs/sky-2024-11-05-07-59-54-654723 190 sky-cmd 50 mins ago 1 min ago 1m 9s 1x[CPU:1+] RUNNING ~/sky_logs/sky-2024-11-05-07-59-53-962026 189 sky-cmd 50 mins ago 1 min ago 1m 12s 1x[CPU:1+] RUNNING ~/sky_logs/sky-2024-11-05-07-59-53-403442 ```

Version & Commit info:

Michaelvll commented 2 weeks ago

https://gist.github.com/Michaelvll/9325ff98e6bd2b454dfd661fb9c0ebab