MetaRossi
commented
9 months ago Should be fixed in the attached branch. Changed console output to only use instance names.
Closed MetaRossi closed 9 months ago
MetaRossi
commented
9 months ago Should be fixed in the attached branch. Changed console output to only use instance names.
Need to move the new line behavior into _render_to_console because the newlines are misbehaving often.
Console: | => python main.py __config/config.toml default 2024-01-21 15:45:20.096047 - Logging to: 20240121_154520.log 2024-01-21 15:45:20.096899 - Regions: {'us-east-1': 'Virginia, USA', 'us-west-1': 'California, USA', 'us-west-2': 'Arizona, USA', 'us-west-3': 'Utah, USA', 'us-midwest-1': 'Illinois, USA', 'us-south-1': 'Texas, USA', 'europe-central-1': 'Germany', 'me-west-1': 'Israel', 'asia-south-1': 'India', 'asia-northeast-1': 'Osaka, Japan', 'asia-northeast-2': 'Tokyo, Japan'} 2024-01-21 15:45:20.414596 - Instance names: {'gpu_1x_h100_pcie', 'gpu_2x_a100', 'gpu_1x_a10', 'gpu_2x_a6000', 'gpu_4x_a100', 'gpu_4x_a6000', 'gpu_1x_rtx6000', 'gpu_1x_a100_sxm4', 'gpu_1x_a100', 'gpu_1x_a6000', 'gpu_8x_h100_sxm5', 'gpu_8x_v100', 'gpu_8x_a100', 'gpu_8x_a100_80gb_sxm4'} 2024-01-21 15:59:31.204904 - No instances available. Started at: 2024-01-21 15:45:20.095735, Duration since start: 0:14:11.109169 2024-01-21 15:59:53.176988 - Available Instances: {'gpu_1x_rtx6000'}, Availability Duration: 0:00:20.411604 2024-01-21 17:02:23.367632 - No instances available. Last available at: 2024-01-21 15:59:54.471173, Duration since last available: 1:02:28.896459 2024-01-21 17:02:57.619382 - Available Instances: {'gpu_4x_a6000'}, Availability Duration: 0:00:33.219084 2024-01-21 17:04:42.780924 - Available Instances: {'gpu_4x_a6000', 'gpu_1x_a10'}, Availability Duration: 0:02:18.380626 2024-01-21 17:10:29.237646 - Available Instances: {'gpu_4x_a6000'}, Availability Duration: 0:08:04.837348 2024-01-21 17:27:17.324543 - No instances available. Last available at: 2024-01-21 17:10:30.283295, Duration since last available: 0:16:47.041248 2024-01-21 17:29:09.140057 - Available Instances: {'gpu_1x_a10'}, Availability Duration: 0:01:50.694443 2024-01-21 17:58:21.595960 - No instances available. Last available at: 2024-01-21 17:29:10.187484, Duration since last available: 0:29:11.408476 2024-01-21 18:03:50.111851 - Available Instances: {'gpu_4x_a6000'}, Availability Duration: 0:05:27.288726 2024-01-21 19:08:30.804872 - No instances available. Last available at: 2024-01-21 18:03:51.423623, Duration since last available: 1:04:39.381249 2024-01-21 19:21:35.263863 - Available Instances: {'gpu_4x_a6000'}, Availability Duration: 0:13:03.378347 2024-01-21 19:58:21.885595 - No instances available. Last available at: 2024-01-21 19:21:36.330766, Duration since last available: 0:36:45.554829 2024-01-21 19:59:14.887312 - Available Instances: {'gpu_1x_a10'}, Availability Duration: 0:00:51.984271 2024-01-21 20:05:59.901204 - Available Instances: {'gpu_1x_a10'}, Availability Duration: 0:07:36.998163 2024-01-21 20:06:18.705143 - Available Instances: {'gpu_1x_a10'}, Availability Duration: 0:07:55.802102
Logs: 2024-01-21 15:45:20.096 - INFO - Starting job at: 2024-01-21T15:45:20 2024-01-21 15:45:20.096 - INFO - Target sleep Interval (ms): 1100 2024-01-21 15:45:20.096 - INFO - Regions: {'us-east-1': 'Virginia, USA', 'us-west-1': 'California, USA', 'us-west-2': 'Arizona, USA', 'us-west-3': 'Utah, USA', 'us-midwest-1': 'Illinois, USA', 'us-south-1': 'Texas, USA', 'europe-central-1': 'Germany', 'me-west-1': 'Israel', 'asia-south-1': 'India', 'asia-northeast-1': 'Osaka, Japan', 'asia-northeast-2': 'Tokyo, Japan'} 2024-01-21 15:45:20.414 - INFO - Instance names: {'gpu_1x_h100_pcie', 'gpu_2x_a100', 'gpu_1x_a10', 'gpu_2x_a6000', 'gpu_4x_a100', 'gpu_4x_a6000', 'gpu_1x_rtx6000', 'gpu_1x_a100_sxm4', 'gpu_1x_a100', 'gpu_1x_a6000', 'gpu_8x_h100_sxm5', 'gpu_8x_v100', 'gpu_8x_a100', 'gpu_8x_a100_80gb_sxm4'} 2024-01-21 15:59:32.767 - INFO - Instance Type: gpu_1x_rtx6000, Region: us-south-1 - Status: Available - Start: 2024-01-21 15:59:32.765384 2024-01-21 15:59:54.472 - INFO - Instance Type: gpu_1x_rtx6000, Region: us-south-1 - Status: Unavailable - Start: 2024-01-21 15:59:32.765384 - End: 2024-01-21 15:59:53.175340 - Duration: 0:00:20.409956 2024-01-21 17:02:24.401 - INFO - Instance Type: gpu_4x_a6000, Region: us-south-1 - Status: Available - Start: 2024-01-21 17:02:24.400298 2024-01-21 17:02:58.726 - INFO - Instance Type: gpu_1x_a10, Region: us-west-1 - Status: Available - Start: 2024-01-21 17:02:58.725934 2024-01-21 17:04:43.883 - INFO - Instance Type: gpu_1x_a10, Region: us-west-1 - Status: Unavailable - Start: 2024-01-21 17:02:58.725934 - End: 2024-01-21 17:04:42.780061 - Duration: 0:01:44.054127 2024-01-21 17:10:30.284 - INFO - Instance Type: gpu_4x_a6000, Region: us-south-1 - Status: Unavailable - Start: 2024-01-21 17:02:24.400298 - End: 2024-01-21 17:10:29.236025 - Duration: 0:08:04.835727 2024-01-21 17:27:18.447 - INFO - Instance Type: gpu_1x_a10, Region: us-east-1 - Status: Available - Start: 2024-01-21 17:27:18.445614 2024-01-21 17:29:10.189 - INFO - Instance Type: gpu_1x_a10, Region: us-east-1 - Status: Unavailable - Start: 2024-01-21 17:27:18.445614 - End: 2024-01-21 17:29:09.138420 - Duration: 0:01:50.692806 2024-01-21 17:58:22.824 - INFO - Instance Type: gpu_4x_a6000, Region: us-south-1 - Status: Available - Start: 2024-01-21 17:58:22.823125 2024-01-21 18:03:51.425 - INFO - Instance Type: gpu_4x_a6000, Region: us-south-1 - Status: Unavailable - Start: 2024-01-21 17:58:22.823125 - End: 2024-01-21 18:03:50.110174 - Duration: 0:05:27.287049 2024-01-21 19:08:31.887 - INFO - Instance Type: gpu_4x_a6000, Region: us-south-1 - Status: Available - Start: 2024-01-21 19:08:31.885516 2024-01-21 19:21:36.332 - INFO - Instance Type: gpu_4x_a6000, Region: us-south-1 - Status: Unavailable - Start: 2024-01-21 19:08:31.885516 - End: 2024-01-21 19:21:35.262502 - Duration: 0:13:03.376986 2024-01-21 19:58:22.904 - INFO - Instance Type: gpu_1x_a10, Region: us-west-1 - Status: Available - Start: 2024-01-21 19:58:22.903041 2024-01-21 19:59:15.978 - INFO - Instance Type: gpu_1x_a10, Region: us-east-1 - Status: Available - Start: 2024-01-21 19:59:15.977224 2024-01-21 20:06:00.995 - INFO - Instance Type: gpu_1x_a10, Region: us-east-1 - Status: Unavailable - Start: 2024-01-21 19:59:15.977224 - End: 2024-01-21 20:05:59.899878 - Duration: 0:06:43.922654 2024-01-21 20:06:44.047 - INFO - Instance Type: gpu_1x_a10, Region: us-west-1 - Status: Unavailable - Start: 2024-01-21 19:58:22.903041 - End: 2024-01-21 20:06:42.956122 - Duration: 0:08:20.053081