search

coiled / feedback

A place to provide Coiled feedback
14 stars 3 forks source link

Error getting STS token from client AWS session: Could not connect to the endpoint URL #195

Closed tonycombocurve closed 1 year ago

tonycombocurve commented 1 year ago

When creating a cluster on GCP, we see this error in our logs:

2022-08-29 10:11:43 Error getting STS token from client AWS session: Could not connect to the endpoint URL: "http://169.254.169.254/latest/api/token"

Since our entire stack runs on GCP, I don't think that error message makes sense.

Environment, optional

Diagnostics, optional

{'health_check': {'status': 'OK'}, 'local_versions': {'operating_system': 'Windows-10-10.0.19044-SP0', 'python': LooseVersion ('3.9.13'), 'pip': '22.0.4', 'conda': 'None', 'coiled': '0.2.26', 'dask': '2022.8.1', 'distributed': '2022.8.1'}, 'coiled_configuration': {'coiled': {'account': 'tony-piazza', 'server': 'https://cloud.coiled.io', 'token': 'hidden', 'user': 'tony-piazza', 'backend-options': None, 'no-minimum-version-check': False, 'protocol': 'tls', 'scheduler-options': {}, 'worker-options': {}, 'wait-for-workers': 0.3, 'software': None, 'worker': {'cpu': None, 'gpu': None, 'memory': None, 'class': None, 'vm-types': None, 'gpu-types': None}, 'scheduler': {'cpu': None, 'memory': None, 'class': None, 'vm-types': None}, 'name': None, 'shutdown-on-close': True, 'analytics': {'computation': {'interval': '10s', 'code': {'transmit': True}}, 'profile': {'transmit': False, 'interval': '60s'}, 'events': {'interval': '10s', 'allow': ['invalid-worker-transition', 'invalid-task-states', 'worker-fail-hard']}, 'idle': {'timeout': None}}}, 'temporary-directory': None, 'visualization': {'engine': None}, 'tokenize': {'ensure-deterministic': False}, 'dataframe': {'shuffle-compression': None, 'parquet': {'metadata-task-size-local': 512, 'metadata-task-size-remote': 16}}, 'array': {'svg': {'size': 120}, 'slicing': {'split-large-chunks': None}}, 'optimization': {'annotations': {'fuse': True}, 'fuse': {'active': None, 'ave-width': 1, 'max-width': None, 'max-height': inf, 'max-depth-new-edges': None, 'subgraphs': None, 'rename-keys': True}}, 'distributed': {'version': 2, 'scheduler': {'allowed-failures': 3, 'bandwidth': 100000000, 'blocked-handlers': [], 'contact-address': None, 'default-data-size': '1kiB', 'events-cleanup-delay': '1h', 'idle-timeout': None, 'transition-log-length': 100000, 'events-log-length': 100000, 'work-stealing': True, 'work-stealing-interval': '100ms', 'worker-ttl': '5 minutes', 'pickle': True, 'preload': [], 'preload-argv': [], 'unknown-task-duration': '500ms', 'default-task-durations': {'rechunk-split': '1us', 'split-shuffle': '1us'}, 'validate': False, 'dashboard': {'status': {'task-stream-length': 1000}, 'tasks': {'task-stream-length': 100000}, 'tls': {'ca-file': None, 'key': None, 'cert': None}, 'bokeh-application': {'allow_websocket_origin': ['*'], 'keep_alive_milliseconds': 500, 'check_unused_sessions_milliseconds': 500}}, 'locks': {'lease-validation-interval': '10s', 'lease-timeout': '30s'}, 'http': {'routes': ['distributed.http.scheduler.prometheus', 'distributed.http.scheduler.info', 'distributed.http.scheduler.json', 'distributed.http.health', 'distributed.http.proxy', 'distributed.http.statics']}, 'allowed-imports': ['dask', 'distributed'], 'active-memory-manager': {'start': False, 'interval': '2s', 'policies': [{'class': 'distributed.active_memory_manager.ReduceReplicas'}]}}, 'worker': {'blocked-handlers': [], 'multiprocessing-method': 'spawn', 'use-file-locking': True, 'connections': {'outgoing': 50, 'incoming': 10}, 'preload': [], 'preload-argv': [], 'daemon': True, 'validate': False, 'resources': {}, 'lifetime': {'duration': None, 'stagger': '0 seconds', 'restart': False}, 'profile': {'enabled': True, 'interval': '10ms', 'cycle': '1000ms', 'low-level': False}, 'memory': {'recent-to-old-time': '30s', 'rebalance': {'measure': 'optimistic', 'sender-min': 0.3, 'recipient-max': 0.6, 'sender-recipient-gap': 0.1}, 'target': 0.6, 'spill': 0.7, 'pause': 0.8, 'terminate': 0.95, 'max-spill': False, 'monitor-interval': '100ms'}, 'http': {'routes': ['distributed.http.worker.prometheus', 'distributed.http.health', 'distributed.http.statics']}}, 'nanny': {'preload': [], 'preload-argv': [], 'environ': {}, 'pre-spawn-environ': {'MALLOC_TRIMTHRESHOLD': 65536, 'OMP_NUM_THREADS': 1, 'MKL_NUM_THREADS': 1}}, 'client': {'heartbeat': '5s', 'scheduler-info-interval': '2s', 'security-loader': None, 'preload': [], 'preload-argv': []}, 'deploy': {'lost-worker-timeout': '15s', 'cluster-repr-interval': '500ms'}, 'adaptive': {'interval': '1s', 'target-duration': '5s', 'minimum': 0, 'maximum': inf, 'wait-count': 3}, 'comm': {'retry': {'count': 0, 'delay': {'min': '1s', 'max': '20s'}}, 'compression': 'auto', 'shard': '64MiB', 'offload': '10MiB', 'default-scheme': 'tcp', 'socket-backlog': 2048, 'recent-messages-log-length': 0, 'ucx': {'cuda-copy': None, 'tcp': None, 'nvlink': None, 'infiniband': None, 'rdmacm': None, 'create-cuda-context': None}, 'zstd': {'level': 3, 'threads': 0}, 'timeouts': {'connect': '30s', 'tcp': '30s'}, 'require-encryption': None, 'tls': {'ciphers': None, 'min-version': 1.2, 'max-version': None, 'ca-file': None, 'scheduler': {'cert': None, 'key': None}, 'worker': {'key': None, 'cert': None}, 'client': {'key': None, 'cert': None}}, 'tcp': {'backend': 'tornado'}, 'websockets': {'shard': '8MiB'}}, 'diagnostics': {'nvml': True, 'computations': {'max-history': 100, 'ignore-modules': ['distributed', 'dask', 'xarray', 'cudf', 'cuml', 'prefect', 'xgboost']}, 'erred-tasks': {'max-history': 100}}, 'dashboard': {'link': '{scheme}://{host}:{port}/status', 'export-tool': False, 'graph-max-items': 5000, 'prometheus': {'namespace': 'dask'}}, 'admin': {'tick': {'interval': '20ms', 'limit': '3s', 'cycle': '1s'}, 'max-error-length': 10000, 'log-length': 10000, 'log-format': '%(asctime)s - %(name)s - %(levelname)s - %(message)s', 'pdb-on-err': False, 'system-monitor': {'interval': '500ms', 'disk': True}, 'event-loop': 'tornado'}, 'rmm': {'pool-size': None}}}, 'user_information': {'id': 5808, 'username': 'tony-piazza', 'email': 't.piazza@combocurve.com', 'first_name': 'Tony', 'last_name': 'Piazza', 'membership_set': [{'is_admin': True, 'account': {'can_use_gpus': True, 'backend': 'vm', 'coiled_rate': 100, 'environment_variables': {}, 'members_limit': 10, 'container_registry': {'type': 'gar', 'location': 'us-central1', 'project_id': 'combocurve-try-dask'}, 'active_program': {'usage_percent': 1.199054, 'current_credits': 9880.0946, 'has_quota': True, 'start_date': '2022-08-24T21:44:06.830541Z', 'end_date': None, 'tier': 'free', 'coiled_credit_value': 0.05, 'core_limit': 2147483647, 'core_limit_members': 2147483647, 'node_limit': 2147483647, 'members_limit': 10, 'gpus_limit': 2147483647, 'credit_allotment': 10000.0}, 'options': {'provider_name': 'gcp', 'type': 'gcp_cloudbridge_backend_options', 'gcp_project_name': 'combocurve-try-dask', 'gcp_region_name': 'us-central1', 'gcp_zone_name': 'us-central1-a', 'firewall': None, 'firewall_spec': None, 'network': {}, 'instance_service_account': 'coiled-storage@combocurve-try-dask.iam.gserviceaccount.com'}, 'id': 5911, 'slug': 'tony-piazza', 'name': 'tony-piazza', 'limit': 2147483647, 'user_limit': 10, 'has_quota': True, 'avatar_url': 'https://lh3.googleusercontent.com/a-/AFdZucqdfdH1HcM_KuNaFDpHT5WUMq8z2YSkpT9TjMhY=s96-c', 'private': False, 'early_adopter': False, 'active': True, 'can_use_coiled_hosted': False, 'configured_customer_hosted': True}, 'limit': 1000}], 'has_access_token': False, 'has_ssh_public_key': False, 'avatar_url': 'https://lh3.googleusercontent.com/a-/AFdZucqdfdH1HcM_KuNaFDpHT5WUMq8z2YSkpT9TjMhY=s96-c', 'title': '', 'company': '', 'work_email': '', 'phone_number': '', 'can_contact': True, 'is_staff': False, 'environment_variables': {}, 'has_usable_password': False}, 'core_usage': {'core_limit_user': 1000, 'core_limit_account': 2147483647, 'num_running_cores_user': 0, 'num_running_cores_account': 0, 'num_running_workers_user': 0, 'num_running_tasks_user': 0, 'usage_percent': 1.199054, 'has_quota': True, 'current_credits': '9880.0946', 'credit_allotment': '10000.0000'}, 'notifications': [{'msg': 'VM Shutting Down', 'id': 'c1a12e', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661783455.6132445}, {'msg': 'Successfully pulled docker image', 'id': 'bd231a', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 34, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661783459.6909091}, {'msg': 'Successfully pulled docker image', 'id': 'd17abc', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 30, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661783461.3426049}, {'msg': 'Successfully pulled docker image', 'id': 'aa8109', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 32, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661783470.486325}, {'msg': 'VM Launched', 'id': '996ae2', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784166.4717433}, {'msg': 'VM Launched', 'id': 'c01593', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784178.3642323}, {'msg': 'VM Launched', 'id': 'f8ca31', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784179.1650329}, {'msg': 'VM Launched', 'id': 'b849f9', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784179.4017246}, {'msg': 'VM Launched', 'id': '8263a6', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784179.7796557}, {'msg': 'VM Launched', 'id': '2f2179', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784179.9571013}, {'msg': 'Successfully pulled docker image', 'id': 'b96215', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 34, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784200.9043489}, {'msg': 'Successfully pulled docker image', 'id': 'bc1cbc', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 30, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784209.1521926}, {'msg': 'Successfully pulled docker image', 'id': 'df8c34', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 30, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784209.6704452}, {'msg': 'Successfully pulled docker image', 'id': '4d736c', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 30, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784211.0192275}, {'msg': 'Successfully pulled docker image', 'id': '71c04a', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 31, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784211.7347486}, {'msg': 'Successfully pulled docker image', 'id': '40a358', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 35, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784214.5271463}, {'msg': 'Docker shutdown gracefully', 'id': '5aa2ae', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 1132, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784577.707778}, {'msg': 'Docker shutdown gracefully', 'id': 'edf884', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 366, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784577.7102892}, {'msg': 'Docker shutdown gracefully', 'id': '0a513c', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 1116, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784577.85049}, {'msg': 'Docker shutdown gracefully', 'id': '980ad5', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 366, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784577.8559282}, {'msg': 'Docker shutdown gracefully', 'id': 'f89e35', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 1132, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784577.9194896}, {'msg': 'Docker shutdown gracefully', 'id': 'e76518', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 368, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784577.927399}, {'msg': 'Docker shutdown gracefully', 'id': '374b38', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 1131, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784578.0429935}, {'msg': 'Docker shutdown gracefully', 'id': '1f82c1', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 1238, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784578.1116784}, {'msg': 'Docker shutdown gracefully', 'id': 'e5d352', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 1133, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784578.1542962}, {'msg': 'Docker shutdown gracefully', 'id': '48375e', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 1227, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784578.3957672}, {'msg': 'Docker shutdown gracefully', 'id': 'ae8291', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 1107, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784578.432867}, {'msg': 'Docker shutdown gracefully', 'id': '159791', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 1226, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784578.456471}, {'msg': 'Docker shutdown gracefully', 'id': '86a976', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 1134, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784578.4883127}, {'msg': 'Docker shutdown gracefully', 'id': '4bd9c0', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 1220, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784578.4941752}, {'msg': 'Docker shutdown gracefully', 'id': 'ccb710', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 1118, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784578.4953027}, {'msg': 'VM Shutting Down', 'id': 'ed9623', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784578.5074427}, {'msg': 'VM Shutting Down', 'id': '927e5f', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784578.578216}, {'msg': 'Docker shutdown gracefully', 'id': '740810', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 377, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784578.583223}, {'msg': 'Docker shutdown gracefully', 'id': '5b3d7f', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 1223, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784578.5957587}, {'msg': 'Docker shutdown gracefully', 'id': '053f0d', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 363, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784578.6311605}, {'msg': 'Docker shutdown gracefully', 'id': '0f746e', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 1239, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784578.6433594}, {'msg': 'VM Shutting Down', 'id': '188a17', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784578.722607}, {'msg': 'Docker shutdown gracefully', 'id': '78bdc7', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 368, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784578.7432454}, {'msg': 'Docker shutdown gracefully', 'id': '882dba', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 1133, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784578.771231}, {'msg': 'VM Shutting Down', 'id': '2ce19b', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784578.8157754}, {'msg': 'VM Shutting Down', 'id': '8a5a8b', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784578.9373345}, {'msg': 'VM Shutting Down', 'id': 'de7af1', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784578.9491608}, {'msg': 'VM Shutting Down', 'id': '7b031c', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784579.0524645}, {'msg': 'VM Shutting Down', 'id': '968a29', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784579.0548077}, {'msg': 'VM Shutting Down', 'id': '48c54a', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784579.1184535}, {'msg': 'VM Shutting Down', 'id': 'b1573d', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784579.1442685}, {'msg': 'VM Shutting Down', 'id': '3df2b4', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784579.1705067}, {'msg': 'VM Shutting Down', 'id': 'f7002d', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784579.1708462}, {'msg': 'VM Shutting Down', 'id': 'c5cd7d', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784579.194775}, {'msg': 'VM Shutting Down', 'id': '41e09c', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784579.237002}, {'msg': 'VM Shutting Down', 'id': 'bf3683', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784579.2565556}, {'msg': 'VM Shutting Down', 'id': '8d50f9', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784579.2889285}, {'msg': 'VM Shutting Down', 'id': 'f89d44', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784579.3032107}, {'msg': 'VM Shutting Down', 'id': '87e85e', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784579.348599}, {'msg': 'VM Shutting Down', 'id': '397f96', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784579.4365578}, {'msg': 'VM Shutting Down', 'id': '63a0b0', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784579.4787984}, {'msg': 'VM Shutting Down', 'id': '33a0e7', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661784579.593463}, {'msg': 'VM Launched', 'id': '629653', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785652.117039}, {'msg': 'VM Launched', 'id': 'a897eb', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785657.5877166}, {'msg': 'VM Launched', 'id': '6bbc0e', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785663.1190195}, {'msg': 'VM Launched', 'id': '7bfbd7', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785663.6177814}, {'msg': 'VM Launched', 'id': '77919b', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785664.0039184}, {'msg': 'VM Launched', 'id': '046987', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785686.5380678}, {'msg': 'Successfully pulled docker image', 'id': 'f6f9e3', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 31, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785688.8684962}, {'msg': 'Successfully pulled docker image', 'id': '67f64c', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 37, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785689.951487}, {'msg': 'Successfully pulled docker image', 'id': 'fce563', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 29, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785693.3034773}, {'msg': 'Successfully pulled docker image', 'id': '3fb06d', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 31, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785694.7074308}, {'msg': 'Successfully pulled docker image', 'id': 'c0ad4d', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 33, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785696.3357346}, {'msg': 'Successfully pulled docker image', 'id': '0d941b', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 32, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785718.9281058}, {'msg': 'VM Launched', 'id': '8e284f', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785957.22345}, {'msg': 'VM Launched', 'id': '1b1b47', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785957.6762247}, {'msg': 'VM Launched', 'id': '8a3ade', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785958.7556665}, {'msg': 'VM Launched', 'id': 'cf5029', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785961.8788981}, {'msg': 'VM Launched', 'id': 'f85336', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785969.3550606}, {'msg': 'VM Launched', 'id': '132ca4', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785969.867504}, {'msg': 'VM Launched', 'id': 'ce2cc5', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785970.2797878}, {'msg': 'VM Launched', 'id': '4526f0', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785970.3446023}, {'msg': 'VM Launched', 'id': 'b4949c', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785970.8269396}, {'msg': 'VM Launched', 'id': 'e16d26', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785971.101532}, {'msg': 'VM Launched', 'id': '1c2fba', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785971.5288196}, {'msg': 'VM Launched', 'id': '5f6c28', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785971.641443}, {'msg': 'VM Launched', 'id': '835800', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785972.1087117}, {'msg': 'VM Launched', 'id': 'ba69c5', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785972.5084364}, {'msg': 'Successfully pulled docker image', 'id': '65e067', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 33, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785996.8863795}, {'msg': 'VM Launched', 'id': '915aff', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': '', 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785996.9660633}, {'msg': 'Successfully pulled docker image', 'id': '4b953b', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 32, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785997.0898743}, {'msg': 'Successfully pulled docker image', 'id': '4c16be', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 35, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785997.4480119}, {'msg': 'Successfully pulled docker image', 'id': '956752', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 33, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785997.596956}, {'msg': 'Successfully pulled docker image', 'id': 'ff3fbd', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 29, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661785999.9614599}, {'msg': 'Successfully pulled docker image', 'id': '168009', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 32, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661786002.9035058}, {'msg': 'Successfully pulled docker image', 'id': '3b2d78', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 31, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661786003.3960545}, {'msg': 'Successfully pulled docker image', 'id': '9872b8', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 33, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661786004.753765}, {'msg': 'Successfully pulled docker image', 'id': '2b0eec', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 34, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661786005.1877086}, {'msg': 'Successfully pulled docker image', 'id': 'b3f235', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 33, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661786005.5519137}, {'msg': 'Successfully pulled docker image', 'id': '46c9c7', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 34, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661786005.8218186}, {'msg': 'Successfully pulled docker image', 'id': 'ab68a3', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 36, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661786009.0782194}, {'msg': 'Successfully pulled docker image', 'id': 'f1345d', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 38, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661786009.4665334}, {'msg': 'Successfully pulled docker image', 'id': '439fab', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 38, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661786010.2227867}, {'msg': 'Successfully pulled docker image', 'id': 'b59262', 'username': 'tony-piazza', 'account': 'tony-piazza', 'level': 20, 'event_type': 'vm_event', 'elapsed': 32, 'logging_context': {'user.username': 'tony-piazza', 'user.id': 5808, 'user.is_authenticated': True}, 'timestamp': 1661786029.9535468}]}

ntabris commented 1 year ago

This should be fixed once we release coiled 0.2.28.

ntabris commented 1 year ago

The fix has been released so I believe this issue is resolved.

tonycombocurve commented 1 year ago

I just updated to version 0.2.32 and I still see this message:

2022-10-12 19:27:08 - root - WARNING - Error getting STS token from client AWS session: Connect timeout on endpoint URL: "http://169.254.169.254/latest/api/token"

ntabris commented 1 year ago

Hm, okay. I'll think about this more. It shouldn't affect anything, but ideally you wouldn't be seeing this.

Is that showing in cluster logs or client-side logs?

tonycombocurve commented 1 year ago

It appears to only be in the client-side logs.

tonycombocurve commented 1 year ago

I am seeing it twice in the client-side logs for my app: once we the cluster is created and again when performing work on it.

tonycombocurve commented 1 year ago

After reviewing my code, the warning seems to be generated whenever we create a new cluster or connect to an existing cluster.

hayesgb commented 1 year ago

cc: @ntabris

ntabris commented 1 year ago

Okay, sorry for the delay on this but we think it should (actually this time) be fixed in coiled=0.2.42.

Apparently different versions of aiobotocore give different errors and there was a variety we weren't catching but (I think) are now.