FIRST-Tech-Challenge / fmltc

FIRST Machine Learning Toolchain
Other
38 stars 14 forks source link

Models fail to start training #302

Closed diltondev closed 1 year ago

diltondev commented 1 year ago

After making a dataset, trying to start training with a model does not work, and just produces an error, I've included the stacktrace I could find in the console. The error screen on the client side is also below.

Screen Shot 2023-02-18 at 9 39 05 AM

  "textPayload": "CRITICAL:root:model_trainer.start_training_model - creating eval job - except Traceback (most recent call last): ...   File \"/workspace/model_trainer.py\", line 308, in start_training_model ...     eval_job_response = ml.projects().jobs().create(parent=parent, body=eval_job).execute() ...   File \"/layers/google.python.pip/pip/lib/python3.9/site-packages/googleapiclient/_helpers.py\", line 134, in positional_wrapper ...     return wrapped(*args, **kwargs) ...   File \"/layers/google.python.pip/pip/lib/python3.9/site-packages/googleapiclient/http.py\", line 907, in execute ...     raise HttpError(resp, content, uri=self.uri) ... googleapiclient.errors.HttpError: <HttpError 400 when requesting https://ml.googleapis.com/v1/projects/ftc-ml-13312/jobs?alt=json returned \"Field: master_config.image_uri Error: Failed to read the container uri [gcr.io/ftc-ml-13312/object_detection:2021_11_25]. Please make sure that the image exists\". Details: \"[{'@type': 'type.googleapis.com/google.rpc.BadRequest', 'fieldViolations': [{'field': 'master_config.image_uri', 'description': 'Failed to read the container uri [gcr.io/ftc-ml-13312/object_detection:2021_11_25]. Please make sure that the image exists'}]}]\"> ... ",
  "insertId": "63f0eb2f000dabed8daf7abf",
  "resource": {
    "type": "gae_app",
    "labels": {
      "project_id": "ftc-ml-13312",
      "zone": "us12",
      "version_id": "v1",
      "module_id": "default"
    }
  },
  "timestamp": "2023-02-18T15:13:51.895981Z",
  "labels": {
    "clone_id": "00c61b117cfedcc11cb577ce74ba61b021a08854f8ff7fcb5e569f75f28030aa09558449bf50f8faf28f5eec3bc75c4c8bcd60c9aba5390fdf99"
  },
  "logName": "projects/ftc-ml-13312/logs/stderr",
  "receiveTimestamp": "2023-02-18T15:13:51.902403774Z"
}```
diltondev commented 1 year ago

Issue found and resolved: Re-uploading the docker images fixed the issue