search

aaronwmorris / indi-allsky

Software to manage a Linux-based All Sky Camera.
GNU General Public License v3.0
219 stars 35 forks source link

(Minor issue for just a few files) upload_sync.py - ('Request failed with status code', 412, 'Expected one of', <HTTPStatus.OK: 200>) #1429

Open gaitskell opened 1 month ago

gaitskell commented 1 month ago

This is a very minor issue since it affects very few files/operations but it may offer insights elsewhere?

The ./misc/upload_sync.py sync command works well - thank you.

However, at the tail end of the syncrhonization, or if I ctl-C and start again the last few files cause the following error shown below. (I am not sure what causes the Precondition Failed error ... ('Request failed with status code', 412, 'Expected one of', <HTTPStatus.OK: 200>).

Is this something that can be trapped? I can't work out what the specifc error is - 

|    s3   |   IndiAllSkyDbStarTrailsTable    |   192    |    0    |
|    s3   | IndiAllSkyDbStarTrailsVideoTable |    67    |    0    |
|    s3   |  IndiAllSkyDbPanoramaVideoTable  |    71    |    0    |
|    s3   |      IndiAllSkyDbImageTable      |   6873   |    15   |
|    s3   |  IndiAllSkyDbPanoramaImageTable  |    0     |    0    |
|    s3   |    IndiAllSkyDbFitsImageTable    |   7053   |   111   |
|    s3   |    IndiAllSkyDbRawImageTable     |    -     |    -    |
|  upload |      IndiAllSkyDbVideoTable      |    -     |    -    |
|  upload |     IndiAllSkyDbKeogramTable     |    -     |    -    |
|  upload |   IndiAllSkyDbStarTrailsTable    |    -     |    -    |
|  upload | IndiAllSkyDbStarTrailsVideoTable |    -     |    -    |
|  upload |  IndiAllSkyDbPanoramaVideoTable  |    -     |    -    |
|  upload |    IndiAllSkyDbRawImageTable     |    -     |    -    |
|  upload |    IndiAllSkyDbFitsImageTable    |    -     |    -    |
+---------+----------------------------------+----------+---------+
2024-07-27 14:22:57,904 [INFO] MainProcess upload_sync.sync() [197]: Entries to upload: 128
2024-07-27 14:22:57,905 [INFO] MainProcess upload_sync.addUploadEntries() [240]: Adding 11 upload entries (117 remaining)
2024-07-27 14:22:57,911 [ERROR] MainProcess upload_sync.addUploadEntries() [250]: IndiAllSkyDbVideoTable file missing: /var/www/html/allsky/images/ccd_f3ee6189-ab30-44e0-a7a7-75ec905af83c/20240629/allsky-timelapse_ccd1_20240629_day.mp4
2024-07-27 14:22:57,914 [ERROR] MainProcess upload_sync.addUploadEntries() [250]: IndiAllSkyDbVideoTable file missing: /var/www/html/allsky/images/ccd_f3ee6189-ab30-44e0-a7a7-75ec905af83c/20240315/allsky-timelapse_ccd1_20240315_day.mp4
2024-07-27 14:22:57,945 [INFO] MainProcess miscUpload.s3_upload_asset() [478]: Uploading to S3 bucket
2024-07-27 14:22:57,969 [INFO] MainProcess miscUpload.s3_upload_asset() [478]: Uploading to S3 bucket
2024-07-27 14:22:57,982 [INFO] MainProcess miscUpload.s3_upload_asset() [478]: Uploading to S3 bucket
2024-07-27 14:22:57,995 [INFO] MainProcess miscUpload.s3_upload_asset() [478]: Uploading to S3 bucket
2024-07-27 14:22:58,009 [INFO] MainProcess miscUpload.s3_upload_asset() [478]: Uploading to S3 bucket
2024-07-27 14:22:58,014 [INFO] MainProcess miscUpload.s3_upload_asset() [478]: Uploading to S3 bucket
2024-07-27 14:22:58,033 [INFO] MainProcess miscUpload.s3_upload_asset() [478]: Uploading to S3 bucket
2024-07-27 14:22:58,046 [INFO] MainProcess miscUpload.s3_upload_asset() [478]: Uploading to S3 bucket
2024-07-27 14:22:58,060 [INFO] MainProcess miscUpload.s3_upload_asset() [478]: Uploading to S3 bucket
2024-07-27 14:22:58,073 [INFO] MainProcess miscUpload.s3_upload_asset() [478]: Uploading to S3 bucket
2024-07-27 14:22:58,192 [INFO] MainProcess upload_sync._fileUploadWorkerStart() [1079]: Starting FileUploader process 1
2024-07-27 14:22:58,224 [INFO] MainProcess generic.connect() [52]: Connecting to storage.googleapis.com (443) as * with gcp_storage
2024-07-27 14:23:00,057 [INFO] MainProcess generic.put() [66]: Uploading /var/www/html/allsky/images/ccd_f3ee6189-ab30-44e0-a7a7-75ec905af83c/exposures/20240726/night/26_21/ccd1_20240726_210119.jpg
Exception in thread Upload-1:
Traceback (most recent call last):
  File "/home/admin/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 2601, in _prep_and_do_upload
    created_json = self._do_upload(
  File "/home/admin/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 2407, in _do_upload
    response = self._do_multipart_upload(
  File "/home/admin/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 1920, in _do_multipart_upload
    response = upload.transmit(
  File "/home/admin/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/google/resumable_media/requests/upload.py", line 153, in transmit
    return _request_helpers.wait_and_retry(
  File "/home/admin/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/google/resumable_media/requests/_request_helpers.py", line 155, in wait_and_retry
    response = func()
  File "/home/admin/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/google/resumable_media/requests/upload.py", line 149, in retriable_request
    self._process_response(result)
  File "/home/admin/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/google/resumable_media/_upload.py", line 125, in _process_response
    _helpers.require_status_code(response, (http.client.OK,), self._get_status_code)
  File "/home/admin/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/google/resumable_media/_helpers.py", line 108, in require_status_code
    raise common.InvalidResponse(
google.resumable_media.common.InvalidResponse: ('Request failed with status code', 412, 'Expected one of', <HTTPStatus.OK: 200>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
    self.run()
  File "/home/admin/indi-allsky/indi_allsky/uploader.py", line 112, in run
    raise e
  File "/home/admin/indi-allsky/indi_allsky/uploader.py", line 108, in run
    self.saferun()
  File "/home/admin/indi-allsky/indi_allsky/uploader.py", line 139, in saferun
    self.processUpload(u_dict)
  File "/home/admin/indi-allsky/indi_allsky/uploader.py", line 454, in processUpload
    response = client.put(**put_kwargs)
  File "/home/admin/indi-allsky/indi_allsky/filetransfer/gcp_storage.py", line 119, in put
    blob.upload_from_filename(
  File "/home/admin/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 2923, in upload_from_filename
    self._handle_filename_and_upload(
  File "/home/admin/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 2793, in _handle_filename_and_upload
    self._prep_and_do_upload(
  File "/home/admin/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 2619, in _prep_and_do_upload
    _raise_from_invalid_response(exc)
  File "/home/admin/indi-allsky/virtualenv/indi-allsky/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 4773, in _raise_from_invalid_response
    raise exceptions.from_http_status(response.status_code, message, response=response)
google.api_core.exceptions.PreconditionFailed: 412 POST https://storage.googleapis.com/upload/storage/v1/b/allskycamera-images/o?uploadType=multipart&ifGenerationMatch=0: {
  "error": {
    "code": 412,
    "message": "At least one of the pre-conditions you specified did not hold.",
    "errors": [
      {
        "message": "At least one of the pre-conditions you specified did not hold.",
        "domain": "global",
        "reason": "conditionNotMet",
        "locationType": "header",
        "location": "If-Match"
      }
    ]
  }
}
: ('Request failed with status code', 412, 'Expected one of', <HTTPStatus.OK: 200>)
aaronwmorris commented 1 month ago

Looks like I might have misunderstood one of the parameters or just copied some code I did not need.

I had a condition that did not allow overwriting existing files in the GCP Object Storage. The file is already uploaded, but the indi-allsky DB did not record it (possibly due to the other issue you posted), so it was re-trying the upload, but the condition did not allow it to overwrite.

Merged #1430 to remove the condition preventing overwrites.