search

gleepwurp / hassio-addons

My personal hassio addons
0 stars 0 forks source link

S3 Transfer fails when snapshot is invalid/low-size #1

Open gleepwurp opened 3 years ago

gleepwurp commented 3 years ago

Snapshot that are 10240-bytes long seem to be causing S3 backup issues. Deleting those specifics snapshots resolve the issue, but we should be able to identify them and skip automatically, with a warning to clean up.

gleepwurp commented 3 years ago

hmm, upon further testing, seems that the error occurs always when trying to upload the 2nd Snapshot:

  File "/usr/lib/python3.8/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/usr/lib/python3.8/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/usr/lib/python3.8/site-packages/botocore/handlers.py", line 529, in validate_ascii_metadata
    value.encode('ascii')
AttributeError: 'NoneType' object has no attribute 'encode'
[cont-finish.d] executing container finish scripts...
[cont-finish.d] 99-message.sh: executing... 
[cont-finish.d] 99-message.sh: exited 0.
gleepwurp commented 3 years ago

Really seems to have issues with certain file sizes: 7004160 bytes, 10240 bytes. Windows if they're invalid snapshots.

gleepwurp commented 3 years ago

OK, nothing related to the file at all. ExtraArgs had a "None" value as the homeassistant version, and was causing issues when encoding/translating of that extra args by the Boto3 module.

ExtraArgs: {'StorageClass': 'STANDARD', 'Metadata': {'type': 'partial', 'name': 'addon_6c175ec2_s3-backup_1.2', 'date': '2021-03-30T02:29:00.367733+00:00', 'homeassistant': **None**}}

That part of the metadata comes from the Home-Assistant Supervisor API, where it gets all the details about the snapshot being backed up.

Seems only some snapshot have the homeassistant version missing:


DEBUG:__main__:Local file /backup/9a7744d4.tar found in S3 with matching size of 2220339200 bytes
DEBUG:__main__:Local file /backup/53300b24.tar found in S3 with matching size of 1011886080 bytes
DEBUG:__main__:Local file /backup/cd539f42.tar found in S3 with matching size of 6084311040 bytes
WARNING:__main__:Local file /backup/4e81250e.tar not found in S3
Bucket: hassio-main
Key: backup/4e81250e.tar
ExtraArgs: {'StorageClass': 'STANDARD', 'Metadata': {'type': 'partial', 'name': 'addon_a0d7b954_grafana_6.1.4', 'date': '2021-03-29T14:30:38.214375+00:00', 'homeassistant': None}}
WARNING:__main__:Local file /backup/888670f3.tar not found in S3
Bucket: hassio-main
Key: backup/888670f3.tar
ExtraArgs: {'StorageClass': 'STANDARD', 'Metadata': {'type': 'partial', 'name': '', 'date': '2021-03-29T06:00:43.354400+00:00', 'homeassistant': '2021.3.4'}}
DEBUG:__main__:Local file /backup/1affa7c7.tar found in S3 with matching size of 6798069760 bytes
WARNING:__main__:Local file /backup/a33613a6.tar not found in S3
Bucket: hassio-main
Key: backup/a33613a6.tar
ExtraArgs: {'StorageClass': 'STANDARD', 'Metadata': {'type': 'partial', 'name': 'addon_a0d7b954_unifi_0.20.2', 'date': '2021-03-10T02:54:02.362986+00:00', 'homeassistant': None}}
WARNING:__main__:Local file /backup/f7bc35a4.tar not found in S3
Bucket: hassio-main
Key: backup/f7bc35a4.tar
ExtraArgs: {'StorageClass': 'STANDARD', 'Metadata': {'type': 'partial', 'name': 'core_2021.3.3', 'date': '2021-03-13T20:25:48.234066+00:00', 'homeassistant': '2021.3.3'}}```