`configsync | CRITICAL: An error occurred` after OPNsense 22.7.11 upgrade, possibly related to Boto3 update

[ndejong]

System update to OPNsense 22.7.11 (and maybe before this) causes the OPNsense repo hosted py39-boto3 (v1.26.13) and py39-botocore (v1.29.13) packages

This Boto3 version does not seem to be working with ConfigSync using Amazon-S3 or Google-S3 storage providers. The Digital Ocean storage-provider still works fine. Given the Digital Ocean setup requires an Endpoint URL override, a quick test to see if the others work if their respective URL overrides are supplied does not work-around the issue.

Errors in the logs like -

configsync | CRITICAL: An error occurred (AccessDenied) when calling the PutObject operation: Access denied.
configsync | CRITICAL: An error occurred (NoSuchKey) when calling the ListObjectsV2 operation: The specified key does not exist.

Stack trace using --debug on a manual test_parameters action shows -

2023-01-31T15:55:15+1000 configsync[7143] DEBUG: __put_object() - client_params[Key]=internal-ool1/opnsense/config-test.xml
2023-01-31T15:55:16+1000 configsync[7143] CRITICAL: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
Traceback (most recent call last):
  File "/usr/local/opnsense/scripts/ThreatPatrols/ConfigSync/configsync.py", line 205, in <module>
    response = configsync_cli()
  File "/usr/local/opnsense/scripts/ThreatPatrols/ConfigSync/configsync.py", line 78, in configsync_cli
    return storage_provider.test_parameters(
  File "/usr/local/opnsense/scripts/ThreatPatrols/ConfigSync/configsync/storage_provider/__init__.py", line 89, in test_parameters
    return self.storage_provider.test_parameters(**kwargs)
  File "/usr/local/opnsense/scripts/ThreatPatrols/ConfigSync/configsync/storage_provider/boto3_compatible.py", line 96, in test_parameters
    return self.__sync_files(files=config_files, overwrite_existing=True)
  File "/usr/local/opnsense/scripts/ThreatPatrols/ConfigSync/configsync/storage_provider/boto3_compatible.py", line 223, in __sync_files
    response = self.__put_object(
  File "/usr/local/opnsense/scripts/ThreatPatrols/ConfigSync/configsync/storage_provider/boto3_compatible.py", line 285, in __put_object
    response = boto_client.put_object(**client_params)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 515, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 934, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
{
  "message": "An error occurred (AccessDenied) when calling the PutObject operation: Access Denied",
  "status": "fail"
}

Manually testing credentials outside of ConfigSync shows these credentials are valid and the breakage was noticed right after a system upgrade to 22.7.11

[ndejong]

Bit of digging seems to show -

• the upgrade bumped us from py39-boto3-1.24.30 to py39-boto3-1.26.13 and py39-botocore-1.27.30 to py39-botocore-1.29.13
• Boto3 did internally change the way they handle endpoints determination in 1.26.0 diff which would align with the breakage here
• By returning None from __endpoint_url() when the provider is AWS restores functionality for AWS-S3
• Unsure what's going on with Google-S3 at the moment

[ndejong]

Resolved commit