This change makes Sematic artifact storage and logging work with non-AWS S3-compatible storage. Tested with minio with self-signed certificate and Helm-deployed sematic cluster.
Key parts of this change:
S3 Storage plug-in boto client is re-worked to support custom endpoints, as well as ignoring TLS errors.
New support in Helm chart for letting the user provide AWS keys as a secret (not sure how this one has gone missing for so long from today's charts??)
Support in Helm chart for using the re-worked boto client / custom S3 endpoint
Changes to api_endpoint to allow disabling of TLS verification, which is needed because api_client directly talks to storage
Also proposal that sematic retry decorator can ignore KeyboardInterrupt somehow, as it currently catches KeyboardInterrupt and that is really annoying during debugging.
See changes to docs/deploy.md for the jist.
Discussion about verify=False / skipping TLS verification:
Note that this change could perhaps support providing the user SSL cert (e.g. verify="path/to/cert.pem" instead of verify=False see e.g. https://stackoverflow.com/a/30405947 ), but this is tricky because:
sematic api_client needs the same cert, but conditionally where it queries storage directly ...
worker pods also need the cert itself somehow, need to be either baked into image or somehow magically inherited thru ConfigMap
LocalResolver / non-worker-pod runners, i.e. developer machines, need the cert somehow
I read up on requests and it seems it sometimes (always?) ignores /usr/local/share/ca-certificates so not sure if it's reliable to recommend installing the cert there.
One solution would be to make the sematic-server stop using redirects / self-signed urls for storage, as that can also be a pain when there are firewalls and other proxies in the picture. Not common for AWS, but possible elsewhere. I.e. all stores and reads must go thru sematic-server, and that alone has the proper cert / auth to read from storage. Note that docker-registry actually has this feature for these same reasons, see "redirect disable" https://docs.docker.com/registry/configuration/#redirect
Another solution is to just accept the verify=False approach used in this PR, as it can also be really useful for testing, and to put verify=False` behind a feature flag of sorts that gets set both server-side and user sematic yaml config.
This change makes Sematic artifact storage and logging work with non-AWS S3-compatible storage. Tested with minio with self-signed certificate and Helm-deployed sematic cluster.
Key parts of this change:
api_endpointto allow disabling of TLS verification, which is needed becauseapi_clientdirectly talks to storageKeyboardInterruptsomehow, as it currently catchesKeyboardInterruptand that is really annoying during debugging.See changes to
docs/deploy.mdfor the jist.Discussion about
verify=False/ skipping TLS verification:verify="path/to/cert.pem"instead ofverify=Falsesee e.g. https://stackoverflow.com/a/30405947 ), but this is tricky because:api_clientneeds the same cert, but conditionally where it queries storage directly ...requestsand it seems it sometimes (always?) ignores/usr/local/share/ca-certificatesso not sure if it's reliable to recommend installing the cert there.One solution would be to make the sematic-server stop using redirects / self-signed urls for storage, as that can also be a pain when there are firewalls and other proxies in the picture. Not common for AWS, but possible elsewhere. I.e. all stores and reads must go thru sematic-server, and that alone has the proper cert / auth to read from storage. Note that docker-registry actually has this feature for these same reasons, see "redirect disable" https://docs.docker.com/registry/configuration/#redirect
Another solution is to just accept the
verify=Falseapproach used in this PR, as it can also be really useful for testing, and to put verify=False` behind a feature flag of sorts that gets set both server-side and user sematic yaml config.