search

tweag / chainsail

Replica Exchange sampling as-a-service
MIT License
11 stars 1 forks source link

Logging for user code container #204

Open dorranh opened 3 years ago

dorranh commented 3 years ago

As discussed there are some considerations for enabling logging in the container which runs user-defined code:

Currently our remote logging uses graphite which does not have a security layer. If we want to log from the Docker container running the user-defined code, we need to find a safe way to avoid having the user attacking our logging server (e.g. spamming a bunch of logs) which would affect other user's jobs which need to log to the same server.

One idea for this would be to expose a second grpc endpoint for streaming logs back to the controller (which can then safely submit them to the logging server).

dorranh commented 3 years ago

@simeoncarstens @SaeedHK

dorranh commented 3 years ago

After thinking about this a bit more, I think the following (assuming a Kubernetes implementation of Node) would be one option. S till not sure if this provides the level of security we are after though.

Add a logging endpoint to the user code gRPC server which streams logs from the user container. Then, block (e.g. via a k8s network policy) all outbound connections from the user code container which target our internal services. This would allow the pod to continue to access the internet for installing dependencies, etc. while providing at least a bit of isolation from our other internal components.