Closed nealef closed 1 year ago
You should use the gRPC plugin interface, so the compatibility is happening at a protobuf API level not a go plugin level. https://github.com/containerd/containerd/blob/main/docs/PLUGINS.md#proxy-plugins
I had created a content store based on the snapshotter plugin. However, it’s unclear how to connect my store implementation with the RPC service. For snapshotter we:
// Convert the snapshotter to a gRPC service,
snsvc := snapshotservice.FromSnapshotter(rs)
// Register the service with the gRPC server
snapshotsapi.RegisterSnapshotsServer(rpc, snsvc)
With that in place my snapshotter works fine.
However, for the content store I am a little lost. For content there's a RegisterContentServer()
but that's something tha's implemented in content/contentserver/contentserver.go
that uses a content store. So I am now sure how to make the association so my Info()
, Status()
functions are invoked.
You need to disable the built in one. See the containerd config here as a reference: https://github.com/hinshun/ipcs/blob/master/cmd/containerd/config.toml#L4
It's done. When I try to do a pull I get an "unimplemented" error. I was wondering if the IPCS implementation chose to use the .so
plugin method because they were having the same type of trouble. With the proxy method I am unclear how to let the grpc mechanism know we can handle those content store requests.
I'm not sure I understand. Why not just use these:
cssvc := contentserver.New(cs)
cssvc.Register(rpc)
IPCS should be using the proxy mechanism, its been a few years so it's unclear why it chose the .so
plugin method first. Probably to facilitate easier debugging / stacktraces when it was under heavy development.
With my gRPC implementation I had:
sv := contentserver.New(cs)
api.RegisterContentServer(rpc, sv)
Because using:
sv.Register(cs)
Resulted in:
sv.Register undefined (type "github.com/containerd/containerd/api/services/content/v1".ContentServer has no field or method Register)
As contentserver.New(cs)
returns api.ContentServer
not service
for:
func (s *service) Register(server *grpc.Server)
The registration provides no feedback so it's not clear if things are set. I see a connection come in:
146069 accept4(7, {sa_family=AF_UNIX}, [112 => 2], SOCK_CLOEXEC|SOCK_NONBLOCK) = 8
But no traffic flows and when I do a pull:
# ctr image pull docker.io/library/httpd:latest
ERRO[0000] active check failed error="unknown service containerd.services.content.v1.Content: not implemented"
ERRO[0000] active check failed error="unknown service containerd.services.content.v1.Content: not implemented"
Which seems to tell me I really haven't registered properly.
Got it - i had incorrectly named the type content.store
in the function that creates a content store. As a result the registration got garbage in and I go garbage out. I am now getting the gRPC messages. I am able to handle Delete/Walk/ListStatuses. Pull is proving less co-coperative where I am still getting "Not Implemented" messages but this should be easier to track down.
Thanks for the suggestions, they helped enormously.
I think the registration is still (a/the) problem. The value returned by sv := contentserver.New(cs)
is:
fmt.Printf("sv - %q type: %s\n",sv,reflect.TypeOf(sv));
sv - &{%!q(*content.store=&{/var/lib/containerd/io.containerd.content.v1.content /var/lib/containerd-mssnap-grpc/cache 0xc0001351e0})} type: *contentserver.service
The significant part is that the type is *content.service
, therefore it should be possible to invoke:
sv.Register(rpc)
So why does it tell me:
sv.Register undefined (type "github.com/containerd/containerd/api/services/content/v1".ContentServer has no field or method Register)
I have created a discussion on containerd
to pursue this further. https://github.com/containerd/containerd/discussions/8611
When I build a plugin
containerd
will complain with "plugin was built with a different version ...". I am building with the same level ofgolang
ascontainerd
and mygo.mod
has the same level ofcontainerd
that I am running but I cannot get rid of this. How did you avoid it?