allow crm code to run from ccrm

[gainsley]

Previously, our architecture required a CRM service running on edge-site to convert platform-independent APIs to platform-specific API calls to deploy VMs/Clusters/AppInsts/etc. To be able to support platforms where it is not feasible to run the CRM service on edge-site (typically because there is another layer of software managing the infrastructure), we want to be able to run the platform-specific code from the CCRM service, which runs off-edge-site alongside the Controller.

This PR refactors the common CRM code (from pkg/crmutil/controller-data.go), moving the parts that are specific to a single CRM on-edge into pkg/crm. The remaining code is modified to support being included in either the CRM, which is a single-instance process using notify for communication with the Controller, or the CCRM which is a horizontally scaled set of processes that use direct access to etcd and GRPC to communicate with the Controller. Code in pkg/crmutil is meant to be shared between both CRM and CCRM.

In the common pkg/crmutil/controller-data.go code, there was a lot of functionality that was specific to a single-instance process running over notify. The following changes were made:

• The functions in controller-data.go now use a "InfoSender" object to send data back to the controller. In the CRM case, this writes to caches which trigger sending over notify as before. In the CCRM case, this writes directly back to the GRPC API.
• The "vmResourceActionEnd" trigger that used to run after each ClusterInst/AppInst is now moved to the CRM or CCRM.
• All of the code in controller-data.go related to applying TrustPolicyExceptions is rewritten in pkg/controller/trustpolicyexception_changes.go.
• The VMPool-related code is moved only to the CRM. It is written in such a way that it cannot support being done from the CCRM, and I did not want to rewrite it since no one is using it. So VMPool is only support for CRM on-edge.

The platform code also requires changes to be able to run from CCRM. This was only partially completed.

• Platforms that use reservations of subnets/IPs can do so via the SyncFactory
• The cert refresh thread which used to be part of the VMProvider code is now moved to the CRM pkg/crm/crm.go. For the CCRM, the Controller has a periodic thread that runs to trigger CCRMs to refresh certs in pkg/controller/ccrm_periodic.go.
• The cloudlet ssh key refresh thread is replaced by the CloudletSSH package, which provides on-demand signed ssh certs. TODO:
• Some platforms use mutexes to prevent multiple threads from modifying the infra at the same time. These need to be handled such that things safely work if multiple processes are modifying the infra at the same time.
• The VMProvider code has a lock to protect against simultaneous writing to a VM (network config files, IP tables rules). This can probably be replaced by using file locks to get a fencing token on read, and then the same file lock to abort a write if the fencing token is stale (same concept as etcd transactions). Note the file lock is not actually held between the read and the write.

Other changes:

• The fieldMap is now its own object instead of a map[string]string. I have changed the way a field flag for a field that is a struct is interpreted, i.e. CloudletInfoFieldStatus, which refers to the Status (edgeproto.StatusInfo) field of edgeproto.CloudletInfo. Previously, setting the Status field doesn't imply anything on its own, but it was required if a subfield of StatusInfo was set, i.e. CloudletInfoFieldStatusTaskName. The CopyInFields() function used to require both to be set to copy in the TaskName, but I have changed it such that if the parent CloudletInfoFieldStatus flag is set, it copies in the entire struct. I have also updated the way the hand-written code uses flags, as sometimes it was checking if the flag or its parent was set, or it was really trying to check if the flag or any of its children were set. The FieldMap code is now auto-generated and ends up in api/edgeproto/alert.pb.go.

I still need to run tests against a real infra like Openstack, and maybe take care of the AppInstRuntime TODO.

[gainsley]

Hi Lev, I found a bunch of issues after more testing. In particular, I didn't realize the proxyCerts object was called by the infra-specific code, so I needed to split it into a stateless and cloudlet-specific part, and a persistent and cloudlet-independent part (cache).

[levshvarts]

Hi Lev, I found a bunch of issues after more testing. In particular, I didn't realize the proxyCerts object was called by the infra-specific code, so I needed to split it into a stateless and cloudlet-specific part, and a persistent and cloudlet-independent part (cache).

Sounds good. I'll still skim through the changes to build a better picture in my head, but won't focus on specifics. Will do a more in-depth review after your changes.

[gainsley]

Hey Lev, I already pushed the fixes, so the PR is complete.

[gainsley]

Thanks Lev also for taking on that code review! So I have done a wide range of tests, besides the usual unit and e2e tests, I also set it up (you can see this in the director changes) to run make test-start-dns, which let me create a cloudlet on Openstack from a CRM started via e2e tests. I also tested a k3s deployment with the operator changes on Openstack (both those openstack tests used the acceptance tests to test). But yeah, I only tested openstack with CrmOnEdge=true. All the CrmOnEdge=false testing is via the fake platform in unit/e2e tests.

I think that when we implement the OSM platform with CrmOnEdge=false we'll have a chance to find any other bugs lurking there.