Making the agents platform-agnostic

Mierdin commented 8 years ago

Many network engineers are commenting that it can be difficult to get compute capacity at remote locations especially. Should add a section to the docs that explains the agent can run anywhere, and specifically mention the network switch vendors that are allowing things like agents and containers to run on their platforms.

The idea of running an agent on a switch should also be explored in greater detail. Cisco and Juniper have options to do this on some of their platforms, should look into this more.

Mierdin commented 8 years ago

Adding label "high-effort" purely because of the research effort involved here. Hoping someone with experience with some of these platforms can help here.

Mierdin commented 5 years ago

From @namachieli in https://github.com/toddproject/todd/issues/148:

Ran into a sort of a "no duh" issue today when trying to push a testrun to two agents on different architectures, easy enough to work around. I'm still in POC and most likely a production deployment will see use of the same agent hardware globally though, i do see a case where i want to run a VM in AWS to poke VPN portals.

I found 30 - Removing MakeFile and 83 - Agent on Network Equipment that seem related, but they mostly talk about this philosophically.

This also addresses that if you just follow the docs for deploying todd-server on say, Arch:x86_64, you end up with a todd-nativetestlet-ping testlet that pushed to potentially Arch:arm (rPi) agents.

I would think there are quite a few ways to implement something like this, both manually and automatically. I think both would require some shared logic to support.

Shared Logic (Some ideas, not exhaustive)

Assumption: We know the arch of client/server. How do we make sure Agents get the correct testlet?
1. Use different buckets - /opt/server/assets/testlets/arch/[x64_86, arm, etc]
  - Only push testlets in the correct bucket
2. Manage Testlet like a package you 'add' to ToDD.
  1. Add the requirement for testlets to have an output to the arg [arch]. When ToDD server registers a new testlet, it check for the output and tracks it.

Manual

Configure architecture in agent.cfg and server.cfg as applicable to tell the server more about the agent.
Include in group definition and require a given group criteria must all be same agent architecture
- Probably not worth being this restrictive for the effort it takes to implement less restrictive methods

Automatic

todd-agent startup include parsing hostnamectl (Great info about environment here)

I would love to hear more ideas.

Mierdin commented 5 years ago

From @writemem in https://github.com/toddproject/todd/issues/176:

What is the official stance of this project on cross-platform support? I'd like to be able to run todd-agent from my MacOS machine. I can see a use case for on-site engineers to (easily) bring up a local agent as part of change validation. I'd also like to, eventually, run todd-server from darwin for development purposes, but I realize there's a lot more to that.

I've rewritten the get_addresses factcollector in Go. This works on my MacOS machine. But, after the first run, it downloads the original bash script which causes it to break. I can put a darwin/amd64 compiled binary in /opt/todd/server/assets/factcollectors on the server and then i'm able to run tests from my MacOS box. But, of course, that's not going to work well for other OS/arch.

When this happens on the agent: Agent task received: {"type":"DownloadAsset","assets":["http://x.x.x.x:8090/factcollectors/get_addresses"]} Downloading http://x.x.x.x:8090/factcollectors/get_addresses to /opt/todd/agent/assets/factcollectors/get_addresses

It induces this error: Error gathering facts: parsing "/opt/todd/agent/assets/factcollectors/get_addresses" output: invalid character '\n' in string literal

Some initial thoughts on how to implement this:

have todd-agent collect GOOS and GOARCH and include those in the DownloadAsset http request as query params. e.g. factcollectors/get_addresses?GOOS=darwin&GOARCH=amd64. Perhaps the make could cross-compile the code below for darwin/amd64 and serve that if the proper query parameters are in the request.
perhaps there could be an agent-only installation and this could be compiled for the current GOOS and GOARCH. this would leave the problem of get_addresses being replaced, so that would need to be given some more thought.
somehow use this code for all os and architectures -- precompiled on the server or compiled on-demand?

thoughts?

package main

import (
    "encoding/json"
    "fmt"
    "log"
    "net"
)

func main() {
    type IPs struct {
        Addresses []string `json:"Addresses"`
    }
    var ips IPs
    addrs, err := net.InterfaceAddrs()
    if err != nil {
        log.Fatal(err)
    }
    for _, a := range addrs {
        ip, _, err := net.ParseCIDR(a.String())
        if err != nil {
            log.Fatal(err)
        }
        ips.Addresses = append(ips.Addresses, ip.String())
    }
    j, err := json.Marshal(ips)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(string(j))
}

c2rad commented 5 years ago

@Mierdin Thanks for organizing my comment appropriately. Would you be open to a PR which adds factcollector support for macOS at a minimum or any os/arch at best? Getting the agent to run properly seems like a good first step.

As an aside, I've also been poking at Cisco cat 9ks and have run agents successfully. Disk space out-of-the-box seems to be a limiting factor (with git being a prereq for 'go get' etc...). External storage can be added to alleviate this concern. But, what I've done as a mitigation is to compile/install on another linux amd64 machine and copy the files to guestshell. More to come on this if it's a topic of interest.

toddproject / todd