Improve Developer UX

[adam-stokes]

Provide a mechanism for running deployments+tests locally in the same fashion as CI

[adam-stokes]

So far not really a good sign:

 [adam:~/Projects/e2e-testing] feat-1682 ± mage -init
magefile.go created
⏚ [adam:~/Projects/e2e-testing] feat-1682(+2/-1) ± ls
cli         dist  fleet_amd64-sshhosts        go.mod  internal     Makefile  NOTICE.txt  README.md       tools
commons.mk  e2e   fleet_elastic_pkg-sshhosts  go.sum  magefile.go  notice    outputs     stack-sshhosts
⏚ [adam:~/Projects/e2e-testing] feat-1682(+2/-1) ± mage -l
Error determining list of magefiles: failed to list mage gofiles: exit status 1:
⏚ [adam:~/Projects/e2e-testing] feat-1682(+2/-1) 1 ±

I prefer to have full language support which is why I thought magefile would be good, however, it may just as well that we utilize our existing cli and expand on it. @mdelapenya wdyt?

[cachedout]

Hi @adam-stokes and @mdelapenya

I'm curious about this issue. Could one or both of you provide a more detailed explanation of what set of problems set to this issue and what we hope to achieve? Thanks!

[mdelapenya]

@cachedout with the rush to move the tests to real VMs, we have lost the ability to run the tests from local (thanks to the Docker approach we followed in the past). We are currently supporting running the tests using Jenkins, only, and we'd like to provide the very same experience in local than on the CI. @adam-stokes can share more insights about the internal details, but as a spoiler, we'd like to move the ansible calls part to the command line instead of having them embedded in the Jenkins pipeline.

[cachedout]

@mdelapenya Thanks for the explanation. I did play around a bit with the E2E tests recently and noticed that all but the k8s tests (I think?) were failing when run locally. I'm assuming this is the reason for that?

@adam-stokes Can you please share a bit more about the specifics of the work that needs to be done here? This feels very nebulous right now and if we could update this issue to include some specific details, it might help us find some traction.

@kseniia-kolpakova It looks as if this was moved into the "planned" column all the way back in December. Are there teams waiting on this? Does it need to continue to be prioritized or should it be moved back into the backlog? I'm assuming that no work has been done here since I don't see any PRs linking back to this issue but perhaps they simply haven't referenced this issue?

Thanks all. :)

[mdelapenya]

@mdelapenya Thanks for the explanation. I did play around a bit with the E2E tests recently and noticed that all but the k8s tests (I think?) were failing when run locally. I'm assuming this is the reason for that?

Exactly. The the agent will be installed in the current machine, so the code is inferring the target OS and arch to select what agent to install (DEB, RPM, TAR... amd/arm).

Those scenarios not needing to install an agent, or those related to the stand-alone mode, can be run locally, in particular: apm_integration.feature, integrations.feature, stand_alone_agent.feature.

Beside those, the helm charts test suite should work locally, as they use kind for the deployment of the Helm charts.

Finally, welcome back!!!

[cachedout]

@mdelapenya OK, cool. Makes perfect sense. What is the proposed solution to address this problem? (And thanks for the warm welcome back!)

[mdelapenya]

@adam-stokes is currently working in the provisioner part, which is currently implemented at CI-only with Ansible: AWS instances are provisioned at CI time. Ideally, we should be able to port this behaviour and move it next to the developer with tooling. @adam-stokes can provide more insights on this topic.

[kseniia-kolpakova]

@kseniia-kolpakova It looks as if this was moved into the "planned" column all the way back in December. Are there teams waiting on this? Does it need to continue to be prioritized or should it be moved back into the backlog? I'm assuming that no work has been done here since I don't see any PRs linking back to this issue but perhaps they simply haven't referenced this issue?

I don't think anyone is waiting for this. In this case it would have had priority high or blocker. It is in planned column because @adam-stokes works on it.
I agree that often we have too many things planned and in-progress - that are unrealistic to complete. We attempted to clean-up planned column during the last iteration planning meeting (a week ago). Let's continue doing that!

[cachedout]

It is in planned column because @adam-stokes works on it.

Thanks, @kseniia-kolpakova

@adam-stokes Is this actively being worked on? If not, let's move it out of this column. If it is being worked on, but it's simply too large to fit into a single iteration, let's figure out a better plan and start splitting the work up into discrete tasks so that we can better track our progress. Thanks.

[adam-stokes]

@cachedout Let's discuss this one today, I wanted to run the POC I wrote by you and get your input on direction

[adam-stokes]

Closing as the provisioning tool is being extended to support Orka API for macstadium and Windows machines within AWS/GCP which are being tracked in the recent linked issues