onsi / ginkgo

A Modern Testing Framework for Go
http://onsi.github.io/ginkgo/
MIT License
8.09k stars 643 forks source link

Is there a way to populate dynamic data in the Tree Construction Phase? #1361

Closed tjs1989 closed 4 months ago

tjs1989 commented 4 months ago

Hello,

This is more of a question than an issue currently, so apologies if I've raised this in the wrong way. It's the first time that I'm raising a question in Git :)

We're using Ginkgo for many of the services which we have at my employer, and we're currently in the process of implementing it on another one. This service has client secrets which we can retrieve from an endpoint. We then use these secrets to bring back package information etc, and then we can perform operations such as configurations etc on the underlying endpoints.

Eventually our Ginkgo tests will be able to apply and validate configurations on the packages. We therefore have a need to use a DescribeTable, but we don't want to store the secrets in code for the table entries as a matter of security :)

Another caveat is that we also need to get an Auth token before the tests run. This is being taken care of in a synchronizedBeforeSuite.

It is in the above synchronizedBeforeSuite that we are also making the call to get the client secrets, and we pass those to the inner function along with the auth token.

Here in lies the problem. What we are currently trying to do is generate a TableEntry (to avoid a massive list of Entry records) by appending the client secrets to it.

    entries := make([]TableEntry, 0, len(clientSecrets))

    for _, clientSecret := range clientSecret {
        entries = append(entries, Entry(fmt.Sprintf("%s", endpointName), clientSecret.OrgID))
    }

    return entries
}

We then have our test:

var _ = DescribeTable("Getting campaigns by Org ID - All details should be correct",
    func(orgID string) {
        getCampaignsResp, err := client.GetCampaigns(context.Background(), &ourapi.GetCampaignsParams{ClientID: &orgID}, automation.AddAuthorizationHeader(authToken))
        defer getCampaignsResp.Body.Close()
        Expect(err).To(BeNil())
        Expect(getCampaignsResp.StatusCode).To(Equal(http.StatusOK))

        getCampaignsRespBody, err := ourapi.ParseGetCampaignsResponse(getCampaignsResp)
        Expect(err).To(BeNil())

        if len(*getCampaignsRespBody.JSON200) > 0 {
            confirmCampaignsAreCorrect(*getCampaignsRespBody.JSON200)
        }
    },
    generatePackageNameTable("GetCampaigns"),
)

But when we go to run the test Ginkgo doesn't recongnise it as a spec, and our run shows 0 out of 0 specs run. In a dry run debug we see reason to believe that this is because the entries are being generated in the Tree construction phase, which is before the run phase, hence the clientSecrets slice is empty.

What we'd therefore like to know is if there's a way to alter the tree construction phase so that we can build this dynamically? This approach which we have works in our other services, but that's because we're hard coding the data for those suites as it's not sensitive.

Many thanks in advance.

onsi commented 4 months ago

Do you know len(clientSecrets) in advance? If so then you can construct the table passing in a counter for each entry that represents the index into clientSecrets which you construct dynamically in SynchronizedBeforeSuites. If you don't know len(clientSecrets) ahead of time then you won't be able to do this. Ginkgo doesn't support dynamically adding specs after tree generation time. At that point your options are:

tjs1989 commented 4 months ago

Do you know len(clientSecrets) in advance? If so then you can construct the table passing in a counter for each entry that represents the index into clientSecrets which you construct dynamically in SynchronizedBeforeSuites. If you don't know len(clientSecrets) ahead of time then you won't be able to do this. Ginkgo doesn't support dynamically adding specs after tree generation time. At that point your options are:

  • fetch the client secrets in an init (though now every Ginkgo parallel process will fetch the secrets) and use that to construct the tree in the init.
  • overprovision the number of tests (e.g. if len(clientSecrets) < 100 you could just have a hundred entries and simply skip the ones that don't apply)
  • put it all in a single It.

Thanks for the prompt reply Onsi :)

Unfortunately we don't know the len of clientSecrets in advance. I think that I'll go down the init road as it feels like a fair trade off in order to ensure that we keep the use of running things in parallel, plus I don't think that having it all in a single It will ever get past a PR review here :)