run sample in pipeline

[sam-1pass]

Add npm run commonjs-example command to the pipeline. This is to ensure that changes to the code don't break the example.

[AndyTitu]

I have 2 questions:

Why do we want to run the example in the pipeline? I think we achieve the exact same coverage (and even more) with the integration tests.

• If you want to test the packaging of the JS SDK - as this is the only value I see from attempting to run an example - there are a few other steps that need to be taken. That's a larger discussion and I'll wait for an answer before kicking it off.

If we have a good reason to introduce the example to the pipeline, why don't we just change the secret reference in the example, as opposed to creating a new service account token for that weird, general secret reference which is meant for example purposes.

[sam-1pass]

I have 2 questions:

Why do we want to run the example in the pipeline? I think we achieve the exact same coverage (and even more) with the integration tests.

When I pushed changes that separated the client file into static and generated content, the pipeline erroneously passed despite the example command npm run commonjs-example failing. This alone may be reason enough to include running the example command in the pipeline.

If we have a good reason to introduce the example to the pipeline, why don't we just change the secret reference in the example, as opposed to creating a new service account token for that weird, general secret reference which is meant for example purposes.

This is a good point. I overlooked the easier solution 😅

[hculea]

If we have a good reason to introduce the example to the pipeline, why don't we just change the secret reference in the example, as opposed to creating a new service account token for that weird, general secret reference which is meant for example purposes.

The example should first and foremost serve as an example, intuitive for new users of the SDK. Having an op://5edut84whnc8ut4cnw4hu8cw4/xw4cnge5nucnwhu4chnu4/e5c4wgnfgnhuewcnuw4nuh reference in there is not intuitive. With a ~5min time investment, we can achieve both the goals of having the example be intuitive as well as validating at all times that the exports are configured correctly, for regression testing.

[sam-1pass]

Should a proper test suite cover at least the same functionality as the example? If the test cases are passing and the example is failing, I think we should update the test cases. That being said, there's no harm in running the example in the pipeline as well, but it doesn't feel like a good single point of failure to rely upon as its outside the test suite.

[AndyTitu]

When I pushed changes that separated the client file into static and generated content, the pipeline erroneously passed despite https://github.com/1Password/onepassword-sdk-js/pull/15#discussion_r1542466614. This alone may be reason enough to include running the example command in the pipeline.

Okay, I see, so the reasoning is in line with the bullet point I added to my comment above.

Here are some additional thoughts:

• Are we trying to test that the changes in a current PR don't break the packaging? In that case we'd need to add a local NPM dependency to the package.json of the examples submodule. If we do that statically in the code, the examples won't be an accurate representation of what a user does irl (a user will always download from npm instead of cloning this repo and use a local dependency).

Having an op://5edut84whnc8ut4cnw4hu8cw4/xw4cnge5nucnwhu4chnu4/e5c4wgnfgnhuewcnuw4nuh reference in there is not intuitive

• Is there any reason why we can't use the actual vault and item names? op://Test/RogerLogin/password seems even better than op://vault/item/field in terms of UX.

• The 2 above points indicate that we're trying to repurpose the examples for some internal use case, when they are only meant to be some representative snippets for external users. In that case, I'd try and create a separate submodule for testing packaging, or attempt to dynamically modify the package.json/secret reference of the examples in the pipeline.

I definitely don't think creating an additional SA token is the best solution though.

[hculea]

The value in adding the example as a test is twofold:

• we use it as a form of regression testing, confirming that the exports are configured correctly to make sure we capture bugs such as the one we initially overlooked in https://github.com/1Password/onepassword-sdk-js/pull/15
• we confirm that the (most likely) first point of contact of our customers with the SDK's functionality presents at all time a valid, working example. In the Go SDK, for example, this has not always been the case - as changes to the schema occurred, our example package sometimes lagged behind in being updated.

[hculea]

I'd like to be mindful that we don't spend more time debating what the best solution is then actually going about implementing it, since this change will nonetheless be beneficial both to the users and to us, regardless if which way we decide to go.

I am okay to use op://Test/RogerLogin/password as a secret reference. Let's add, as a comment in the example crate, a link to the secret reference syntax: https://developer.1password.com/docs/cli/secrets-reference-syntax/ if we decide to go about it this way.

[AndyTitu]

@hculea I went through this PR again.

Here is the issue I'm concerned with: as it currently stands, if we merge this PR, every future PR will run the code example for the same version beta.6. This indeed tests that beta.6 runs ok in the code example, BUT:

1. it doesn't make sense to run this test in absolutely every PR. If beta.6 ran okay once, it will continue to run okay, even if we run the test multiple times. it seems wasteful to test the examples with the exact same statically defined version as a dependency
2. as it currently stands, we won't test the latest version of the sdk, i.e. the version that includes the changes from the PR that's being opened. To test against regressions we need to test the latest version, not a version from the past.