Why supergraph.config.js is expected to run fedtest for remotely hosted endpoint(subgraph)

[arunarivarasan]

My use case is to test federated compatibility for the endpoint (subgraph) hosted in the remote. so i no need to start subgraph through supergraph.config.js as this is already running. when i tried to run "npx fedtest pm2 --endpoint https://sfcluster.eastus.cloudapp.azure.com:4001/graphql" it is expecting supergraph.config.js. i want to know the way to skip reading config file. Also by default, it is loading supergraph.config.js and during test it is looking for user and inventory subgraph.

error: 2023-01-09T22:17:47.372Z ** 2023-01-09T22:17:47.374Z Starting Apollo Federation Subgraph Compatibility Test 2023-01-09T22:17:47.375Z ** 2023-01-09T22:17:47.375Z starting supergraph... C:\Users\aarivarasan\code\c3master2022\C3\node_modules\execa\lib\error.js:60 error = new Error(message); ^

Error: Command failed with exit code 1: pm2 start supergraph.config.js [PM2][ERROR] Error: Script not found: C:\Users\aarivarasan\code\c3master2022\C3\Usersaarivarasancodec3master2022C3
ode_modules@apollo ederation-subgraph-compatibility-testsdist\subgraphs\inventory\index.js [PM2][ERROR] Error: Script not found: C:\Users\aarivarasan\code\c3master2022\C3\Usersaarivarasancodec3master2022C3
ode_modules@apollo ederation-subgraph-compatibility-testsdist\subgraphs\users\index.js [PM2][WARN] Applications subgraph inventory, subgraph users not running, starting... at makeError (C:\Users\aarivarasan\code\c3master2022\C3\node_modules\execa\lib\error.js:60:11) at handlePromise (C:\Users\aarivarasan\code\c3master2022\C3\node_modules\execa\index.js:118:26) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async startSupergraphUsingPm2 (C:\Users\aarivarasan\code\c3master2022\C3\node_modules\@apollo\federation-subgraph-compatibility-tests\dist\startSupergraph.js:52:9) at async compatibilityTest (C:\Users\aarivarasan\code\c3master2022\C3\node_modules\@apollo\federation-subgraph-compatibility-tests\dist\compatibilityTest.js:15:28) { shortMessage: 'Command failed with exit code 1: pm2 start supergraph.config.js', command: 'pm2 start supergraph.config.js', escapedCommand: 'pm2 start supergraph.config.js', exitCode: 1, signal: undefined, signalDescription: undefined, stdout: '[PM2][WARN] Applications subgraph inventory, subgraph users not running, starting...', stderr: '[PM2][ERROR] Error: Script not found: C:\Users\aarivarasan\code\c3master2022\C3\Usersaarivarasancodec3master2022C3\n' + 'ode_modules@apollo\federation-subgraph-compatibility-testsdist\subgraphs\inventory\index.js\n' + '[PM2][ERROR] Error: Script not found: C:\Users\aarivarasan\code\c3master2022\C3\Usersaarivarasancodec3master2022C3\n' + 'ode_modules@apollo\federation-subgraph-compatibility-testsdist\subgraphs\users\index.js', failed: true, timedOut: false, isCanceled: false, killed: false }

[dariuszkuc]

Hello 👋 Which version of the package are you using (based on the file names it looks like an old version)?

Just tried it locally and below worked for me

npx @apollo/federation-subgraph-compatibility@0.0.14 pm2 --endpoint http://localhost:4001

Thanks, Derek

[arunarivarasan]

Thanks for quick response. i use 0.0.14 version in my package.json. still getting same error. it is looking for supergraph.config.js

[dariuszkuc]

There is something odd in your install - it looks like there is a missing / between federation-subgraph-compatibility-tests and dist folder in the generated supergraph.config.js.

Script logic is pretty simple -> given a template file we replace ${DIST_DIR} with current script location (resolve(__dirname)). I am unsure why __dirname would return you invalid path.

Looks like you are using Windows? Maybe logic has to be updated to account for Windows paths (still unsure why there would be missing path separator there)?

[arunarivarasan]

This is my generated supergraph.config.js

module.exports = { apps : [{ name : "subgraph inventory", script : "index.js", cwd: "C:\Users\aarivarasan\code\c3master2022\C3\node_modules\@apollo\federation-subgraph-compatibility-tests\dist/subgraphs/inventory", wait_ready: true },{ name : "subgraph users", script : "index.js", cwd: "C:\Users\aarivarasan\code\c3master2022\C3\node_modules\@apollo\federation-subgraph-compatibility-tests\dist/subgraphs/users", wait_ready: true }] }

yes i am using windows.

Also along with the missing '/\' , not sure how this folder name 'Usersaarivarasancodec3master2022C3' added in the path as this is not a valid folder in my solution.

C:\Users\aarivarasan\code\c3master2022\C3\Usersaarivarasancodec3master2022C3\n' + 'ode_modules@apollo\federation-subgraph-compatibility-testsdist\subgraphs\inventory\index.js

Also we use nx monorepo for build and packaging.

one more question to this porblem, why supergraph.config.js is created from template, as we already have endpoint which has to be test not user or inventory right ?

[arunarivarasan]

any fix required to solve this ?

[dariuszkuc]

any fix required to solve this ?

@arunarivarasan first I need to be able to reproduce it, it might be some Windows issue (potentially with PM2) or it could be a bad installation. Does using docker command works?

[arunarivarasan]

Thanks. we don't have support for docker in build pipeline so trying to get this working using pm2. Also can you share the docker command information

[dariuszkuc]

Hello 👋 I'm hoping https://github.com/apollographql/apollo-federation-subgraph-compatibility/pull/332 will fix this issue -> released v1.0.1

[arunarivarasan]

Thanks @dariuszkuc. Now path is correct. Thanks for the fix. I have one more question just to make sure i am using this package correctly, when i ran this command, yarn fedtest pm2 --endpoint https://localhost:4001/graphql it is creating supergraph.config.js which has default subgraph users and inventory and it is trying to start that and getting failed.

error: Error: products failed to start at composeDevSubgraph (C:\Users\test\node_modules\@apollo\federation-subgraph-compatibility-tests\dist\composeSupergraph.js:71:15)

My question is, why it is downloading supergraph.config.js and trying to start users and inventory subgraph.

My impression of this command is,

when i run the below command yarn fedtest pm2 --endpoint https://localhost:4001/graphql it will introspect the schema from that endpoint and test for fed compatibility

is my impression wrong ? what is the right way to run only the endpoint and start endpoint for users and inventory.

Also we use this command during build time to test the schema. so is it possible this package will validate the schema file only without endpoint.

e.g yarn fedtest --schema ./resources/test.graphql

without endpoint argument

[dariuszkuc]

From script usage:

@apollo/federation-subgraph-compatibility test script starts a supergraph that consists of your subgraph implementation, two reference subgraphs and Apollo Router. Since script needs to manage multiple processes, you need to specify which process management technology should be used to run the tests. Currently script supports PM2 and Docker Compose.

Executable NPM script invokes common logic from @apollo/federation-subgraph-compatibility-tests package (also used by Github Action). That package should contain both reference subgraph implementation.

Your error says products failed to start (as in your subgraph) -> make sure your subgraph is up and running before running it with PM2 (or specify PM2 ecosystem file that will start it for you). Our healtcheck is very simple -> it just runs query { __typename } (maybe those queries are disabled in your server?).

[arunarivarasan]

Thanks for the response. i have a subgraph hosted in different web server. not in the server where router is running. I want to test that subgraph for compatibility before i compose supergraph during build time. My understanding is, eventhough you have your own subgraph. My understanding from your comment is, eventhough we want to test subgraph "x", this package will expect to run users and product subgraph as default along with the subgraph "x". I am looking for something which only test my subgraph "x" not user and product. Also is there are way to test subgraph with only schema file without endpoint ? currently i see endpoint as mandatory arg.

[dariuszkuc]

This test package verifies compatibility against federation subgraph spec and it is intended for library authors to verify whether their library conforms to the spec. This package is not supposed to test fed compliance of any arbitrary subgraph -> if your service uses supported lib, then it is assumed it is (and you get composition errors if it isn't).

• In order to be able to consistently verify the features, we require a well known expected schema (i.e. products).
• Fed compliance tests require running subgraph as we need to verify the behavior - e.g. during @requires test router also sends additional required fields which should be used for computation (and yes you could hard code the values and fake the results but we have to start somewhere)
• While some tests can be executed directly against a target service, in order to verify full fed compliance we NEED to run the tests against a supergraph (which includes router + other well known schemas, see example above)
• We cannot test against arbitrary schema as we would need some complex logic how to determine appropriate test queries (e.g. how to construct _entities queries and what should be expected input/output)

[arunarivarasan]

Thanks @dariuszkuc