Run Nx dependency graph backwards (leaves first)

HadiModarres commented 2 years ago

Description

I need to run certain command for my packages in a dependency graph, but backwards. So the tree of dependencies will have commands run around the edges at the leaves first the moving to the root.

Motivation

To successfully remove the deployed stacks on AWS we need to destroy stacks that aren't depended on first. So if I have stacks that depend on each other like: stack1 -> stack2 -> [stack3, stack4] I need stacks 3, 4 to be destroyed first then stack2 then stack1. This is the reverse of deployment in which we want stack1 to be deployed first and so on.

seriouscoderone commented 2 years ago

I need this features as well.

my nx.json configures my aws dependencies like this

"targetDependencies": {
    "build": [
      {
        "target": "build",
        "projects": "dependencies"
      }
    ],
    "deploy": [
      {
        "target": "deploy",
        "projects": "dependencies"
      }
    ]
  },

and each project.json for my aws projects specifies adds in the implicit dependencies

i.e.

  "implicitDependencies": ["service-framework-networking", "service-framework-layer-transunion-ssl-layer"]

The above allows me to build and deploy my AWS cdk stacks...

but I want to add a destroy command that will reverse the dependencies, and destroy starting with the leaves

jonathanmorley commented 2 years ago

I think this could be neatly solved by adding support for dependents as a valid value to projects, i.e.

"targetDependencies": {
    "destroy": [
      {
        "target": "destroy",
        "projects": "dependents"
      }
    ]
  }
}

ThomasAribart commented 1 year ago

@HadiModarres @AgentEnder I can try implementing this feature if it's okay for you 👍 #Hacktoberfest

barakcoh commented 1 year ago

we've found that reversing the order of yarn nx affected:apps achieves the desired effect (we're also using it to destroy Pulumi stacks)

ThomasAribart commented 1 year ago

@barakcoh Sure but how did you achieve that ?

barakcoh commented 1 year ago

@ThomasAribart like so

function getAffectedDeployStackApps() {
  return new Promise<string[]>((resolve) => {
    exec(
      'yarn nx affected:apps base=HEAD~1 | grep stack | xargs',
      (_, stdout, __) => {
        resolve(
          stdout
            .trim()
            .replace('- ', ' - ')
            .split(' - ')
            .filter(Boolean)
            .reverse()
        );
      }
    );
  });
}

barakcoh commented 1 year ago

@ThomasAribart apparently, the solution we had in place didn't work reliably. we replaced it with a proper topological sort:

async function destroyStacks() {
  try {
    const adjList = new Map<string, string[]>();
    const affected = JSON.parse(
      (
        await $`yarn -s nx print-affected`
      ).toString()
    );

    const dependencies = Object.values(
      affected['projectGraph']['dependencies']
    ) as Edge[][];
    chain(dependencies)
      .flatten()
      .forEach(({ source, target }) => {
        console.log(`Adding dependency: ${source} -> ${target}`);
        adjList.set(source, (adjList.get(source) ?? []).concat(target));
      })
      .value();

    const sortedStacksWithDependencies = findTopologicalSort(adjList);

    console.log(
      `Destroying stacks with dependencies in order`,
      sortedStacksWithDependencies.join(' > ')
    );

    await $`yarn nx run-many --target=destroy --projects=${sortedStacksWithDependencies.join(
      ','
    )} --args="--infraEnv=${INFRA_ENV} --appsEnv=${APPS_ENV}"`;
}

where findTopologicalSort is an implementation of a topological

FilipPyrek commented 1 year ago

Hi team @jeffbcross @vsavkin @FrozenPandaz @jaysoo, would it be possible to push this feature? 🙏

It's quite important for use-cases when you are using Nx to drive your deployment process which I feel is quite common. 🙂

FrozenPandaz commented 1 year ago

Hey all

In theory, it is reasonable to reverse the task graph. I would love to hear more use cases for this to make sure the solution solves the problem.

To successfully remove the deployed stacks on AWS we need to destroy stacks that aren't depended on first.

Is this something that happens often? What's a more concrete example of when you would do this?

I think this could be neatly solved by adding support for dependents as a valid value to projects, i.e.
"targetDependencies": {
    "destroy": [
      {
        "target": "destroy",
        "projects": "dependents"
      }
    ]
  }
}

I could see this design working but would have to think about it some more. :thinking:

I can try implementing this feature if it's okay for you #Hacktoberfest

I love the enthusiasm but let's gather more info before implementing this.

FilipPyrek commented 1 year ago

Hi @FrozenPandaz 🙂

sounds reasonable. 👍

Let me describe our use-case at @purple-technology:

What we have now

we started in 2019 to build our apps with Lerna monorepo setup - https://github.com/purple-technology/purple-stack
our biggest monorepo now contains 30+ Serverless Framework services and 40+ shared packages and we have couple of a bit smaller monorepos like this
our developers are doing ephemeral deployments of the whole monorepo for every feature they develop and test, so that means that we have to deploy whole big app and remove it every couple of days for each developer.
we created our own extension for Lerna called lerna smart run which helps us to package and redeploy only services which have actually changed while keeping correct order of deployment

What are we migrating to at the moment and why is this issue for us

we are now migrating this old Lerna flow to Nx and doing many other improvements in our pipelines
it's nice that we can set implicit dependencies between every service and take advantage of the caching etc. etc.
in our case the dependency means Fn::ImportValue between stacks, which is a "hard" dependency in CloudFormation, so that's why need to deploy and remove everything in correct order.
Only solution at the moment for us is to create our own script which would build the dependency graph on it's own and trigger the removal Nx scripts in individual services, which is very very crappy solution.

Nx has this beautiful feature of dependency graph and putting there a flag to make the execution in a reverse order sounds like a reasonable feature.

Let me know your any other thoughts on how this could be done in order to meet our needs. 🤔

FilipPyrek commented 1 year ago

Or maybe just some Nx plugin could be sufficient for this 🤔 I saw in the docs it's possible to manipulate the dependency graph from inside a plugin, which could do the job. Let me know what do you think.

ThomasAribart commented 1 year ago

@FrozenPandaz My need is the same as @FilipPyrek

I develop a Serverless application on AWS, with microservices as Cloudformation stacks. We need to remove them from dev accounts every week (this is a policy of the company I work for, to avoid drift). We cannot create a nx script because of the Fn::ImportValue dependency between our core microservice and the others.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it hasn't had any recent activity. It will be closed in 14 days if no further activity occurs. If we missed this issue please reply to keep it active. Thanks for being a part of the Nx community! 🙏

HadiModarres commented 1 year ago

I did the same thing as @FilipPyrek and @ThomasAribart, but I also think this whole approach of using implicit dependencies for correct deployment order is suboptimal as it affects the affected packages reported by Nx, so it'll run scripts like test for all affected packages including ones that I specified in implicit dependencies which is wrong because their source code don't rely on each other to affect tests. I think a possible solution would be to split the concept of dependencies into source dependencies and runtime dependencies, and for runtime dependencies we would have the optiion of setting up which runs the tree root first and tear down which runs the tree leaves first.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it hasn't had any recent activity. It will be closed in 14 days if no further activity occurs. If we missed this issue please reply to keep it active. Thanks for being a part of the Nx community! 🙏

mattzcarey commented 1 year ago

Hey, has any progress been made on this or any further solutions. My usecase is exactly the same as @ThomasAribart and @FilipPyrek, literally to the letter.

Thanks

NathanJAdams commented 1 year ago

Sounds reasonable and would solve a use-case for me too. We have to publish packages, some of which depend on others, if there was say a --reverse-order flag that would be perfect.

leongold2 commented 1 year ago

I second this for the exact same use-case (AWS stacks)

FrozenPandaz commented 5 months ago

I see, that all seems reasonable.

This is kind of complicated but the core team does not have bandwidth right now to work on this.

I believe reversing the task graph would yield the desired outcome?

It would be a quite a bit of work but if someone from the community would like to try implementing this, I believe somewhere around here would be where we would reverse the graph. I'm not sure if it's that simple though. I'm not sure if we could handle "dependent" task inputs for these kinds of tasks :thinking: It seems like for these scenarios, caching is not required though.

RobertAlblas commented 2 weeks ago

Hi @FrozenPandaz 🙂

sounds reasonable. 👍

Let me describe our use-case at @purple-technology:

What we have now

we started in 2019 to build our apps with Lerna monorepo setup - https://github.com/purple-technology/purple-stack

our biggest monorepo now contains 30+ Serverless Framework services and 40+ shared packages and we have couple of a bit smaller monorepos like this

our developers are doing ephemeral deployments of the whole monorepo for every feature they develop and test, so that means that we have to deploy whole big app and remove it every couple of days for each developer.

we created our own extension for Lerna called lerna smart run which helps us to package and redeploy only services which have actually changed while keeping correct order of deployment

What are we migrating to at the moment and why is this issue for us

we are now migrating this old Lerna flow to Nx and doing many other improvements in our pipelines

it's nice that we can set implicit dependencies between every service and take advantage of the caching etc. etc.

in our case the dependency means Fn::ImportValue between stacks, which is a "hard" dependency in CloudFormation, so that's why need to deploy and remove everything in correct order.

Only solution at the moment for us is to create our own script which would build the dependency graph on it's own and trigger the removal Nx scripts in individual services, which is very very crappy solution.

Nx has this beautiful feature of dependency graph and putting there a flag to make the execution in a reverse order sounds like a reasonable feature.

Let me know your any other thoughts on how this could be done in order to meet our needs. 🤔

We're running into this exact same problem. --reverse-order would fix this problem for us as well.

nrwl / nx