Support running an app from a subdirectory

[jmorrell]

There is currently no supported way to run a node app from a directory within your git repo. You can work around it most of the time by using git subtree push but that wouldn't work if you have shared code outside of your directory.

A super common situation is to have the following repo structure:

├── shared-code
├── server
└── frontend-app

Being able to direct the buildpack to look in /server instead of / would be helpful for cases like this.

[87vrvk9k]

@jtushman Did you get anywhere? For me it recognises the cached directories but still spends over a minute refetching. (Unless perhaps it's fetching dev dependencies, not sure how the pruning works exactly).

https://pastebin.com/0NQNJJRS

[jtushman]

Hi chris -- I am still working on it.

@jmorrell is working on a PR that will cache yarn/cache. Maybe that will help.

Yeah I really don't get what is happening on the refetching.

what does your node scripts/remove-workspaces.js look like? i am intrigued

[87vrvk9k]

@jtushman Ah okay. That script is what @danielmahon kindly contributed here: https://github.com/heroku/heroku-buildpack-nodejs/issues/385#issuecomment-422981921 (I just added moving the Procfile for the workspace defined by APP_WORKSPACE to the root).

[jtushman]

yeah that script is too much magic for me -- if I need to do that much tweaking I'd probably just docker this up. But I am hopeful that we are getting close.

[acomito]

Netlify supports this nicely. Just sharing their UI here to give theheroku team some ideas.

Netlify provides one simple field, "Base Directory" for declaring which folder to attempt to build. In my case I have a mono repo with realy-frontend and realy-backend.

• realycrm -- realy-frontend -- realy-backend

I want to build my frontend on netlify. So I have

Seems like something heroku could add "pretty easily"

[LawJolla]

Following up on @acomito , Netlify again upped their game on monorepos.

https://www.netlify.com/blog/2019/10/09/launching-monorepo-support-for-netlify-sites/

It's surprising and disappointing that this isn't more of a Heroku priority. I hope that changes.

[danielleadams]

I know this doesn't make waiting for this feature any better, but just wanted to drop a note that we haven't forgotten about this. I've opened an issue with NPM to ask for an update on what their solution will be, since it's been well over a year since they have stated they will have support soon (stated here).

Issue: https://github.com/npm/cli/issues/513

If we don't get an answer in a reasonable amount of time, then we (Heroku) will want to start considering alternatives that don't factor in any npm submodule functionality or support.

[atomkirk]

I've since moved to deploying all my subdirectory apps with heroku docker and its amazing. If this is something you are still following and want something better than subtree push, I highly recommend it.

[bbugh]

@atomkirk I've heard mixed experiences about docker with Heroku, do you have any references for how y'all are set up?

We're about to have to deal with this issue and your reply is timely.

_{Sent with GitHawk}

[atomkirk]

message me at my username at gmail.com and I can explain

[chrissyast]

@jmorrell here's a workaround. It ain't perfect but works...

1. Make sure to add package.json under server and frontend-app. List dependencies (as you normally do) and specify a start script for the server.
2. Create a root package.json file as follows:

{
  "name": "foobar-app",
  "version": "0.1.0",
  "scripts": {
    "postinstall": "npm install --prefix server && npm install --prefix frontend-app"
  }
}

Notice the --prefix argument which tells npm which subfolder to work on.

1. Add the following to your procfile:

web: npm start --prefix server

Hope that helps,

Why do you have the server and frontend-app in the postinstall but only the server in the procfile?

[danielleadams]

@chrissyast The postinstall script will occur during the build after npm install - it's running installs on the front end and back end code.

The Procfile specifies the "start" command for the server. There's only one server per Heroku dyno, and npm start --prefix server will start the server. Users host their assets (front end files) a few different ways... maybe via a CDN, or through their web server, or a separate web server, but that's why there's only one script for Procfile.

[gregory]

removing the yarn.lock did the trick for me :)

[emil14]

I don't really like the suggested solution. I have a Go app and now I have to add useless package.json just to run nodejs in subdirectory. In would be great if we could use Procfile or something like that to point to where package.json is located

[emil14]

BTW I'm also think that many people here are actually trying to deploy their build artifacts generated by grunt or webpack. But we don't keep our generated stuff in git - we have to build it on Heroku or worse - create new deploy-branches and modify .gitignore file to commit a crime. Should be an easy way to deploy stuff ignored by git

[LawJolla]

Heroku losing more ground. 😢

https://vercel.com/blog/monorepos

[ghost]

Why not let the official buildpacks accept an environment variable specifying the root directory? What's wrong with the blunt approach? An official reply from Heroku would be nice as to why this issue is difficult or taking so long to fix.

[danielleadams]

The reason we are waiting is because npm 7 will support workspaces (https://github.com/npm/rfcs/blob/latest/accepted/0026-workspaces.md), which is a solution for subdirectories in Node projects. Yarn already has subdirectory support, and npm will in the near future.

I helped review the npm workspaces RFC, and it seemed that this would be a good solution for Heroku developers. If we find that npm workspaces is not a fit for Heroku users, we can reexamine. We need to make this decision knowing how npm workspaces work with Heroku users so that the buildpack doesn't step on what npm is offering.

[LawJolla]

@danielleadams thanks for your reply and attention. But for a quick vent, neither Netlify, Vercel, nor AWS Amplify needed NPM 7 to do an incredible job supporting monorepos.

I’ve been reworking my servers to work with the server less architecture so I can, with a heavy heart, migrate away from Heroku.

[danielleadams]

@LawJolla I hear you and appreciate your support of Heroku. I'm fully aware of what the other cloud providers have done, but unfortunately, that is not the direction we opted to go.

[ghost]

Huh? What about the other Buildpacks then? Cause this is really an issue that ends up affecting all of them. Perhaps Heroku needs to admit that buildpacks are not suited for this problem and a new approach needs to be found. Maybe some sort of switchboard that lets us map the directories to their runtimes.

[danielleadams]

@StephenCarboni I hear you. You're welcome to open a support ticket to give that feedback, but the topic being discussed on this Issue is only for subdirectories within Node projects, hence the multiple comments of waiting for NPM's version of workspaces to be released.

[metal450]

I've spent the entire work day trying to get a simple monorepo deployed. I'm solely trying to deploy the frontend at this point. I've tried dozens of solutions, so far to no avail. The 2 apparently simplest from this thread seemed to be:

https://github.com/heroku/heroku-buildpack-nodejs/issues/385#issuecomment-291084067 and https://github.com/heroku/heroku-buildpack-nodejs/issues/385#issuecomment-464553270

However in both cases, although it purports to have pushed successfully, accessing the app in a browser yields 404 not found. My project layout is simply: /my-project ...../shared ...../frontend <-create-react-app

I tried accessing the heroku url directly, as well as accessing /frontend. Both cases, 404.

After reading this entire thread and probably 20 other blog posts about this issue, I'm honestly a bit baffled that such a seemingly common setup is so difficult. I'm not using yarn workspaces nor Lerna, just a simple react app that includes code from a shared folder. If anyone could give me some pointers as to specifically why the 2 comments mentioned above are yielding 404s, I'd be appreciated. I figured this final deployment step would be the simplest/quickest, but seems to be the biggest hangup of this whole project so far :(

[metal450]

Second full work day on this. I gave up on Heroku for the frontend, & deployed it to Netlify, which took about 2 seconds & worked immediately. Now I'm stuck with the backend.

I tried @alexpetralia 's idea of putting release: cd backend && npm install && npm run build. When I push, I see it running the commands, so it appears to work. However, it does not. If I login interactively (heroku run bash), none of the packages have been installed in "backend." If I manually run the exact same commands: cd backend && npm install && npm run build, then it works fine. It just doesn't work when they're in the procfile.

I still cannot comprehend that 3 years later this is so hacky & difficult.

[chrissyast]

@metal450 Would Netlify not work for the backend too?

[joserocha3]

@metal450 Can you try to use this build pack? https://github.com/lstoll/heroku-buildpack-monorepo

Make sure to add the monorepo buildback, then add the node js buildpack below it. Also note that you do not use a Procfile with this buildpack.

I have it working with this monorepo setup like yours. Check the server directory. Notice the heroku-postbuild in package.json. https://github.com/joserocha3/startup-super-pack

[metal450]

@chrissyast: Netlify is for static sites (frontends) only. I think you can maybe get backends to work with some modification ("serverless functions"), which I haven't started looking into. I guess I may just have to abandon Heroku entirely. It just still feels...ridiculous that it could possibly be this inflexible.

[metal450]

@joserocha3: Tried it - build failed, it couldn't find source files in /shared. I think this only works if your subprojects are well and truly completely separate, but if you give it a subdirectory that accesses any shared resources in another, it fails. i.e.:

 /project
 |--- /frontend
 |--- /backend   <-Build fails w/ monorepo buildpack if this accesses anything in /shared
 |--- /shared

[metal450]

Finally got my own hacky solution that works. It's similar to some of the others above, but or some reason using the Procfile didn't work at all (I tried using its release:, web:, and a combination of the both - all to no avail). With no Procfile, the root package.json contains only:

 {
     "scripts": {
       "postinstall": "npm install --prefix backend && npm run build --prefix backend",
       "start": "node backend/dist/app.js"
     }
   }

Make sure the backend's package.json's script tag has "build": "tsc" (it's a typescript project).

This works for the backend. I gave up on Heroku for the frontend, which is on Netlify. So between using Netlify for frontend & this for backend, I have a hacked-together working stack, if or until Heroku finally gets around to supporting something as basic as an official way to give it a subdirectory to run.

[umbe1987]

@jmorrell here's a workaround. It ain't perfect but works...

1. Make sure to add `package.json` under _server_ and _frontend-app_. List dependencies (as you normally do) and specify a _start_ script for the server.

2. Create a root `package.json` file as follows:

{
  "name": "foobar-app",
  "version": "0.1.0",
  "scripts": {
    "postinstall": "npm install --prefix server && npm install --prefix frontend-app"
  }
}

Notice the --prefix argument which tells npm which subfolder to work on.

1. Add the following to your procfile:

web: npm start --prefix server

Hope that helps,

Thanks @jmike for the suggestion, it helped me a lot finding a solution for my case!

DISCLAIMER: It's been difficult!

I have a root folder with a server subfolder (that was created with express-generator-typescript), and a client subfolder (created with angular-CLI).

Instead of using postinstall I used heroku-prebuild, which is specific to Heroku and will not install anything when I build locally (I guess).

The other trick was to specify --production=false when I install with npm install, in order to install also the devDependencies.

Finally, heroku-postbuild is added to build client and server every time the app is updated (pushed to github, hopefully this makes sense...).

I managed to deploy to Heroku adding this package.json to my root folder:

root package.json (verbose because I originated it with npm init defaults)

{
  "name": "tips4family",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "start": "cd ./server && node -r module-alias/register ./dist --env=production",
    "build": "cd ./server && npm run build --prod && cd ../client && npm run build --prod",
    "heroku-prebuild": "npm install --prefix server --production=false && npm install --prefix client --production=false",
    "heroku-postbuild": "cd ./server && npm run build --prod && cd ../client && npm run build --prod",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/umbe1987/tips4family.git"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "bugs": {
    "url": "https://github.com/umbe1987/tips4family/issues"
  },
  "homepage": "https://github.com/umbe1987/tips4family#readme",
  "engines": {
    "node": "13.11.0",
    "npm": "6.13.7"
  },
  "dependencies": {}
}

Side note: to make the angular compile in the correct server dist folder, change client's angular.json to (only relevant part):

"options": {
            "outputPath": "../server/dist/public",
            ...

Note that if your server is not using typescript, you might want to use this instead (omit dist):

"options": {
            "outputPath": "../server/public",
            ...

:feelsgood: :feelsgood: :feelsgood:

[kaikuchn]

I wanted to replace an outdated custom buildpack that just installs node & npm for asset compilation in a subfolder (/assets/). And now I'm flabbergasted that I can't use the official buildpack because I can't tell it where to find my package.json

This is annoying :/ I don't know what workspaces are/will be but I'm not even sure if that would fix my problem...

[oswaldoacauan]

For future reference and for anyone that is running a monorepo using NPM workspaces.

You can workaround this issue by using the workspace option inside .npmrc file. This will make all npm commands to respect your workspace option, and that includes the npm ci that is automatically run in Heroku.

If anyone is deploying via Github Actions and using heroku-deploy action you can do something similar to this:

- name: ⬇️ checkout repo
  uses: actions/checkout@v4

- name: 🥁 update .npmrc with workspace option
  run: |
    printf "\nworkspace=\"your-app\"" >> .npmrc
    cat .npmrc
    git config --global user.email "${{secrets.HEROKU_DEPLOYMENT_EMAIL}}"
    git config --global user.name "github-actions[bot]"
    git add -A && git commit -m "updated .npmrc with workspace option"

- name: ♊️ deploy to heroku
  uses: akhileshns/heroku-deploy@v3.12.14
  ...