Example of two package reference application

[kmalakoff]

After getting a development environment working, I've been browsing how to use electron-compile for production.

It looks like there are some other people with questions / issues:

• https://github.com/electron/electron-compile/issues/113
• https://github.com/electron/electron-compile/issues/61

Also, if you look at this project https://github.com/buz-zard/random/blob/master/electron-compile-1/package.json, there are some great examples of how one might set up the build commands, but unfortunately, without a two package reference application, it is hard to see how to split out all of the development scripting from the built application.

I was wondering if someone has figured out how to produce a two package application configuration that isolates development and production? (eg. includes only the minimal node_modules needed for run time and without build scripts). If so, would you be willing to 1) share a stripped down version or 2) outline the approach and I'll prepare a minimal two package reference application.

[MarshallOfSound]

I was wondering if someone has figured out how to produce a two package application configuration that isolates development and production?

Don't use two package.json files. It is illogical and doesn't make sense when compared to other node project methodologies.

package.json files already have built in methods to separate production and development dependencies. dependencies are for things required in prod. devDependencies are for development only dependencies. If you separate your dependencies correctly into those two categories electron-packager (which electron-compile wraps) will automatically remove all dev dependencies from the packaged version.

https://github.com/electron-userland/electron-packager

Be careful not to include node_modules you don't want into your final app. If you put them in the devDependencies section of package.json, by default none of the modules related to those dependencies will be copied in the app bundles. (This behavior can be turned off with the --no-prune flag.) In addition, folders like .git and node_modules/.bin will be ignored by default. You can use --ignore to ignore files and folders via a regular expression (not a glob pattern). Examples include --ignore=.gitignore or --ignore=".git(ignore|modules)".

[anaisbetts]

^^ what @MarshallOfSound said, as long as you keep all of your dev dependencies like electron-compilers set as such, you should be mostly good.

[kmalakoff]

Thanks guys. Great information...

Understood in principle, but it isn't really clear in practice for me yet!

1) Looking at the reference I have at hand: https://github.com/buz-zard/random/blob/master/electron-compile-1/package.json, it looks a little strange in terms of what is in each section and that all of this would get bundled with the application. For example:

• Should "babel-runtime" be in "dependencies". I'm not sure why it is there if the build step compiles everything?
• Should all the script entries in package.json really be included with the packaged application or only the minimal information related to running the production application?

2) If I'm using modules from my private git account:

• can their source repo locations be removed in the bundled package.json files?
• can I process some of the source files in node_modules when they get packaged for production (for example, to remove their source git repos)? I've been trying to read the api to figure this out.

3) I'm converting to electron-compile from an existing multiple package.json project with webpack for all targets (electron and web) where I'm keeping the build pipelines separate.

These sorts of production release processes are driving me to ask about this. I believe from a security / privacy standpoint, electron is very different than when you are running a node application on your own private servers. This is one of the reasons why the two package.json structure exists.

[MarshallOfSound]

This is one of the reasons why the two package.json structure exists.

The only reason the two package.json structure exists is because someone thought it would be a clever way to solve the rebuilding native dependencies problem that people used to have. No one has this issue any more but people are still recommending this structure (for unknown reasons to me).

Should "babel-runtime" be in "dependencies". I'm not sure why it is there if the build step compiles everything?

I'm not 100% sure on this but I think no, it's a dev dependency as in production everything will already be compiled. @paulcbetts can confirm / deny that

Should all the script entries in package.json really be included with the packaged application or only the minimal information related to running the production application?

Most people do include them but there is no reason you can't strip them out yourself as part of a build step. I don't see why you would need / want to though. Most scripts just run gulp or webpack tasks which are useless without the gulp / webpack files which you should ignore in your electron-packager options (--ignore)

can their source repo locations be removed in the bundled package.json files?

Same answer as before, once you run the packager, just modify that package.json file

can I process some of the source files in node_modules when they get packaged for production (for example, to remove their source git repos)? I've been trying to read the api to figure this out.

Same theory as the above answer, once packaged, just go through with some kind of combination of rimraf and glob and delete all .git directories. Although I believe .git is excluded automatically.

[kmalakoff]

Thank you. Based on your responses, I'll clarify my two problems:

1) how to split up a project across multiple package.json files for different targets - look at how react native projects are set up where there is shared application code and separate build processes per platform

I have a separate build pipeline per target platform in separate package.json files currently using webpack on all two (electron and web like Android and iOS projects) and another package.json for the shared runtime application source code, but I'm trying move to electron-compile on the electron one. I have previously broken these engineering problems down into smaller ones and used webpack to create shared application and platform-specific bundles. With the runtime nature of the electron-compile approach, I'm having a bit of problem understanding the directory structures that I need to bear in mind to keep these pipelines separate while sharing some of the application code.

2) how to use the electron-compile api - I know what I want to do, but I'm not sure how to hook into the build processes to implement them

Diagrams and examples of using the api would help. Specifically, a) how the production vs development environments differ / work, b) examples of using the api to solve some of the above problems

[anaisbetts]

Should "babel-runtime" be in "dependencies". I'm not sure why it is there if the build step compiles everything?

It used to be required because it was a support lib for async/await, but now you don't need transform-runtime or babel-runtime anymore, and they hurt startup time. :fire: it altogether

1) how to split up a project across multiple package.json files for different target

I think you just shouldn't do this at all, you just don't need to.

2) how to use the electron-compile api

Here's the actual code we use in the Slack app for this. targetDir is an app that has been electron-package'd, but not yet ASARed:

export async function createCompileCache(targetDir) {
  let appDir = getTargetAppRoot(targetDir);
  let cacheDir = pn`${appDir}/.cache`;
  mkdirp.sync(cacheDir);

  // NB: You could just use NODE_ENV for this too, if ELECTRON_COMPILE_ENV is unset it will take this
  process.env.ELECTRON_COMPILE_ENV = 'production';
  let compilerHost = createCompilerHostFromProjectRootSync(projectRoot, cacheDir);

  await compilerHost.compileAll(pn`${projectRoot}/src`, (x) => !x.endsWith('.d.ts'));
  await compilerHost.saveConfiguration();

  let shouldDeleteRegexes = [
    /\.webp$/i,
    /^fonts$/i,
    /\.html$/i,
  ];

  // NB: We end up double-including some files that are pretty expensive because
  // the cache now includes all content. Nuke some of the worst offenders, but leave
  // a bunch of files that we end up handing to NativeImage and friends that won't
  // be caught up in electron-compile hooks
  for (let x of await fs.readdir(pn`${appDir}/src/static`)) {
    if (!_.find(shouldDeleteRegexes, (re) => x.match(re)))  continue;

    await rimraf(pn`${appDir}/src/static/${x}`);
  }
}

Note that in this code:

// these two are the same
let foo = pn`./foo/bar`;

let foo = path.resolve('./foo/bar');

[kmalakoff]

Thank you for the responses. Let me see if I understand correctly.

1) how to split up a project across multiple package.json files for different target

I think you just shouldn't do this at all, you just don't need to.

Multiple-folder is a common structure for multiple-target applications and I already have a webpack setup for two different targets (electron and web) where this hierarchical pattern has been working well. I'm having difficulty understanding how electron-compile works under the hood so I'm not sure how to configure it for this structure or if it is even possible yet.

Here's a public react-native example with a structure like:

/{root}
  / android
  / app
  / ios
  / node_modules <- for the app
  package.json <- for the app

Here's what electron-builder suggests:

/{root}
  / app
    package.json <- for the app
    / node_modules <- for the app
  / node_modules <- for electron
  package.json <- for electron

I would like to do something like:

/{root}
  / app
    / node_modules <- for the app
    package.json <- for the app
  / electron
    / node_modules <- for the electron target
    package.json <- for the electron target
  / web
    / node_modules <- for the web target
    package.json <- for the web target

Hopefully, this clarifies what structure I am coming from and what I am trying to achieve. Is the problem that electron-compile does not currently support a target per folder approach or is there a better way to support multiple targets? I would prefer not to duplicate and manually synchronize the app folder and shared node_module dependencies across targets.

---

2) how to use the electron-compile api

Here's the actual code we use in the Slack app for this. targetDir is an app that has been electron-package'd, but not yet ASARed:

Thank you for the code samples. It makes sense, but I have some questions about using the api:

Q1: How does the entry of createCompileCache get hooked into the build process and get called?

Q2: How would I write and use a compiler host to let's say replace the "description" field key with let's say the value "secret" in all package.json files found in the appDir and node_modules? (as an example)

Q3: I've looked at the documentation and it is not yet clear on the relationships between the targetDir, appDir, projectRoot, and cacheDir with respects to the build process. What do these lines do when using the electron-compile's api (for example, does saveConfiguration transfer something from the cacheDir somewhere else? What is the getTargetAppRoot function? Where does projectRoot come from? Are projectRoot and appDir the same?) and where are the files finally stored that get used in a production build?

  let compilerHost = createCompilerHostFromProjectRootSync(projectRoot, cacheDir);
  await compilerHost.compileAll(pn`${projectRoot}/src`, (x) => !x.endsWith('.d.ts'));
  await compilerHost.saveConfiguration();

---

I'm sorry if I'm pushing back on the multiple target approach and asking so many questions. This electron-compile approach looks great, but I'm just trying to adapt it to my needs which I realize may be different than originally intended.

[MarshallOfSound]

I don't have a lot of time to respond fully right now but RE:

I would like to do something like:

It looks like you are attempting a mono-repo which is OK in some situations but I think you might benefit more by splitting the three folders into three separate projects and using them as an npm module (possibly private) within each other.

E.g. The web app will be it's own git repo and then the Electron app can use it as a module.

Off the top of my head that structure would work for you

/{projects folder}
  / app {git repo} (has a dependency on the "web" module")
    / node_modules <- for the app
    package.json <- for the app
  / electron {git repo}
    / node_modules <- for the electron target (has a dependency on the "web" module)
    package.json <- for the electron target
  / web {git repo}
    / node_modules <- for the web target
    package.json <- for the web target

How you configure entry points in the "web" module to work correctly inside the "electron" and "app" modules I have no idea atm. You would have to come up with an export methodology that works for you 👍

But as long as you don't have two package.json files for the Electron client you shouldn't have an issue and if you link the web version as a module within the electron project everything should be fine 👍

I'll respond to other things when I get time 😄

[kmalakoff]

Thank you for the quick response.

Related to your initial answer, I'll quickly elaborate on I would prefer not to duplicate and manually synchronize the app folder and shared node_module dependencies across targets while I wait for your update.

The reason I would like the app code directly referenced from the targets is due to the workflow inefficiencies in propagating changes among targets. Also, the problem with using a module for top level application logic is the turn around time because the app module needs to be built, published or checked in, etc to get updates in the various targets, eg. you would be left to a very slow workflow without the benefit of hot module replacement in the web target (and hopefully soon in electron-compile).

How you configure entry points in the "web" module to work correctly inside the "electron" and "app" modules I have no idea atm. You would have to come up with an export methodology that works for you

I already have this working using dependency injection and in the web target, webpack resolve alias to replace node OS-specific modules with browser appropriate ones.

---

But as long as you don't have two package.json files for the Electron client you shouldn't have an issue and if you link the web version as a module within the electron project everything should be fine

To clarify, the web folder contains an independent target from the electron target, eg. built with webpack for release on browsers. Maybe I should have called it browser to be more clear.

Also, electron-specific code and app-specific would have their own node_modules and package.json. I'm hoping that this can be supported by electron-compile, but I totally understand if adding the complexity of multiple targets might not be the right path for you. If not, I'd still like to understand how electron-compile works to evaluate if it makes sense to fork or go back to webpack for both targets.

[kmalakoff]

I'm still interested in this if anyone else is.