Package paper-land so that it is easy to use

[jessegreenberg]

We want to make it as easy as possible to run paper playground. This means packaging it in a way that it can be launched without using the command line. Putting some notes here now from a quick search on how this is done.

• Static Site Deployment:
  • Build the app and serve it. Doesn't get us all the way there. We are not going to host it anywhere and the user would need to run a server.
• Docker container
  • My understanding is that this offers a way to contain the development environment, but is still difficult for end users. They need to have Docker installed, and maybe other dependencies. They will need to be somewhat tech savvy to get docker running.
• Electron:
  • Can be used to package the app and create an executable that can be run on Windows and macOS.
• Use pkg to package the app.
  • Allows users to install and run the application. Not as much support for macOS though, the result is an .app instead of a .dmg. Apparently this is a bit more difficult to use (but I am not a Mac user so I am not familiar).

I believe Electron with electron-forge (to package into a .exe and .dmg) is the most promising. We also have a little experience with it since we used it briefly for quadrilateral/haptics exploration many years ago.

Apparently, once we have a .dmg and .exe, we can publish it as a release on github. @brettfiedler may already have experience with that.

@brettfiedler does electron seem reasonable to you? Are you aware of any other options you would like us to explore?

I am putting the steps I got from ChatGPT here for future reference. It shouldn't require a lot of code changes, but I do expect troubleshooting and fiddling to get it right.

Packaging your existing project with Electron or Electron Forge may present some challenges, given the complexity and architecture you described, but it is definitely possible. Here are some considerations and steps to help you through the process: Considerations Backend with Express and Node.js: Your Node.js/Express backend will need to run within the Electron main process. Multiple Browser Tabs: This should be supported as you can open multiple BrowserWindows in Electron. Web Directory: Ensure the www directory and other static resources are correctly referenced and accessible within the Electron environment. Database Dependencies: Electron packages need all dependencies, which includes your database and migration scripts. Steps to Package with Electron/Electron Forge Install Electron and Electron Forge: npm install electron electron-forge Update your package.json: Add electron-specific configurations. ```json "main": "main.js", // Entry point for Electron "scripts": { "start": "electron .", "package": "electron-forge package", "make": "electron-forge make" }, "config": { "forge": { "packagerConfig": {}, "makers": [ { "name": "@electron-forge/maker-squirrel", "config": {} }, { "name": "@electron-forge/maker-zip", "platforms": [ "darwin" ] }, { "name": "@electron-forge/maker-deb", "config": {} }, { "name": "@electron-forge/maker-rpm", "config": {} } ] } } ``` Create main.js: This will be the Electron main process script, managing windows and backend services. ```js const { app, BrowserWindow } = require('electron'); const path = require('path'); const express = require('express'); const serverApp = express(); // Your server setup serverApp.use(express.static(path.join(__dirname, 'www'))); require('./server/entry-server.js')(serverApp); // Adjust the server file path as needed let mainWindow; function createWindow() { mainWindow = new BrowserWindow({ width: 800, height: 600, webPreferences: { nodeIntegration: true, }, }); mainWindow.loadURL('http://localhost:3000'); // Replace with your URL mainWindow.on('closed', function () { mainWindow = null; }); } app.on('ready', () => { serverApp.listen(3000, () => { console.log('Server running on port 3000'); createWindow(); }); }); app.on('window-all-closed', function () { if (process.platform !== 'darwin') { app.quit(); } }); app.on('activate', function () { if (mainWindow === null) { createWindow(); } }); ``` Modify Express Server for Desktop Environment: Ensure paths and static content delivery functions appropriately. You might need to adjust routes and asset paths accordingly. Build and Package Your App: Use the Electron Forge commands: npm run package npm run make Distribution After running the packaging commands, you'll get distributable files in the out directory. Distribute these files using GitHub Releases, cloud storage, your website, or appropriate platforms. Conclusion While integrating your existing Express and React-based application into Electron involves several steps and quite a bit of configuration, it is achievable. This will allow you to provide a desktop application that users can run without setting up a development environment, fulfilling your goal for simplicity and ease of use.

[brettfiedler]

Sounds like Electron might be the way to go here. We've used it in the past and we currently use it for other PhET projects I believe (was there a desktop app?). Either way, it sounds the easiest and has certainly worked before.

Any ideas on how we deal with .env file? Should we keep it as a thing that needs to be edited inside of the installation directory or would there need to be any key-value pairs transformed into settings inside the running Paper Playground itself?

[brettfiedler]

We can auto-deploy from e.g., the main branch or just manually release when we feel like it. Can start somewhere before 1.0.0 and set to 1.0.0 at the end of the grant work?

Release Interface on GitHub:


GPT-generated suggested GitHub Actions workflow:

To auto-deploy an Electron app using GitHub Actions, you can set up a workflow that automates the process of building, packaging, and potentially releasing your app across multiple platforms (like Windows, macOS, and Linux). Here’s a step-by-step guide to help you set this up: ### 1. Prepare Your Electron App Make sure your Electron app has a proper `package.json` with all dependencies and build scripts configured. You should also have scripts for packaging and creating distributables (like using `electron-builder` or `electron-forge`). ### 2. Set Up GitHub Actions Workflow Create a `.github/workflows` directory in your repository if it doesn't already exist, and then create a new YAML file for your workflow, e.g., `electron_build.yml`. ### 3. Define Workflow Configuration Here’s a basic example of what your GitHub Actions workflow might look like. This configuration builds and packages an Electron app on Windows, macOS, and Linux: ```yaml name: Build and Release Electron App on: push: branches: - main pull_request: branches: - main jobs: build: runs-on: ${{ matrix.os }} strategy: matrix: os: [windows-latest, macos-latest, ubuntu-latest] steps: - name: Check out source code uses: actions/checkout@v2 - name: Set up Node.js uses: actions/setup-node@v2 with: node-version: '14' - name: Cache node modules uses: actions/cache@v2 with: path: ~/.npm key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }} restore-keys: | ${{ runner.os }}-node- - name: Install dependencies run: npm install - name: Build and Package App run: npm run dist - name: Upload Artifacts uses: actions/upload-artifact@v2 with: name: ${{ matrix.os }}-app path: ./dist/* release: needs: build runs-on: ubuntu-latest if: github.event_name == 'push' && github.ref == 'refs/heads/main' steps: - name: Download Artifacts uses: actions/download-artifact@v2 - name: Create Release id: create_release uses: actions/create-release@v1 env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} with: tag_name: ${{ github.run_number }} release_name: Release ${{ github.run_number }} draft: false prerelease: false - name: Upload Release Asset uses: actions/upload-release-asset@v1 env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} with: upload_url: ${{ steps.create_release.outputs.upload_url }} asset_path: ./*.zip asset_name: app-${{ matrix.os }}.zip asset_content_type: application/zip ``` ### Explanation: - **Triggers**: This workflow triggers on push and pull request events to the `main` branch. - **Jobs**: Defined under `jobs`, this example includes building the app across different operating systems using a matrix strategy. - **Steps**: - **Checkout**: Pulls the code from the repository. - **Setup Node.js**: Sets up the Node environment. - **Cache Dependencies**: Caches `node_modules` to speed up future runs. - **Install and Build**: Installs dependencies and builds the Electron app. - **Upload/Download Artifacts**: Uploads the packaged app as artifacts, which can be used in later steps or jobs. - **Create Release**: If the push is to the `main` branch, it creates a GitHub release. - **Upload Release Asset**: Uploads the built app to the GitHub release. ### 4. Add Secrets (if needed) If your workflow requires environment variables or secrets (like deployment keys), add them in your repository’s settings under **Settings** > **Secrets**. ### 5. Commit and Push Commit your workflow file and push it to your repository. GitHub Actions will automatically start running the workflow based on the events you specified. This setup will give you a robust starting point for automating the deployment of your Electron app via GitHub Actions. Adjust the workflow as needed based on your specific build and release requirements.

[jessegreenberg]

Any ideas on how we deal with .env file

Good question. I think we have options, what is the ideal experience?

1) Just leave as is. Local build does not support ChatGPT and only runs on local file system. 2) As you suggested, let them edit somehow. If the .env can't be usesd then another config file. 3) Add a UI that lets them enter in keys for the AI or DB usage, and access to restricted files. I have seen projects do this, but it could raise security questions - I wouldn't want to enter my keys if I were a developer finding this project for the first time.

Something else?

[brettfiedler]

Yeah, just to note what we currently have as possible variables:

DATABASE_URL=
STORAGE_TYPE=
ALLOW_ACCESS_TO_RESTRICTED_FILES=
OPENAI_API_KEY=
PORT=

The restricted file access should probably be revisited as well, since it is very specific to our database. I'll make another issue for that.

Is it obvious what other config files are for option 2 (json, yml)? I know Minecraft uses json settings/preferences. I think probably the hidden nature of .env makes it most difficult. Makes sense for some of those keys, but as long as it doesn't go to git (.gitignore settings.json or something), perhaps we are ok?

[jessegreenberg]

After a quick search it seems possible to use a config file that the electron app can read without needing to rebuild it, and it seems like we can use whatever format we want.

[jessegreenberg]

Taking some notes on changes necessary (will continue to edit):

• electron-forge would only run if I removed the engine lines from package.json.

  "engines": {
  "node": "9.3.x",
  "npm": "5.6.x"
  }

  Hopefully this doesn;t break something else.

• Ran npx electron-forge import - Failed the first time and had to run it from terminal with admin priveleges. Eventually it succeeded but made significant automated changes to the project structure.
• I created a basic electron main.js file to an /electron/main.js
• The following was added to package.json to point to the main file, and configure forge.

"main": "main.js",
"scripts": {
  "start": "electron-forge start",
  "package": "electron-forge package",
  "make": "electron-forge make"
},
"config": {
  "forge": {
    "packagerConfig": {},
    "makers": [
      {
        "name": "@electron-forge/maker-squirrel",
        "config": {}
      },
      {
        "name": "@electron-forge/maker-zip",
        "platforms": [
          "darwin"
        ]
      },
      {
        "name": "@electron-forge/maker-deb",
        "config": {}
      },
      {
        "name": "@electron-forge/maker-rpm",
        "config": {}
      }
    ]
  }
}

• I renamed the 'start' command to start-forge because we already use start to start the local server.
• Then, I needed to create a build with of the app with webpack. I tried running the webpack build and it asked me to install webpack-cli. It downloaded and installed. Command was this one in package.json: "build": "webpack",. The command got stuck at one point and I had to hit 'enter' to get it to continue (this is a general Windows thing that I have seen on various tasks but it still bit me here).
• When complete, it added a .js file for each entry page (board.js, camera.js, creator.js, editor.js, ... and so on) all in the www directory.
• IT IS NOT CLEAR TO ME THAT THE LAST TWO STEPS WERE NECESSARY.
• Running npm run start-forge seems to just work!! It opens Paper Playground index in an Electron window. Using "Camera", "Canvas" and "Display opens each of those in a new Electron window. It is able to use the remote database, and each Electron window is able to communicate over localStorage - I can successfully add debug programs and see results on the Display page 🥳
• npm run package seems to create win32 build, I can enter the directory and launch electron windows by doubel clicking out/paperprograms-win32-x64/paperprograms.exe
• npm run make seems stuck after ~5 minutes.
• It eventually failed with "The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters". I don't fully understand that. I tried npm prune, no luck.
• My understanding is that this is happeneing because of deeply nested node_modules. A suggestion to use this configuration in package.json fixed the problem.

  "config": {
  "forge": {
    "packagerConfig": {
      "asar": true
    },

HOWEVER, that zips everything into a single file. One drawback with that is that it is not possible to have a configuration file for the user to edit values (like OpenAI keys, or database configurations).

• THis is a promising start but I am having a hard time getting the server to run in the packaged version. Works OK in a test build but absolute file paths get messed up when packaged. Above tests were successful only because I had a server running locally in a separate process.

[jessegreenberg]

I have been having a heck of a time packaging the server-side code for Electron. As a stepping stone I have been trying to bundle the server code with webpack so we have a single file to run within Electron. The node_modules used by this project have made that very difficult. But finally made a breakthrough today, I can bundle the server-side code with webpack if I remove this line:

// app.use( require( 'heroku-ssl-redirect' )( [ 'production' ] ) );

Seems totally safe to remove, and it looks like this is just for an old heroku deployment of the legacy project.

Next is configuring the full-stack electron app to use bundled front end/back end code and start a server from its entry point.

[jessegreenberg]

OK, I have a package that I can run from the root paper-land directory like this:

./out/paperprograms-win32-x64/paperprograms.exe

That is a great step forward!

However, it does not work if I try to run paperprograms.exe if I run from paperprograms-win32-x64. Error is

stderr: Error copying default data: Error: ENOENT: no such file or directory, scandir 'C:\Users\Jesse\Documents\Development\phetsims\paper-land\out\paperprograms-win32-x64\resources\app\server-dist\default-data

A couple of problems to solve 1) Relative path to default-data is a problem. 2) We will need to package the contents of www/ directory into the electron build so that the bundled server can serve relevant files from within the package. 3) The .env file will need to be packaged. I am not sure yet if we will be able to support configurations without rebuilding because of "asar": true 4) There will likely be other problems with relative paths in the existing code. 5) There will be MacOS specific problems to solve (there have been a few Windows specific things already).

@brettfiedler FYI, I am about 9 hours into this - I have made progress but it is difficult to say how much more time this will take.