search

wasp-lang / wasp

The fastest way to develop full-stack web apps with React & Node.js.
https://wasp-lang.dev
MIT License
12.93k stars 1.15k forks source link

Improve how we generate code (especially SDK) during `wasp build` #1769

Open Martinsos opened 5 months ago

Martinsos commented 5 months ago

Update: Read the comment below for the latest info

Original description

Since after restructuring we have the line "wasp": "file:.wasp/out/sdk/wasp", in user's package.json, this meant that we also need to change something about how Wasp project is built.

Why? Because wasp build generates code in .wasp/build, while wasp start generates wasp code in .wasp/out. This was done on purpose, because we didn't want those two to mix, since the generated code can be (and is, although not a lot) different.

So wasp start generates framework code + SDK code in .wasp/out, and that code imports and uses user's code from the root of the project. wasp build works the same way, but writes generated framework code + SDK code into ./wasp/build.

The problem, as I said above, is that now, with restructuring, this won't work well for code generated with wasp build because user's code, via its package.json, depends on SDK in .wasp/out/sdk, while it should depend on SDK in ./wasp/build/sdk.

We discussed how to best fix this, and what we did for the moment, is had wasp build write its SDK code into ./wasp/out/sdk instead of ./wasp/build/sdk, overwriting ./wasp/out/sdk. This is a hack, but it is not problematic because currently SDK is exactly the same code be it for wasp start or wasp build -> code is different for generated framework code, but not for SDK.

Also, we did this in a bit hacky fashion in the code: search for ../out/sdk/wasp in the codebase to see what was done.

We should however look into a nicer solution, because this one is, as mentioned, somewhat hacky, and also could be causing some issues if in the future there would appear some differences between the SDK code generated for the wasp start and the one generated for the wasp build.

What seems to be the best idea for the future is to also copy all the user's code into the .wasp/build directory, instead of leaving the user's code as it is. Then, when copying that code, we can rewrite its package.json to have a different path for importing SDK, one that imports the SDK generated by wasp build. One way to do it is to put user's code somewhere next to generated and framework code, update imports of user's code in the generated code as needed, and that should work. Or, another idea is to copy the user's code into .wasp/build, but then not generate the framework and SDK code directly in .wasp/build, but put it two levels lower, potentially in .wasp/build/.wasp/out -> this way we replicate the relative paths among generated code and user's code, and don't have to modify imports of user's code. Maybe there are some better ideas also. What is important here is copying user's code -> this sounds right. It enables us to do any rewrites if needed. It also means that if user does wasp build and then changes something in their user code, those later changes won't be reflected in the built code, as one would expect -> what one expects from wasp build is that whatever is built there is based on the state of the code as it was at the moment of running wasp build, and is standalone, encapsulated.

TODO: Link to a PR where we made the hacky fix.

sodic commented 5 months ago

1 What's the problem?

The new structure in Wasp 0.12.0 includes significant changes relevant to our build process. This comment explores what happened and how we've solved it. All mentioned changes were introduced with:

1.1 Requirements

This is a list of stuff we need for our build to work:

  1. We need the user's src dir inside Docker's build context. The generated code in .waps/build/server imports the user's code directly using relative paths.
  2. We need the user's package.json and package-lock.json files inside Docker's build context. The user specifies dependencies using these two files. We need these dependencies to build the project.
  3. We need .wasp/out/sdk inside Docker's build context. The user's package.json file includes a local dependency for Wasp's SDK (the wasp package), which points to /.wasp/out/sdk/wasp. NPM automatically creates a symbolic link in /node_modules to make this work: https://github.com/wasp-lang/wasp/blob/52a1910a74b961b1d16cad443718b16d9df58eef/waspc/data/Cli/templates/basic/package.json#L4
  4. We must to mirror the project's structure inside the Docker image. The framework code in .wasp/build/server imports the user code from src using relative imports. The directory structure inside the image must match the outside structure for this to work.
  5. The wasp package in the user's package.json file must point to the correct location inside the container. If we break this, NPM won't be able to create a symlink and resolve the dependency.
  6. The build procedure must clean all generated code it uses every time. The cleaning is an extra security measure. It ensures we don't accidentally include anything sensitive in the build.

1.2 Constraints

This is a list of our tools' limitations:

  1. Docker requires .dockerignore to be in the build root directory. It ignores it otherwise (source).
  2. Docker doesn't allow using files and folders outside its build directory/context (source).

1.3 Nice-to-haves

Ideally, we'd want to have every item in this list, but none are necessary:

  1. Docker's build context should be as small as possible (i.e., it shouldn't include any unnecessary files, especially secrets).
  2. We don't want to add new files (e.g., Dockerfile or .dockerignore) to the project's root directory.
  3. We want to avoid manipulating the user's package.json file at the top level.
  4. Our Dockerfile should be as dumb as possible.
  5. wasp build files' location should be completely separate from wasp start files.

2 Proposed Solution

2.1 Docker build root

Because we need several files from the project's root inside the container (see requirements 1 and 2), we can either:

  1. Use the project's root directory as the build root, which implies:
    • A breaking change to our build procedure.
    • Choosing between wish 1 (small build context) and wish 2 (no extra files in the root dir).
  2. Continue using .wasp/build as the build root, which implies:
    • Copying the necessary files (src, package*.json) into .wasp/build.

Decision: We went with Option 2 because it requires fewer and less visible compromises. Here's the change that made it happen:

2.2 The location of .wasp/out/sdk

Because we need .wasp/out/sdk during the build, and because package.json needs to point to its correct location, we can either:

  1. Manipulate the package.json file during the build process.
  2. Stick with the current directory structure and replicate it inside the container (and clean .wasp/out/sdk on each build).

Decision: We went with Option 2 because it was much easier to implement. Here are the changes that made it happen:

2.2.1 Why is manipulating the package.json a problem?

Our build has three distinct parts:

  1. Running wasp build.
  2. Building the server with Docker.
  3. Building the web-app with Vite.

Manipulating the package.json during the build process only solves the problem for step 2: Building the server with Docker.

Both wasp build and Building the web-app with Vite use the top-level package.json and thus rely on the SDK being in .wasp/out/sdk. Therefore, we would have to either:

  1. Create .wasp/out/sdk during wasp build anyway (regardless of what we do when building the server with Docker).
  2. Completely change our build procedure.

3 Conclusion

The current solution satisfies all requirements and almost all nice-to-haves. The exceptions are:

  1. Our Dockerfile should be as dumb as possible - It's quite the opposite since it includes logic to replicate the entire build structure inside the container, and it's all hardcoded.
  2. wasp build files' location should be completely separate from wasp start files - wasp build and wasp start use the same location for the SDK: .wasp/out/sdk

We'll come up with a long term solution that also takes care of these two points, most likely by fully replicating the necessary subset of project in the build folder.