Closed Cuttsy27 closed 2 years ago
As cool as this sounds, it would be nye impossible to do. IMO this would be a different tool (and should probably be built by the maintainers of the specific deps)
I love this idea, but how would the implementation work? There is the possibility to break existing code etc.
Yeah with the current architecture this is not possible since we copy entire files over. If you have clashing files all your current progress would be overwritten.
Trying to think through the steps required here:
That's going to be a little difficult. And by a little, I mean damn near impossible. If anyone can find a way to make this work, I will be absurdly impressed.
First of all I want to say thank you all for the great work and effort you guys put into this. I'm loving this stack so far and I see great potential, I'm definitely using it when I need to do some new project.
The topic of this issue is quite complex stuff and as you said, nearly impossible because once the files go to user land this tool will lose control and won't be able to easily modify those files. With this clarified, I have some ideas to throw into the mix. Sorry if I don't explain something properly as English is not my native language.
Imagine that each addon can define 2 different types of template files:
The addon would perform operations upon installing, which could be copying base files (currently done) and applying patches to certain files (those could be conditionally applied based on installed packages aswell). The installation process could have some sort of flags in order to know whether it's a fresh/initial installation or a later installation of a single addon (though this installation process should always be aware of the already installed addons).
Difficulties:
git patch
. This would easily allow to add/remove lines for existing files (though it's quite strict and maybe difficult to mantain aswell)Extra benefits:
trpc
addon currently has base-context.js
, auth-context.js
, auth-prisma-context
), and just have the base file and then the modifications applied depending on the other installed addons.Another semi-related idea:
runPkgManagerInstall
so it could run the operations, instead than of the current imperative wayI'll keep thinking on this.
To expand a bit the idea with patches, take the template/addons/trpc/base-context.ts
file (I added the async
word to avoid some conflicts)
// src/server/router/context.ts
import * as trpc from "@trpc/server";
import * as trpcNext from "@trpc/server/adapters/next";
export const createContext = async (opts?: trpcNext.CreateNextContextOptions) => {
const req = opts?.req;
const res = opts?.res;
return {
req,
res,
};
};
type Context = trpc.inferAsyncReturnType<typeof createContext>;
export const createRouter = () => trpc.router<Context>();
Then we could have 2 patches:
template/addons/trpc/context-prisma.diff
:
diff --git a/template/addons/trpc/base-context.ts b/template/addons/trpc/base-context.ts
index f66d8d9..b6cbb39 100644
--- a/template/addons/trpc/base-context.ts
+++ b/template/addons/trpc/base-context.ts
@@ -2,6 +2,8 @@
import * as trpc from "@trpc/server";
import * as trpcNext from "@trpc/server/adapters/next";
+import { prisma } from "../db/client"; + export const createContext = async (opts?: trpcNext.CreateNextContextOptions) => { const req = opts?.req; const res = opts?.res; @@ -9,6 +11,7 @@ export const createContext = async (opts?: trpcNext.CreateNextContextOptions) => return { req, res,
- `template/addons/trpc/context-auth.diff`:
```diff
diff --git a/template/addons/trpc/base-context.ts b/template/addons/trpc/base-context.ts
index f66d8d9..36fe3ab 100644
--- a/template/addons/trpc/base-context.ts
+++ b/template/addons/trpc/base-context.ts
@@ -1,14 +1,21 @@
// src/server/router/context.ts
import * as trpc from "@trpc/server";
import * as trpcNext from "@trpc/server/adapters/next";
+import { unstable_getServerSession as getServerSession } from "next-auth";
+
+import { authOptions as nextAuthOptions } from "../../pages/api/auth/[...nextauth]";
export const createContext = async (opts?: trpcNext.CreateNextContextOptions) => {
const req = opts?.req;
const res = opts?.res;
+ const session =
+ req && res && (await getServerSession(req, res, nextAuthOptions));
+
return {
req,
res,
+ session,
};
};
Applying them one after the other:
➜ git apply --3way template/addons/trpc/context-prisma.diff
Applied patch to 'template/addons/trpc/base-context.ts' cleanly.
➜ git apply --3way template/addons/trpc/context-auth.diff
Applied patch to 'template/addons/trpc/base-context.ts' with conflicts.
U template/addons/trpc/base-context.ts
And would yield the following file with some conflicts:
// src/server/router/context.ts
import * as trpc from "@trpc/server";
import * as trpcNext from "@trpc/server/adapters/next";
import { unstable_getServerSession as getServerSession } from "next-auth";
import { authOptions as nextAuthOptions } from "../../pages/api/auth/[...nextauth]";
import { prisma } from "../db/client";
export const createContext = async (opts?: trpcNext.CreateNextContextOptions) => {
const req = opts?.req;
const res = opts?.res;
const session =
req && res && (await getServerSession(req, res, nextAuthOptions));
return {
req,
res,
<<<<<<< ours
prisma,
=======
session,
>>>>>>> theirs
};
};
type Context = trpc.inferAsyncReturnType<typeof createContext>;
export const createRouter = () => trpc.router<Context>();
This possible conflicts would be a problem, not sure how they could be handled during installation.
Sorry to spam, but I have dreamed about this.
Let's asume for a moment that the files are generated using a template engine such as handlebars, ejs, eta... Then the same template would be used to generate the same file with different configs. Asume aswell that in the template variables we can find which addons are being installed so we can add code conditionally.
With this in mind, take that the user does a fresh with just one addon (say tRPC). The templates would yield the files without any code for any other addon. If then the user wants to install a new addon (say Prisma), the installer code would do the following for each installed addon (just tTRPC in this case):
Considerations:
git apply --3way
then those conflicts could be handled by the user with their conflict resolution tool of choice (ex: easily done in most IDEs)For the examples I'm going to use EJS and some simpler code as it was a file in the tRPC addon. Take that we have the following template file in template/addons/trpc/context.ejs
:
import * as trpc from "@trpc/server";
<% if (addons.nextAuth.inUse) { %>
import { authOptions as nextAuthOptions } from "../../pages/api/auth/[...nextauth]";
<% } %>
<% if (addons.prisma.inUse) { %>
import { prisma } from "../db/client";
<% } %>
If the user initially installs just tRPC, this would yield:
import * as trpc from "@trpc/server";
If the user then installs the Prisma addon, following the above described flow we would get:
import * as trpc from "@trpc/server";
import * as trpc from "@trpc/server";
+import { prisma } from "../db/client";
diff --git a/template/addons/trpc/context.ts b/template/addons/trpc/context.ts
index 8e7351d..81efa94 100644
--- a/template/addons/trpc/context.ts
+++ b/template/addons/trpc/context.ts
@@ -1 +1,2 @@
-import * as trpc from "@trpc/server";
\ No newline at end of file
+import * as trpc from "@trpc/server";
+import { prisma } from "../db/client";
Then for step 4 we could encounter 2 cases:
(a). If the user has not modified the file, applying the patch yields no conflicts and we would get the proper merged file:
```ts
import * as trpc from "@trpc/server";
import { prisma } from "../db/client";
(b). If the user has modified the file, let's say to this:
import * as trpc from "@trpc/server";
import * as userLandPackage from "@userland/package";
Then applying the patch would result in conflicts, that would need to be managed by the user:
import * as trpc from "@trpc/server";
<<<<<<< ours
import * as userLandPackage from "@userland/package";
=======
import { prisma } from "../db/client";
>>>>>>> theirs
I hope the above makes sense 🤔
@ochicf I think if we choose to let the user to handle any of the merge conflict by using the patch, it will break the purpose of this project itself. Since the purpose of it is to simplify the process of creating the boilerplate so people can use it and add what they need after that. and by letting the user resolve the merge conflict manually will make it much painful to the user itself since probably they will have a ton of merge conflict.
@krsbx you have a really good point. Maybe I wasn't clear enough, but with my last comment I was disregarding my own idea of patching files when creating the project the first time. Hence, creating the initial boilerplate would result in a conflict free project, as we would be just using a template engine to handle conditionals and other injected stuff in the generated files.
But to solve the problem of incrementally add and modify the files once the project is created, I cannot think of anything that could ensure that we don't potentially get conflicts because of the freedom by the user to modify the files freely. My proposal only would yield conflicts when installing addons afterwards, and if the user has modified some of files affected by the new install... which IMHO it's still better than have to do all modifications manually.
Of course It's far from ideal and would add other problems and complexities, I was just trying to throw ideas into the mix to see if we find a way to handle this and it's worth implementing it (which I have some doubts ATM).
So you really got me thinking about how this could be possible. My curiosity has gotten the better of me and I am experimenting with using jscodeshift
to build an AST we can parse, and update, to add in new packages. Will report back if I can get anything even remotely working.
Could also look at astro's way of doing their astro add <integration>
command. it lets you incrementally add integrations and modifies the necessary files:
I've been playing a bit with the jscodeshift
and managed to modify the files during initial installation. For now I've focused on testing the codemods to see if it's the right approach, so then we could implement the script to install addons after the initial installation.
I've pushed some code in my fork with a test just for the trpc/base-context.ts
file. See diffs with main.
Some highlights:
trpc/base-context.ts
transform
suffixed file matching the name of the template, in this case trpc/base-context-transform.ts
usingPrisma
and usingAuth
). Other options could be passed depending on the codemod. Note that this is a great tool also for handling relative imports, as instead of being "harcoded" in the template we can dynamically resolve them during installationWould love to hear your thoughts.
EDIT: added a commit fixing an import, for some reason when running the transform
file in jscodeshift thread does not find module if imported with .js
extension
I've pushed more changes. Some comments:
Now that I have worked out the codemods stuff (and I'm quite happy with it) I'll do an attempt of allowing installation of new packages. Will post here if I make any progress.
I'm also not sure If I have to keep spamming this issue or maybe create a PR to avoid notifying everyone?
Very interesting stuff you have made. I gave up on codemods quite early because it seemed really overcomplicated for what I initially wanted it to be (maybe there is a market for a better tool here @MWhite-22 :smiley:)
Will post here if I make any progress.
Sounds good. If you get it to a working point without it being too complex for contributors to use I would gladly incorporate this into the existing installers, don't know about you @nexxeln?
I'm also not sure If I have to keep spamming this issue or maybe create a PR to avoid notifying everyone?
I don't think anyone minds the "spam", the findings are very interested (I'll admit the initial "git --diff
" approach was beyond my knowledge 😉). With that said, when you get to a somewhat working point, perhaps a draft PR is more appropriate?
Super appreciate the work you have put into this. I don't know if this issue's original point (adding dependencies to an existing project) is something we want to incorporate (I think this is beyond a scaffolding CLI which this is, please confirm @TheoBr & @nexxeln), but this could definitely be used for the scaffold process!
Yeah we can definitely include it in the scaffolding, as for the installing deps thing, that can be made into another package?
Also this is really exciting stuff!
Thanks for your kind words!
I gave up on codemods quite early because it seemed really overcomplicated
Yes the codemods literally blew my mind initially 🤯 it feels such a powerful tool though.
but this could definitely be used for the scaffold process!
My thoughts exactly. IMHO at least it's worth just for conditionally apply changes in the initial installation.
I think this is beyond a scaffolding CLI
To be honest with you, I'm really interested into looking for a potential solution for this because the starters/CLIs I've used are consumed in a way that your created project becomes a snapshot of that starter/CLIs in that point in time and then you are on your own (it's difficult to add updates/new features added in starter). And personally I've been into situations where I'm maintaining several projects created from the same starter that they require a lot of time to keep updated.
Jumping back to our issue at hand, I'll try to break down the next decisions we need to take:
create-t3-app
I was thinking that running the create-t3-app
from a project created with it should detect it and prompt different options than when creating a new project. Initially we would allow to install new packages, but new features could be potentially added (example: update installed packages).
This feels like a solution relatively simple to implement and easy to consume, as we would remove the need to have different CLIs or commands, but let me know your thoughts on this.
create-t3-app
AKA: how to detect if it's a t3-app
I see 2 detection approaches:
t3-app
information somewhere we can read afterwards. We want to store information such as the version of create-t3-app
it was installed from, packages installed, etc. We could store it in package.json
under a t3app
property or in a new file .t3apprc
.create-t3-app
is run. For the project I don't know how to detect it, but for each package would be possible (check if certain files added by the package exist)I think option (1) is much easier and clearer. The main drawback is that users will be free to change the package.json
(or whatever file we store the state of the installation) and it would cause to yield different results or even malfunction. Note that future modifications of this data should be commited by the user so missing that could cause issues aswell.
When selection a new package we must:
Step 1 is pretty straightforward: can behave as if it was the initial installation
For step 2, I see some complications:
templates
to avoid replacing userland codeIn order to properly comply with the above, we need to track which packages are being installed. This is actually what the PkgInstallerMap[keyof AvailablePackages].inUse
property does. IMHO we should rename this property to installing
and add installed
for the packages already installed. For example, imagine that we install trpc
on initial project creation. Later we want to install prisma
, so the packages
object passed to the installers would be:
{
trpc: { installed: true, installing: false },
prisma: { installed: false, installing: true },
}
This would allow the trpc
installer to run conditional logic:
// in /addons/installers/trpc.ts
if (packages.trpc.installing) {
// case [a]: initial installation of the package, copy templates files
}
if (packages.prisma.installing || (packages.trpc.installing && packages.prisma.installed)) {
// case [b]: prisma is being installed OR trpc is being installed and prisma was already installed
// apply codemods to trpc files with prisma modifications
}
Note that some codemods would still be applied once (ex: adding the same import or adding a variable declaration should be skipped if already there).
Sorry for the wall of text. I'm probably missing ton of things but with the above tackled we could have a good starting point.
Hey! Just wanted to give my input on this because I actually built a similar project a while ago called next-gen which has this exact feature. It's currently unmaintained because I don't have the capacity, oh well..
Imo using a library like jscodeshift will be absolutely indispensable for this, the other approaches I tried were way too brittle, It's a bit complicated to get started, but once you get the hang of it it will make the whole process a lot easier.
Blitz does a lot of things with jscodeshift, they've written a really nice abstraction for creating your own new recipes as well, you should take a look: https://github.com/blitz-js/blitz/tree/canary/recipes.
The nice thing about the jscodeshift approach over templates for scaffolding is that you immensely cut down on combinations of templates. I.e. you won't need a _document.tsx
version for withPackageA, withPackageB, withPackageA+B, ... You can just write the codemods and sequentially apply them. This was actually the reason why I initially used them for next-gen, because I wanted to support as many different combinations as possible.
I'm also in the discord, hmu if you have any questions @ochicf .
I've created a new branch in my fork (created from the codemods branch) where I'm handling all of these reinstalls/incremental feature addition. IMHO it would be best to first migrate all existing code to apply codemods, so I wanted to keep my original branch branch with just that work so we can work on that independently. @biowaffeln this is where your knowledge of codemods and the references you gave will be super helpful ❤️
Now I'll quote some parts of my previous comment that I could work on:
(b) how to detect if a project was created using create-t3-app
Did some basic implementation to store the installation info in package.json
.
Some notes:
createdAt
and updatedAt
because I can, not because they really matter 😆 installed: false
(c) what the process of installation of a new package should do [...] This is actually what the PkgInstallerMap[keyof AvailablePackages].inUse property does. IMHO we should rename this property to installing and add installed for the packages already installed. [...]
I've done this, but I've kept the inUse
for now, being:
const pkg: AvailablePackages = 'tailwind';
const installed = projectPkgJson?.createT3App?.packages?.[pkg]?.installed ?? false;
const installing = packages.includes(pkg);
const inUse = installed || installing;
I'll continue working on the reinstalls/incremental feature addition. My goal is to "close the circle" and have it implemented for at least a combination of 2 packages (ex: install 1 first, then install the other and expect all the changes to apply).
moved all the codemods to a different folder in root and configured it as a project reference. I'm not an expert with TS but I had to do this for everything to work smooth. See readme for more info: template/README.md#why-a-sub-project
I see potential here for a separate package that's usable in other projects as well. We can create a new repo under t3-oss if you're down and have it as a dependency for create-t3-app. That way other projects can use it as well. What do you think?
I see potential here for a separate package that's usable in other projects as well. We can create a new repo under t3-oss if you're down and have it as a dependency for create-t3-app. That way other projects can use it as well. What do you think?
That would be so cool. But to be on the safe side maybe I would keep it as a referenced project during the the refactor and make it a separate package afterwards?
Not sure if it's the best approach but I'm feeling such a noob with codemods that I feel it's too ambitious to open that package from minute zero 😆
An important thing though: there will be codemods super specific with the templates of this project so maybe those should stay in the project itself (which is closer to the actual files being modified), and leave the external package for helpers/common codemods?
@juliusmarminge answering https://github.com/t3-oss/create-t3-app/pull/208#issuecomment-1188816432 in this issue.
I am more than happy to jump in and work on some parts of this. Is the basic transforms done? In that case we can merge them in here somewhere and I (and perhaps others) can start working on some scaffolding stuff maybe?
I only applied transform for a single file as a proof of concept trpc/base-context.ts
.
Should we maybe create a new issue regarding codemods and regroup there? This way we separate from this, which needs the codemods done.
Also in order to properly split the work:
Here you have a (semi)detailed list of the affected files. I tried to keep simple with symbols and such but... I don't know if it's clear as I'm doing this with my eyes half closed 😴
(you can search the codemod key to see affected files in the wall of text above)
/transform/base/src/server/env-schema.mjs
/template/base/src/server/env-schema.mjs
/transform/addons/next-auth/api-handler.ts
/template/addons/next-auth/api-handler.ts
/transform/addons/trpc/context.ts
/template/addons/trpc/context.ts
/transform/addons/trpc/index-router.ts
/template/addons/trpc/index-router.ts
/transform/base/src/pages/_app.ts
/template/base/src/pages/_app.ts
/transform/base/src/pages/index.ts
/template/base/src/pages/index.ts
BTW @juliusmarminge I don't have right to modify the GitHub project (cannot add items/tasks to it)
BTW @juliusmarminge I don't have right to modify the GitHub project (cannot add items/tasks to it)
Now you do
we can use my fork's branch as starting point? how could multiple people push there?
FWIW, I think we should target the @next branch so we dont do something that will have to be changed in the near future.
Sorry for the radio silence, I'm currently swamped with work as we are trying to deliver a lot of things before the holidays.
On monday we did a call with @biowaffeln to put things in common and he has done a great job and managed to advance a lot with the codemods. He has a lot more experience with those so he will submit a MR from his fork and we can review and test it. @biowaffeln let me know if you need anything from me ;)
Once that's up we will have the base we need to keep working on incrementally adding features.
Finally some update on this. I'm making some progress on moving this project towards using git patches instead of copying and pasting files. I have a partially working project (has tailwind and trpc). If anyone wants to test this, you could run:
git clone https://github.com/minsk-dev/create-t3-app-template
cd create-t3-app-template
git checkout pkg/tailwind
git format-patch main -o patches -U1
git checkout pkg/tailwind+trpc
git format-patch main -o patches -U1
git checkout pkg/trpc
git format-patch main -o patches -U1
git checkout main
git checkout -b dev
git apply patches/0001-feat-tailwind-adds-boilerplate.patch
git apply patches/0002-feat-tailwind-adds-dependencies.patch
git apply patches/0003-feat-tailwind-adds-imports.patch
git apply patches/0004-feat-tailwind-updates-index.patch
git apply patches/0001-feat-trpc-adds-boilerplate.patch
git apply patches/0002-feat-trpc-adds-dependencies.patch
git apply patches/0003-feat-trpc-adds-api.patch
git apply patches/0004-feat-trpc-finishes-setup.patch
git apply patches/0005-feat-tailwind-trpc-adds-query.patch
echo "patches" >> .gitignore
git add .
git commit -m "chore: inits project"
pnpm install
pnpm dev
Going to create some quality of life for developing with this template, but this is very doable.
moving this over to https://github.com/minsk-dev/t3-cli/issues/2. if no one opposes
Functionality to run create-t3-app in an existing t3 project to add a missing feature, ie. Next Auth, if it wasn't included in original project init.
Adds to the "only use what is needed" axiom while continuing to use t3 boilerplate when enhancements are required.