search

scaffold-eth / scaffold-eth-2

Open source forkable Ethereum dev stack
https://scaffoldeth.io
MIT License
1.25k stars 783 forks source link

Create a SE2 cli tool / NPX Install #251

Closed carletex closed 4 months ago

carletex commented 1 year ago

Since the core functionality of SE2 it's almost there (we are in the fixing bugs / small improvements phase), I think it's time to start working on a CLI tool for SE2. This is also related to #121 (NPX Install)

Goal

Having a tool that:

Builder workflow

Just writing a rough example of what an end goal for this could look like:

npx create-se-2 my-project
# - It will check dependencies (nodejs >= 18, yarn & git installed)
> Do you want to install the dependencies? (Y/n)
# ......... yarn installing everything (including the CLI). See implementation section
# postinstall: run cli set up. See implementation section
> What smart contract dev framework do you want to use
(*) Hardhat
( ) Foundry
# ... extra config

Now SE2 is installed on myproject. Besides all the chain / deploy / start, you could (yarn could just wrap the se-cli calls)

yarn add-extension
> Which extension do you want to add (Select multiple)
(*) Sign-In with Ethereum
(*) The Graph
( ) Backend API Example
( ) Foundry

Also some utils:

yarn clean # Remove all the extra stuff (index page => empty, remove example UI, all unused stuff)
yarn remove-contracts # Remove the hardhat/foundry folder, an any component that interacts with it on the contract.
// ....

Implementation

There are a bunch of options to go about this, throwing some ideas out there and its reasons.

1. NPX install Could be a separate repo create-se2-app with a simple script that uses simple-git to clone the repo (+ check deps + run yarn install)

-> Reasons

2. Hardhat vs Foundry This is the main piece of configuration for the project, and it's a must-have.

I see two options:

  1. Having both folders in the projects that generate the same hardhat_contracts.ts file (so the front-end doesn't care).
    • CONS: Bigger/Polluted repo. Can be confusing
    • PRO: Easy to manage and create (basically just create the folder and yarn commands)
  2. Having them as separate repos, you select one of them on the ****NPX install and it gets pulled in the corresponding folder.
    • CONS: Mechanics more complicated
    • PRO: Everything is cleaner. More scalability with future config options.

**** I think we'll need to do it through the CLI, and run it on package.json post-install. So people who just clone the repo, also get the opportunity to choose and install (This is telling me that the NPX script just uses the SE2 CLI under the hood.)

3. Extensions

For 1, 2 & 3 we could also use NPX templating, but I think that would make the repo unusable if you directly clone it. And SE2 is a starter kit, not a framework / lib that you install and use (like NextJS)... so I think this is the right approach.

This is ambitious, but we don't need to implement all of these at once. But I feel that deciding the right architecture is key here, since it can be a pain shifting afterward.

Would love to hear everyone's thoughts, ideas, etc on this. I've already started tinkering with it, y'all can do the same! So we'd have more info / formed opinions in a a couple of weeks.

damianmarti commented 1 year ago

It is great that SE-2 has reached this maturity!!! 🎉🎉🎉

I can help build a The Graph extension. Maybe I can start building something using a subgraph and then we can decide how to incorporate it as an extension...

austintgriffith commented 1 year ago

I think there are a lot of 50/50 preferences in the modern eth dev stack and this makes a lot of sense if we want to cover most builders.

It sounds weird, but maybe [ CRA and JS ] vs [ Next and TS ] is one of the options and it forks scaffold-eth 1 vs scaffold-eth 2.

could I even come to this create app and say I need a simple node script and that sets up the project as a type=module and templates in a simple read / write with ethers as a node script?

I know these are lots of random things, but also... as we are cli choosing our stack, being able to select from templates would be really cool... like DEX, multisig, staking app, splitter, etc

I think also it needs to look dope - like give it some color and emojis and make it feel like a cli with some love/thought added to it

carletex commented 1 year ago

@austintgriffith thanks for your comment!

In my opinion, SE-2 should stay opinionated (even tho we allow choosing between some important stuff + adding extensions). If we go crazy with the amount of config we let the user choose, it's going to hurt the maintenance & scalability of the project, and we'd spend most of the time dealing with conflicts instead of improving the tool itself.

I think we need to find the right balance. We are already tinkering with it, so let's regroup soon and talk about the findings.

I think also it needs to look dope - like give it some color and emojis and make it feel like a cli with some love/thought added to it

:space_invader: :sparkles: yes! :rainbow: :rocket:

carletex commented 1 year ago

To test the NPX install I created:

https://github.com/carletex/create-se2-app

Published on NPM:

https://www.npmjs.com/package/create-se2-app

It's just a basic example that clones the SE-2 repo to a given directory.

npx create-se2-app

I notice that I need do

npx create-se2-app@latest

If I want to pull newer versions. If not it takes an old version from your system cache.

ToDos

https://user-images.githubusercontent.com/2486142/227143276-7882cfe7-314f-4f94-9a72-23e004f190ff.mp4

sverps commented 1 year ago

ToDos

Additionally, I think most users would want a blank git (similarly to how other create-... tools work). We could give the option in the cli Preserve git history: (no)

carletex commented 1 year ago

Additionally, I think most users would want a blank git

I personally keep the history so I can easily back-merge stuff from SE-2... but I don't know about other people. Yep, prolly could be a cli option!

(similarly to how other create-... tools work)

Yep, they all do that haha But I think the way we are doing it here (at least for now) it's different. With other tools (nextjs) you can update with yarn/npm updates. Not the case for SE-2. BTW, it could be an option for the SE-2 cli! (fetch from the latest version and try to merge)

I'm still not sure about a bunch of stuff, but I think it'll become more clear as we tinker.

Thanks!

sverps commented 1 year ago

I personally keep the history so I can easily back-merge stuff from SE-2... but I don't know about other people. Yep, prolly could be a cli option!

Keeping the history is mostly just useful for collaborators and maintainers. I don't think that's the target group of the people using npx create-se-2. Most people using npx create scripts want to build their own thing that usually resides in a personal git project.

I think maintainers and collaborators should really just clone the project.

carletex commented 1 year ago

I think maintainers and collaborators should really just clone the project.

Agree.

Keeping the history is mostly just useful for collaborators and maintainers. I don't think that's the target group of the people using npx create-se-2. Most people using npx create scripts want to build their own thing that usually resides in a personal git project.

Yep, you might be right. How would you envision getting updates from SE-2 in your project? Or once you clone, you lose the link with SE-2? That might be the way to go since SE-2 is meant to be a starter kit.

sverps commented 1 year ago

How would you envision getting updates from SE-2 in your project? Or once you clone, you lose the link with SE-2?

Yeah I think that's the behavior that people would expect if they are somewhat familiar with other create-... tools. I think our target audience are primarily tinkerers that will play around, try to complete some speedrunethereum challenges (we could have the tool give some templating options for that), start a PoC of an idea they had, etc. I doubt they'll try to update, cause that might involve solving merge conflicts on files they tinkered with, which could turn complex quite quickly.

I expect developers that want to go that route to be experienced enough to start by cloning our repo.

technophile-04 commented 1 year ago

I really like the Implementation ideas mentioned by Carlos here and I think it's a way to go for the initial iteration.

Sharing some good examples which might be helpful for future / reference need :

Example 1 : create-t3-app

Love how their project is structured and love their DX of using their cli.

What if we follow the same pattern as others having a template dir which consists of base template of the app in our case will be an monorepowith nexjs inside packages dir and things like hardhat/foundry will be added dynamically when initially choosing from cli ?

This will help us extend options when running npx create-se2-app to scaffold dApp.

Example: In the current situation, if we plan to give options for The Graph and SIWE when initializing create-se2-app : If people choose both then there might be conflicts, for eg: In _app.tsx with imports or wrapping along the provider, etc.

Example 2 (future's future consideration) : superplate

Again their structure is worth looking at

They have their cli-repo here -> https://github.com/pankod/superplate and all the template repos here -> https://github.com/pankod/superplate-core-plugins they are also using templating engine which I think is a bit out of scope for us

Above examples might be a bit overpowered / against our moto of keeping things simple and minimal but we can always reference the libraries they are using for providing a better developer experience 🙌

Resources :

  1. create-t3-app (nice commented code)
  2. superplate
  3. Create CLI with Node.js

Most people using npx create scripts want to build their own thing that usually resides in a personal git project.

That might be the way to go since SE-2 is meant to be a starter kit.

Although even I like to preserve git history while building to get new features but I see most people like to start with a clean slate since as Samuel mentioned it will reside in their own git / github repo and might don't look for new features added constantly once they start with their project.

carletex commented 1 year ago

Thanks for the detailed response @technophile-04 !!

I think templating (like create-t3-app) is the most powerful/scalable choice, but still not sure if it's the best for us, because I still don't know how exactly it'd look like and what it would take to maintain the project if we follow that path.

Doing some tinkering and having something to test will be ideal!

Some questions that I have:

The solutions that I proposed in my first message are definitely hackier :D Wrote that thinking about simplicity, keeping the repo usable, and just having some cli tools to help with some actions / configurations.

Let's keep discussing! Thanks <3

technophile-04 commented 1 year ago

Yeah create-se2-app will be a separate repo having structure something like this : 

create-se2-app/
├── cli/
│   └── cli.ts
├── template/
│   ├── base/
│   │   ├── packages/
│   │   │   └── nextjs
│   │   ├── package.json
│   │   ├── yarn.lock
│   │   └── ...
│   └── extras/
│       ├── solidityFramworks/
│       │   ├── hardhat
│       │   └── foundry
│       └── next_src/
│           └── _app/
│               ├── theGraph.tsx
│               ├── theGraph-siwe.tsx
│               └── siwe.tsx
├── package.json
└── tsconfig.json

But even I am unsure how we gonna keep create-se2-app up to date with OG se-2 repo :(

The solutions that I proposed in my first message are definitely hackier :D Wrote that thinking about simplicity, keeping the repo usable, and just having some cli tools to help with some actions/configurations.

Having done some research I think this is the way to go at least for inital version because here we are directly cloning se-2 best way to get updated repo without doing any fancy stuff. Also get us going with Foundry / hardhat easily

Doing some tinkering and having something to test will be ideal!

++

rin-st commented 1 year ago

Glad that we're finally started to work on it, and I love the discussion! 🚀 Hope I'll find time to tinker with it on weekends

rin-st commented 1 year ago

omg there is so much information :D. Great research @carletex and @technophile-04 !

Most people using npx create scripts want to build their own thing that usually resides in a personal git project.

Agree

The solutions that I proposed in my first message are definitely hackier :D Wrote that thinking about simplicity, keeping the repo usable, and just having some cli tools to help with some actions/configurations.

Having done some research I think this is the way to go at least for inital version because here we are directly cloning se-2 best way to get updated repo without doing any fancy stuff. Also get us going with Foundry / hardhat easily

Agree too, but it will definitely take time to find best solution

roninjin10 commented 1 year ago

Try to get ahead of reserving whatever npm package name you want it to be named like npx create-scaffold-2 or whatever you think you might end up going with

carletex commented 1 year ago

Try to get ahead of reserving whatever npm package name you want it to be named like npx create-scaffold-2 or whatever you think you might end up going with

https://github.com/scaffold-eth/se-2/issues/251#issuecomment-1480765238

We are not sure about the final name tho!

roninjin10 commented 1 year ago

When you go the template route I saw this really nice tool that lattice team uses in mud https://github.com/uetchy/create-create-app#defaultpackagemanager-default-undefined

carletex commented 1 year ago

After some digging with @damianmarti, I see 3 options (I'm sure there are some middle-ground also) for "extensions":

  1. Each package only adds a new packages/{extension} folder and doesn't touch any other directory (root, the nextjs package). It'd come with a nice README, that will tell you how to set it up (install deps, add workspace to root package.json, react context, etc)

-> Super easy to integrate (just need to feed the CLI tool with the options), scalable. maintainable... but requires manual setup from the builder and it doesn't work out of the box.

  1. We have a templating system (like handlebars) in place, and we dynamically/conditionally add/remove stuff to different places that (package.json, react components - context, etc) when generating the project after the cli choices. In this method, there is only a single source of truth (eg: a single _app.tsx file with all the template logic)

-> Everything works magically out of the box but the templates can get messy if there are a lot of extensions that update the same files. But it could be an option if we consider that there is not much overlap and we only a have a few "core/curated" extensions.

  1. Similar to 2, but in this case instead of having a single source of truth, we have folders with all the different combinations, so we don't need a template system, just fetching the right file after the CLI choices.

-> Things are more separated than in 2, but you also need to keep in sync the files if you make a global change.

Would love to hear some thoughts!

damianmarti commented 1 year ago

I think that we should think about some extensions we want to have available and then checks how will be to add each extension using each approach. In this way, we can take a decision with more information available.

For example, for the foundry and the subgraph extensions may be the first option is good enough, but maybe for another kind of extension this leaves a lot of work to be done to the user and something like the second option is the way to go.

The option I less like is the third one ;-)

luloxi commented 1 year ago

Although even I like to preserve git history while building to get new features but I see most people like to start with a clean slate since as Samuel mentioned it will reside in their own git / github repo and might don't look for new features added constantly once they start with their project.

It could be an option on the install script to remove development .git folder marked as “yes” by default

Could people still clone and run the repo, or if we implement this all installs should go through the "create"?

It could be both, just by using Inquirer.js to mimic the “create” behavior. Inquirer.js is a tool for creating prompts and collecting input to then execute actions based on that input. I can attempt to implement it and then run it from an alias in the package.json on the root folder (but where's that npx create config?)

Question: Is there a foundry branch somewhere? Couldn’t find it (but haven’t tried them all out)