Demeteorizing Vulcan and how you can help

eric-burel commented 4 years ago

Goal

Progressively transition from Meteor to Next app.

First being able to connect any type of JS frontend to a Vulcan backend
Then reuse Vulcan code in Next, both backend and frontend
Then use Next as our main framework

Some code will be specific to Next, however this transition will also allow to reuse Vulcan code in other type of React frameworks (eg Gatsby), at least for the frontend.

Things to get sorted out

Multi-environment NPM packages

[ ] How to create npm packages that works differently on the client and the server? The idea is to provide fullstack package as we are used to do in Vulcan.

For Storybook I've created a specific Webpack handler, but it would be better to have some out-of-the-box pattern.

Usually NPM expect you to define a unique main field that points to your package built code (eg dist/index.js). Instead, we may want to point a directory (related SO question), maybe with `exports: { ".": "./dist"}. We could also put built file directly at the project root.

We probably will need to define Webpack loaders/config at least for client side imports, that can differentiate environments.

Mongo

[x] How to handle the Mongo database? Probably a simple Docker command in development

Others

[ ] How to handle startup logic. On the client it's ok (see Storybook code in Vulcan) but on the server we don't really have a startup system. We might probably explicitely call a function in our Next app.
[ ] Equivalent to Meteor boilerplate (setting up the barebone HTML page) => Custom document
[ ] What kind of example app people want to see in 2020?
[ ] Easy deployment, including deploying a mongo database
[x] Figure out npm link with Lerna, to allow a pattern similar to METEOR_PACKAGE_DIRS under development => calling lerna bootstrap or lerna link should work
[ ] ... others

Things to do

Steps to go from Meteor packages to NPM packages

[ ] Copy the package in the packages folder of vulcan-next-starter
[ ] Make it work in the Next application, demeteorize
[ ] When the package is ok, move it to the Lerna monorepo vulcan-npm (WIP: we need to add a Webpack config in this repo to handle multi-env package)
[ ] Update the Meteor packages to reuse the NPM packages as much as possible => from meteor/vulcan-core to @vulcan/core

The goal is to both allow to use Vulcan outside Meteor, in Next, and keep maintaining the Meteor version with a minimal amount of code. To be done for most existing package, however note that packages may exist in Next already. Eg handling styled components. In this case no additional work is needed, it's user responsibility to install relevant packages.

Specific demeteorization tasks

[x] Move to Mongoose for collections https://github.com/VulcanJS/Vulcan/issues/2479
[ ] Graphql Accounts issue PR
[x] From Jest to Mocha (should not be too hard)

Documentation of equivalent patterns

Coming from Meteor, we need to get a grasp of equivalent patterns in Next.

[x] npm link vs a 2-repo install with METEOR_PACKAGE_DIRS => lerna bootstrap should install and link. However npm i your-package@x.x.x may sometime fails. Adding manually the package in package.json seems to work.
[ ] Handling environment specific settings , with next.config
[ ] Testing packages altogether
[x] Create a new package in the monorepo: cd packages && mkdir vulcan-whatever && cd $_ && npm init, define index.server, index.common and index.client (or just index for only one environment). Or lerna create
...

Tooling

[x] Setup Jest in the monorepo
[x] Jest in the Next app
[x] TypeScript in the monorepo
[x] Export to JS when publishing on NPM
[ ] Eslint/prettier in Lerna
[ ] A webpack config in the monorepo able to handle NPM package with index.client and index.server to have 2 different build. For example @vulcan/core index.server.ts reexports @vulcan/users => it needs Webpack + our custom helper to know which index file to load from @vulcan.users automatically.

Feature-wise goals

In order to make the process easier, here are some intermediate goals. Each may need some of tasks listed above to be achieved beforehand.

[ ] Use @vulcan/users useCurrentUser hook in vulcan-next-starter: ~~need to be able to connect to Vulcan OK~~, need a custom Webpack config in the Next app to load packages with environment-linked index files, need to be able to build our TypeScript code in the monorepo
Same but in Vulcan Meteor meteor/vulcan:core package, in order to start reusing the same code. Not sure how to import our multi-env NPM packages in Meteor though, we may need explicit @vulcan/core/index.[environment] imports.

Use case idea

Working on a simple business use case would allow us to create example more easily. We could for example develop a Job board, or an app similar to "Awesome Vulcan" to publish vulcan ressources.

[This document is a Work in Progress. It will be updated with additional questions, answers, and process to move from Meteor to Next]

Ressources

Contribute

Post a comment on this issue if you want to help or take care of one of the tasks.

comus commented 4 years ago

hello, something i tried to do it last year, but i didn't have time to finish it. really.

i just share what i did, for the "How to create npm packages that works differently on the client and the server?"

it s easier with webpack config. it's good to wrap it with next.config.js, such as withVulcan

config.resolve.mainFiles

for example https://github.com/comus/react-vulcan-proposal/blob/master/src/withVulcan.js https://github.com/comus/react-vulcan-proposal/blob/master/examples/simple/next.config.js

and "Mongo database", i am not use mongodb now. but i have a solution for local development.

there is a node js library called 'mongodb-prebuilt', it can download the mongodb binary and run it.

i have a bin script: https://github.com/comus/react-vulcan-proposal/blob/master/src/bin/react-vulcan.js

with the bin, people can just type npx @vulcan/anyname mongo to launch a mongodb for development.

eric-burel commented 4 years ago

Thanks for the withVulcan helper. So it seems that you went for a specific loader (edit: resolver not loader) too in order to handle both client and server. Maybe that's the only solution indeed.

For reference of "mainFiles" webpack config: https://webpack.js.org/configuration/resolve/#resolvemainfiles

eric-burel commented 4 years ago

I've made a first update of the monorepo to show how it could work: https://github.com/VulcanJS/vulcan-npm

I've only added Typescript, but we would also need to use Webpack in order to have transparent usage between server and client. It's not tested yet.

@comus I have reused your idea of using mainFiles to make a more generic helper in @vulcan/webpack package. It may be used in Next like you did but also in any kind of app that uses Webpack.

Since Node servers use Webpack less often, we could also consider that index.js is the default server export. So that you can still do const whatever = require('@vulcan/core') in a server that do not use Webpack, it would be equivalent to const whatever = require('@vulcan/core/index.server').

We can safely consider that client side the developer is able to setup Webpack to import Vulcan, or explicitely call const whatever = require('@vulcan/core/index.client).

Don't know if I am clear but I think we will figure this out correctly.

One pain point though is that we usually use package.json main field to tell NPM that the built JS file is the root of the package (eg dist/index.js), while here we have multiple files. We may need to put built files at the root of the package.

eric-burel commented 4 years ago

@comus I've pushed a new test, sadly I could'nt have "multi-env" import to work. I have pushed a new commit with a demonstration (see _app.ts, it loads @vulcan/multi-env-demo). It is weird because the "log" is correct, but I still have a build error, very weird.

Also, I made a test with Webpack in the monorepo and build did work. However we face a big problem: the text editor can't understand our index.client/index.server thing. It says that it does not find the package if you just type @vulcan/multi-env-demo, because it looks for an index, even when the build work. There is no way for VS code to tell whether you are editing client code or server code. Note that it explains poor support of Meteor in text editors.

This may end up being too complicated. In which case we would be forced to do import '@vulcan/core/server and import '@vulcan/core/client depending on the environment, without relying on Weback.

To sum it up:

@vulcan/core would import only code that works on BOTH client and server
@vulcan/core/server code that works on BOTH client and server + code that work on server only
@vulcan/core/client code that works on BOTH client and server + code that work on client only

I don't think it's a big deal in the end.

The experiment with resolve.mainFiles try to be a bit smarter and select the right code automatically but it may raise too many issues with tooling .

eric-burel commented 4 years ago

It also means that we will lose the pattern of having 2 components with same name but different implementation in server and client, eg to handle components that do not support SSR. But again not a big deal, it was seldom used and very Vulcan specific, we have other alternatives in Next (using NoSSR components for example, dynamic imports and so on)

VulcanJS / vulcan-next