[VEDA2 Refactor] Exploration - Registry options and build pipeline to creating new core logic library

sandrahoang686 commented 9 months ago

Description: As part of the refactor work for veda-ui v2, we need to figure out how to access components from the instance level to consume as a library. This will likely require us to wrap the current ui logic with a layer to easily access components by importing through npm registry (to be determined). Current veda instances will still use the veda-ui logic as is (git submodule) but we also need to support consuming this veda-ui v2 as a library.

Acceptance Criteria:

[x] Research best registry option (Github, npm, etc). There might be restrictions from our stakeholders so we should determine what those are to make this choice. Likely npm
[ ] Determine how much overhead is needed to create access layer around veda-ui to consume react components at the instance level through chosen registry (npm for example)
[ ] Support current and new way of consumption
- [ ] Current veda-ui structure is intact and does not introduce any breaking changes to current build pipeline and current instances can still use veda-ui exactly the same
- [ ] Can also consume veda-ui as a library for the new test instance with same build pipeline
[ ] Document findings in this ticket so work can be started on #800

Resources: https://miro.com/app/board/uXjVN6lkBnc=/

sandrahoang686 commented 9 months ago

Another thought, i guess depending on whether or not we use npm for v2.. to develop this locally with the new test instance, we will probably have to have both running at the same time and may need to do an npm link of veda-ui in the new instance code

hanbyul-here commented 9 months ago

We will find out more in the research phase, but I believe where to publish/host our packages should not block us from using npm-related tools (such as npm link). But it is a good point that it might be an extra burden to run both veda-ui and test instance at the same time. We can consider something like sandbox https://visex.netlify.app/sandbox that can be run inside of veda-ui too. (I will detail it in #803 )

hanbyul-here commented 8 months ago

Research for the best registry option

NPM registry

https://docs.npmjs.com/cli/v9/using-npm/registry An NPM registry is just a collection of Node.js packages, metadata, and several API endpoints that are primarily accessed via the npm (Node Package Manager) command line tool. (ex. npm install) - So basically, where the package is hosted. To resolve packages by name and version, npm talks to a registry website that implements the CommonJS Package Registry specification for reading package info.

NPM & Yarn

NPM is the standard Node Package Manager. Our current structure requires any client to use Yarn as a package manager. (Because we use the protocols link: Yarn respects it, but NPM doesn’t: https://github.com/NASA-IMPACT/veda-ui/blob/main/package.json#L79-L81). This protocol is needed for custom Parcel modules, which won't be required once the refactoring is done.

Options

1. Use GitHub repo itself as a registry

This is how I am experimenting right now. An example of how to use GitHub repo as a dependency is: https://codesandbox.io/p/sandbox/using-veda-ui-nsz24n?file=%2Fpackage.json%3A12%2C1-13%2C1

Pros: We don’t need to depend on anything more than we do now. Cons: We have to commit the built file. Not making full use of the npm life cycle is possible. (Since GitHub repo cannot run ‘prerelease’ steps of NPM lifecycles: .) Big size of the package (since the package will include all the source codes)

2. Host & use private registry

Private registry example - Verdaccio: https://verdaccio.org/ (used for veda-ui rc version: https://github.com/developmentseed/ui-library-seed/blob/develop/DEVELOPMENT.md#publishing-rc-versions)

Pros: We can keep things private as much as possible. ?: impossible to find it for outside users. I put this as ? because it might be pro while the l library is under development. Cons: We need to maintain the registry instance :[ Additional configuration on user side to point the package to the right registry

3. GitHub Package

https://github.com/features/packages Pros: No need for an additional service provider. ?: It is not as popular as NPM (People won’t be able to find this package unless they definitely know where to find it. - But this might be a good thing until the package is officially ready to be published.) Cons: Additional configuration on the user side to point the package (

4. NPM registry (The registry run by NPM inc.)

Pros: It is the most popular registry for Node dependencies. We would likely want to move the package to NPM at some point in the future if we expect any outside users to use it. And we don’t have to do this if we start with NPM. NPM is default configured to use NPM public registry, so people don’t need an extra step to install the dependencies. ?: Very searchable. Cons: As far as I can check, NASA hasn’t used NPM registry before, which means this might require an additional process. If we want to be private, we have to pay ($7)

The options for registries are not mutually exclusive to each other. Ex. We can release both to GitHub package and NPM. Or we can decide to use a specific registry for a specific phase of development and then move on to the other option.

sandrahoang686 commented 6 months ago

UPDATES: Current experimental branch to have an additional libraray pipeline: https://github.com/NASA-IMPACT/veda-ui/tree/test-dependency Codepen that uses veda-ui as a dependency: https://codesandbox.io/p/sandbox/using-veda-ui-nsz24n Files to notice:

package.json https://github.com/NASA-IMPACT/veda-ui/blob/test-dependency/package.json#L27
app/scripts/export-components.js https://github.com/NASA-IMPACT/veda-ui/blob/test-dependency/app/scripts/export-components.js

hanbyul-here commented 6 months ago

I consulted with @amarouane-ABDELHAK and it looks like there wouldn't be a problem to use NPM for VEDA-UI. Considering NPM is the most popular registry for Node dependencies (and the least amount of cons from the options), we are going to move towards using NPM.

oliverroick commented 6 months ago

Quick high-level rundown for how I think building the package and publishing a release will work.

Building the package

We create a entry module, which imports and exports the components we want to publish in the npm package.
To create the bundle, we use rollup, as it allows us to create ES and CommonJS modules with relatively little configuration. The bundle will be written to the dist directory and committed. The bundle should include:
- A CommonJS module
- A ES module
- Source maps
- TypeScript definitions

I’ve only looked briefly at the component structure on VEDA UI and I feel like this should be possible. However, I might not be aware of all intricacies of VEDA UI components so I may have overlooked some potential pitfalls.

Publishing the release

Publishing the module will involve the following steps:

Make a new build from main
Create a new version using npm version
Commit the changes using the version number as the commit message
Create a tag and push the tag
Run npm publish to publish the package to the registry

It’s a few steps to remember, we can automate this as follows.

We configure postverion to:
- create and commit a new build
- tag the commit
- push the last changes and the new tag to git
We configure a Github action to publish the package to the registry. That we you only need to run npm version to start the release process. Alternatively, we run npm publish after making the release from our local machines, which allows more granular control over who can publish packages to npm.

hanbyul-here commented 6 months ago

Thanks for this run-down @oliverroick 🙇

I haven't been sure if it is better to have a separate bundler dedicated to building a library or to find a way to use Parcel to build a library. Either should be fine as long as we document what is happening. (Btw, I only used Rollup when it was the only option to build esbuild that is tree-shakable - is this still true?)
One thing to note is that we are already versioning the VEDA UI. (And we are using the versioned UI to update the UI from the instance more properly, for example, using v0.15.0 for VEDA dashboard https://github.com/NASA-IMPACT/veda-config/pull/377 . We are using a git tag for the releases, and I think npm version will automatically create a tag. We might need to re-establish how we release generally: https://paper.dropbox.com/doc/Veda-Process-discussion-Synthesis--CMq3JqoNPee2nYEuc7HW1obdAg-vtenpoWc4XvR8KDW1yKjZ

oliverroick commented 6 months ago

I haven't been sure if it is better to have a separate bundler dedicated to building a library or to find a way to use Parcel to build a library.

I'm sure there is a way to shoehorn the build for the library into the current parcel builder. It depends how we want to build individual instances in the future. In my opinion, a UI library should not prescribe how projects using the library are build. It locks projects into one way of doing things and leaves little room for customisation individual projects might need.

One thing to note is that we are already versioning the VEDA UI. (And we are using the versioned UI to update the UI from the instance more properly, for example, using v0.15.0 for VEDA dashboard https://github.com/NASA-IMPACT/veda-config/pull/377 .

We can continue increasing the version numbers in VEDA UI and publish the npm package using the same version numbers. The versioning and deployment strategies should be affected. I envisage the packaging the library and building individual instances as two separate, independent processes.

hanbyul-here commented 5 months ago

In my opinion, a UI library should not prescribe how projects using the library are build. It locks projects into one way of doing things and leaves little room for customisation individual projects might need.

To clarify my point, I meant the bundler only for VEDA-UI. I totally agree that the bundler used for veda-ui should be agnostic to how it is used. My concern was more about having two bundlers for UI for the time that we should keep both ways of using VEDA-UI (as a submodule and as a dependency) - which I anticipate won't be a short time.

Ex. if you look at the experiment I pushed a while ago: https://github.com/NASA-IMPACT/veda-ui/blob/test-dependency/package.json#L27, I used Parcel to build a 'library', mainly to avoid having another bundler as a dev dependency. If the other bundler is preferred with a clear reason, I don't have much objection as long as we keep the build pipeline separate, but I wanted to make my point clear.

oliverroick commented 5 months ago

I was suggesting Rollup because it's a bundler for JavaScript modules whereas Parcel seems more focussed on bundling applications. We'd get a couple of convenient features like being able to produce CommonJS and ES modules without much configuration overhead (though I'm sure we can configure Parcel to do that as well.)

Adding another dependency is a concern, although it shouldn't affect bundle size and it's temporary. We should only introduce Rollup if we're sure that we will remove Parcel altogether at some point.

dzole0311 commented 5 months ago

Adding my two cents, but I'm quite new to the project and I might be overlooking some complexities.

I have used Rollup and Parcel (but I have only a limited experience with it) before for bundling small libs, and I do agree Parcel comes with a a level of configuration complexity (that might not be even that big of a time consumer). But once its configured, I found it to be pretty straightforward to use.

Maybe I have missed this in the various threads, but are there any specific pros or cons between the two bundlers that would make Rollup an ideal candidate to switch to for bundling the VEDA-UI? Ie, some features that would require much less tweaking than in Parcel or ease of integration within github actions when publishing new versions (although the latter is a completely different step independent than the bundling itself)?

I'm also sharing the concern though that if we opt in for changing the bundler for the UI we might end up having two of them in the project for a long time that can essentially do the same thing: outputing ES and CommonJS modules, generating sourcemaps and TS definitions, etc. But then again, this is mostly because I don't have a clear timeline of how the project will evolve.

We should only introduce Rollup if we're sure that we will remove Parcel altogether at some point

I completely agree on this one.

Btw, I only used Rollup when it was the only option to build esbuild that is tree-shakable - is this still true?)

Parcel should support tree-shaking of built ES modules output.

These are just my thoughts while wrapping my head around this topic and following through the threads, and just trying to add my two cents with my limited knowledge of some of the VEDA software parts.

oliverroick commented 5 months ago

It looks like setting up Parcel to build a library is straightforward enough. Let's move forward with Parcel then.

hanbyul-here commented 5 months ago

Closing ticket since the ticket is going out of its scope. The bundler option discussion can go to #800

In case this helps @oliverroick, changing the attributes in package.json to build a library with Parcel previously caused me problems—it seemed those values didn't play nicely with the application's current build pipeline. I thought about experimenting with its api to make the build pipelines totally separate https://parceljs.org/features/parcel-api/ but didn't make much progress.

NASA-IMPACT / veda-ui