paritytech / substrate-api-sidecar

REST service that makes it easy to interact with blockchain nodes built using Substrate's FRAME framework.
https://paritytech.github.io/substrate-api-sidecar/dist/
GNU General Public License v3.0
247 stars 152 forks source link

moving from express to fastify #1152

Closed Gioyik closed 1 year ago

Gioyik commented 1 year ago

A few weeks ago I contacted @Imod7 to discuss what the team maintaining sidecar thinks about moving the express http implementation to fastify.

In my experience, express offers extensive documentation, and it's usually a safe bet as it's well known and supported by the OpenJS Foundation for continuity; however, express hasn't evolved or took advantage of most of the recent libs or runtime APIs in Node.js, most of the releases are dep version bump and security releases (which is great, nothing wrong with that), which is good, but not substantial features or improvements to the core of the framework.

In the other hand, fastify has evolved to be stable, fast, take advantage of runtime improvements and maintain a secure framework (they even have a LTS schedule). As well fastify is backed by the OpenJS Foundation for continuity, ] the development team is strong, diverse and the roadmap looks promising with the plugins and the continuous support of typescript. Out of the box fastify comes with a good set of features with little overhead, they have benchmarks comparing with other frameworks, I think you can run the same benchmark to compare and form your own opinion.

Now, why sidecar should make the move to fastify? From my perspective after the presentation where I saw sidecar, the main premises were light and fast (sorry if I am miss quoting), so the transition to fastify in my opinion is the next step for the long run. Today it could be that express doesn't represent a heavy part of the normal execution of the API endpoint calls, but the introduction of the framework can help reduce inefficient calls in the stack and ease the debugging process. Maybe reducing the cost of the server where sidecar runs as it's a bit more efficient or behaving more stable under high load scenarios, even if that's not a thing today.

For the things mentioned before, I would like to have time to run sidecar and get a few numbers, but I haven't had the time yet. Finally, if a change like this sounds beneficial for the project, I would like to offer slots of my time to work on the migration, as it helps me understand some internals and cooperation between components while I work on it.

TarikGul commented 1 year ago

Hey @Gioyik, thanks for bringing this up. So this is something I have considered a lot for the current implementation of sidecar. I agree on all the main points you bring up with fastify. The support and ongoing development is way larger than express, and offers way more performance capabilities etc.

There are a few things to note moving forward:

  1. Sidecar is going to go through a big rewrite soon, and will introduce SidecarV2. It's basically a time for the team and I to improve upon all the nags we currently have.
    • Extensibility for parachains
    • Improved debugging systems
    • Better logging, and log detection
    • Increased Error Documentation
    • Performance gains
    • Ability to plugin and attach controllers based on unique pallet logic. (Builds on parachain extensibility)
    • Increased WASM usage for high computation work. We currently use this for calcing fees, and staking payouts, but it would be nice to expand where we can.

This is just a short list of the things we plan on improving upon, but it will make sidecar a little more flexible, and also IMO easier for the community to contribute to, and be apart of.

  1. Changing Sidecar over to fastify right now would bring a bit of overhead while we are already planning and designing a rewrite. I would say its best to wait for the new and improved version. There is definitely a very high chance we move forward with fasitfy instead of express in the new iteration ;) We'll ofcourse be open sourcing the whole repo so there will be plenty of room for collaboration, and contribution!
Gioyik commented 1 year ago

thank you @TarikGul for the comment, and I am glad to hear that fastify sounds like a good future step for sidecar. I didn't know the project will be passing through a big rewrite, but all the points sound promising. I agree that making the transition to fastify would not be a smart/efficient effort today, I would be keeping an eye about the evolution of the project. Feel free to ping the issue when the time comes to continue the conversation and evaluation of how complex/hard it would be to make the switch, I will be around. :)

Quick question, is there a channel to watch the discussion of upcoming changes in the list that you mentioned? I think I could add input and maybe some expertise in the Improved debugging systems and Performance gains as I have worked in those scenarios before.

TarikGul commented 1 year ago

So for the time being we have evaluated a rewrite and what the could potentially mean. Currently we decided to move away from any efforts to migrate to fastify. The cost just doesn't fit the actual effort required. That being said we have leaned towards the direction of completely rewriting sidecar in rust if the time ever comes. This will mean a lot of things such as migrating off of polkadot-js, and adopting libraries such as desub.

Going to close the issue, but I appreciate the participation.