search

awslabs / llrt

LLRT (Low Latency Runtime) is an experimental, lightweight JavaScript runtime designed to address the growing demand for fast and efficient Serverless applications.
Apache License 2.0
8.11k stars 359 forks source link

why not hermes? #110

Open ljharb opened 9 months ago

ljharb commented 9 months ago

I note that hermes isn't mentioned in your "why not an existing solution" section. It'd be great to have an analysis of that one, since its goals seem much more aligned with LLRT's.

richarddavison commented 9 months ago

Hi 👋. That’s a very good question. There are a couple of reasons why Hermes wasn’t used:

ljharb commented 9 months ago

Makes sense! It'd be great to add that info to the readme.

mishushakov commented 9 months ago

+1 here for Hermes. Makes little sense to build a new runtime with QuickJS, which receives no regular updates and has no contributors other than Bellard.

richarddavison commented 9 months ago

QuickJS development was paused for quite some time but has recently been picked up again: https://github.com/bellard/quickjs/commits/master/ Also as: https://github.com/quickjs-ng/quickjs Or: https://github.com/OpenQuickJS/quickjs

LLRT uses Rust bindings from the crate rquicks which applies patches to QJS.

QuickJS is used all over the place in production and is rather stable :)

Hermes would be great if it supported a more recent ECMA standard and also focused on interpreter rather than ahead of time.

mishushakov commented 9 months ago

Nice to see its getting updated again. But it still rubs me the wrong way that the industry is so reliant on what is essentially someone's side-project.

Have you had a chance to discuss the matters with the Hermes team? They seem really nice and maybe would take your points into consideration for future version?

tmikov commented 9 months ago

We (the Hermes) are actively working on improving EcmaScript compatibility (scheduled for a beta release in the first half of 2024) and interpreter performance. However running directly from source will likely never be our strength - when running from source, we execute a full compiler pipeline for every function. QuickJS by comparison has a pretty simple compiler that doesn't take a lot of time.

I suspect that Hermes+bytecode will be faster than QuickJS both for startup and execution, but that may not be practical for this project.

It also seems like QuickJS has pretty great Rust bindings. For Hermes there is https://github.com/laptou/jsi-rs, but I am not sure how it compares.

mishushakov commented 9 months ago

Rust/C bindings is the №1 thing I'm missing with Hermes personally. But does JSI really matter now if static Hermes supports zero-cost FFI? In which scenarios would you rather use JSI instead of FFI + JS wrapper?

ammarahm-ed commented 9 months ago

QuickJS also has it's own bytecode format that makes startup almost as good as Hermes.

It's also much nicer to work with, super easy to embed and modify based on your specific needs, compiles on almost any low end system under a minute or so. It is also much smaller in size and supports es2022.

CanadaHonk commented 9 months ago

I agree with the main reasoning from this issue, but wouldn't a serverless environment thrive from AOT as they are usually deployed then untouched until the next deployment? Why not build AOT on deploy and use that bytecode? (I'm not really a serverless person so might be completely off)

tmikov commented 9 months ago

I am not advocating for Hermes here, I think QuickJS is a great engine. However I want to be clear about the relative performance of each. Below are benchmarks I just ran from source on my computer with hyperfine. The results include loading of the respective binary and compilation.

Command Mean [ms] Min [ms] Max [ms] Relative
./qjs v8-crypto.js 717.5 ± 1.0 716.4 719.2 1.22 ± 0.01
./hermes v8-crypto.js 586.1 ± 4.1 582.6 596.5 1.00
./qjs v8-deltablue.js 987.7 ± 5.9 982.1 1002.3 1.55 ± 0.02
./hermes v8-deltablue.js 637.9 ± 6.7 630.0 651.5 1.00
./qjs v8-raytrace.js 356.4 ± 3.0 352.1 361.6 2.25 ± 0.03
./hermes v8-raytrace.js 158.2 ± 1.7 157.0 164.3 1.00
./qjs v8-regexp.js 975.2 ± 16.2 953.1 998.1 2.71 ± 0.06
./hermes v8-regexp.js 359.7 ± 4.2 354.2 365.3 1.00
./qjs v8-richards.js 1.166 ± 0.016 1.146 1.200 1.59 ± 0.03
./hermes v8-richards.js 0.734 ± 0.006 0.723 0.743 1.00
./qjs v8-splay.js 378.8 ± 6.9 371.3 395.6 2.33 ± 0.08
./hermes v8-splay.js 162.5 ± 4.8 158.6 179.5 1.00
richarddavison commented 9 months ago

I really like the potential of Hermes! The underlying engine in LLRT is not set in stone, if there is a better alternative we should consider it :) @tmikov I would be very interested to see how Hermes would perform on Lambda with limited resources running as interpreter. Would the cost of compiler pipeline have significant impact on startup? Right now, QJS is a good option since it's small, offers a fast startup, has rather recent ECMA support and a fast interpreter. What we ideally wan't to avoid is to enforce a compile step to get great performance. LLRT already a llrt compile command that converts a JS source file to QJS bytecode and packs it with zstd compression. This should only be used to reduce startup of larger bundles and not be required for generic use cases.

mishushakov commented 9 months ago

I believe this is what you get when you're running the code from source. Pre-compiled byte-code should be even quicker. What the metrics really say is that with Hermes the particular examples execute faster compared to QJS.

mishushakov commented 9 months ago

In my opinion the biggest issue with QuickJS is the lack of incentives. Facebook/Meta has a team building and maintaining Hermes, they are accepting pull requests from outside and are shipping new versions regularly.

saghul commented 9 months ago

As one of the QuickJS-ng maintainers: we do have incentives. Mine is to build txiki.js a small but fully featured runtime.

I emailed Fabrice when starting the project and that "awakened the beast" sort of speak :-) Reigniting the project was one of the goals, so there is that!

tmikov commented 9 months ago

@richarddavison performance when running Hermes directly on JS source is not something that we aggressively optimize for, so unfortunately I don't have more concrete perf data. However Hermes is used in that mode by RN devs all the time - development mode bundles are not precompiled to bytecode - so it does work even on relatively weak mobile hardware. I have no intuition how Lambda compares to that.

Hermes continues to improve rapidly. For example, we will be improving our interpreter performance significantly using some new techniques and we expect up to 2x additional improvement without a JIT.

Of course we would love if LLRT would use Hermes - the more projects use an engine, the more varied the feedback and discovered problems, the better the engine becomes. However, if I try to be objective, I see the following factors against Hermes:

I suppose none of these are individually a deal breaker, but together they might make Hermes less appealing for your use case. It's important to weigh these factors against the benefits Hermes can offer, such as its potential for performance improvement and the support it receives from Meta.

Ultimately, the decision you have made is based on what best suits the needs and constraints of your project. We're here to provide any necessary help, should you eventually decide to experiment with Hermes.

mishushakov commented 8 months ago

I have decided to "take the matter into my own hands" and started building a Rust-wrapper for Hermes: https://twitter.com/ci_step/status/1760353806255063181

richarddavison commented 8 months ago

@mishushakov @tmikov I think we should explore how Hermes would perform for our use case. There is undoubtedly benefits of using an engine backed by an entire team that shares some common goals with LLRT such as fast startup, lightweight and great performance.

mishushakov commented 8 months ago

Hey everyone, I have just published my ongoing work on Rust + Hermes wrapper. Feel free to give it a star. Contributions are welcome ❤️

https://github.com/rust-hermes/rusty_hermes

calavera commented 8 months ago

Is the data in this comparison accurate? https://test262.fyi/#|v8,sm,jsc,hermes,qjs,boa

Based on that, it looks like Hermes is still missing many core APIs that we'd have to completely re-implement. It'd be good to have confirmation on whether that's an accurate state of the project or not.

tmikov commented 8 months ago

@calavera while Hermes does not have perfect test262 score, reading the numbers in that table can be misleading. It cannot be said that Hermes only supports 42% of the JS language or 54% of the builtins. The results are most likely skewed by checking for functionality in more recent versions of the spec, or our conscious decision to never supports things like "with", semantics of the "arguments" object in sloppy mode, or local "eval()".

If you look at the results for language/identifier-resolution, it shows 36%. On the face of it, that sounds very bad. Who would use an engine that only support 36% of the identifier resolution in the language?? But if you drill down, this is caused by all the tests containing "with".

It also seems that the test runner isn't using the latest version of Hermes because some random tests that I checked actually pass with Hermes.

With all of that said, we are committed to dramatically improving our test262 coverage and adding missing functionality. Can you give me an example of something that is not supported but should be?

EDIT: it also seems to show that Hermes does not support Intl at all, which is not accurate.

EDIT2: I realized that the tests are run on Linux and Hermes does not support Intl on Linux.

EDIT3: Hermes, as a AOT compiler, issues warnings. It looks that the test runner is mis-interpreting these warnings as errors in some cases and incorrectly deciding that Hermes has failed that particular test.

calavera commented 8 months ago

Can you give me an example of something that is not supported but should be?

I don't have any specific example at the moment. I was just curious about that big difference in coverage. Thanks for the explanation.

guest271314 commented 3 months ago

Wow. Popularity and branding. SOS.

Here's how various JavaScript runtimes stack up for speed of reading standard input and writing standard output.

bun is faster than deno which is faster than node, consistently. node, deno, and bun are using the same file.

There's only a few dozen javaScript runtimes, interpreters, engines to choose from A list of JavaScript engines, runtimes, interpreters.

Using only one (1) JavaScript runtime at a time is like trying to build a house using western framing techniques with only one (1) kind and size of nail.

node is not the first choice for embedding a JavaScript runtime, at 113 MB. Bun is not the choice for streaming where HTTP/2 is not supported. Deno is not the runtime if you want to actually use dynamic import() where the script is created in the running script, you'll get errors, consistently.

Havn't tried Hermes, yet. I'll get around to it.

bun is used to run .ts file as it is consistently faster than deno and node.

You'll also notice txiki.js tjs is faster than node.

What is missing from them all is consistent implementation of reading stdin and writing tostdout, as ECMA-262 doesn't specify reading standard input and writing to standard output.

So there's a different I/O implementation, if at all for the basic computing procedure of reading standard input and writing standard output. For example, llrt has no means to read standard input, not does SerenityOS's LibJS.

0   'nm_bun'    0.10480000007152557
1   'nm_qjs'    0.1135
2   'nm_deno'   0.12330000007152557
3   'nm_tjs'    0.20080000007152557
4   'nm_typescript' 0.24
5   'nm_spidermonkey'   0.35310000002384184
6   'nm_d8' 0.4397000000476837
7   'nm_nodejs' 0.5328000000715256
8   'nm_llrt'   0.7057000000476837
guest271314 commented 3 months ago

For a "serverless" comparison, compare llrt to VM Labs WASM Workers Server, and WasmEdge.

guest271314 commented 3 months ago

QuickJS is 1.2 MB.

With the capability to import C shared modules.

That's why a whole bunch of applications are using QuickJS, from Bytecode Alliance's javy, to VM Ware Labs WASM Workers Server, to WasmEdge.

You have to really be doing something special to compete with qjs in the embedded, "serverless" domain.

guest271314 commented 3 months ago

The question is almost rhetorical.

Why not Deno instead of Node.js?

No confusion over CommonJS not being the official module loader for JavaScript.

No legacy callback API's.

WebSocket server built in.

WHATWG Stream support and Web API's implemented by default.

See where this is going?

guest271314 commented 3 months ago

@tmikov

With all due respect, this is llrt repository.

I don't entertain preferences for JavaScript engines, runtimes, and interpreters. I experiment with, test, and break them all, equally, without exception.

This is FOSS. Fork the source, remove QuickJS, insert Hermes.

Then you can do a legit 1:1 test, which can be challenging in the JavaScript domain. There's WHATWG Streams which not controlled by TC-39. There's a big difference from fetch() supporting duplex: "half" for full-duplex streaming support; ReadableStream being an async iterable, or not, et al.

I'm on topic. I'm in the field testing everybody's gear, without an interest in any one specifically. Instead of what I see is folks from specific organizations promoting their gear, versus, some other gear.

Well, there's no reason there can't be a Hermes fork of LLRT. The users in the field can test implementations independently, as it's impossible mathmatically for anybody to prove their own claims internally. Somebody other than the person who came up with the hypothesis has to verify the steps to reproduce actually reproduce, for that hypothesis to have a chance at becoming an emergent truth.

So, I'll test all of you folks gear, and share the results, if you really want objective test results, I'll test your gear until it breaks, without a single coin being involved, nor any predisposed notions of preference.

Without that is just a my gear is bigger than yours contest.

tmikov commented 3 months ago

@guest271314 my apologies. I immediately realized that I had confused the repository (I thought this was in Hermes), so I deleted my comment right away.

guest271314 commented 3 months ago

@tmikov

No worries.

Took hermes for a spin in the meantime. The lack of capability to read standard input or write to standard output is a serious omission.

SerenityOs's js (ladybird-js) sufferes from that, too.

To be honest I have never had a reason to use React. That appears to be a substantial part of the documentation.

With QuickJS we don't have all of those framework distractions. We can just build whatever we want in C and wind up with a 1-5 MB executable.

Good luck!

tmikov commented 3 months ago

@guest271314

Took hermes for a spin in the meantime. The lack of capability to read standard input or write to standard output is a serious omission.

Hermes is a JS engine, not a runtime like Node, so this is not really an omission, but I understand that it makes it impractical for quick tests.

SerenityOs's js (ladybird-js) sufferes from that, too.

You will find that all industry JS engines suffer from that. Implementing and maintaining an I/O library is orthogonal to the JS engine. After all, this is what LLRT provides.

With QuickJS we don't have all of those framework distractions. We can just build whatever we want in C and wind up with a 1-5 MB executable.

Fair enough. If performance is satisfactory (2x slower), and the additional features of Static Hermes are unnecessary, you should definitely stick with QuickJS. As I have said multiple times, QuickJS is a very elegant minimalistic runtime.

guest271314 commented 3 months ago

You will find that all industry JS engines suffer from that.

That's not true.

QuickJS is an engine. In qjs we have the --std module where std.in and std.out are exposed. std.in being the only engine or runtime implementation I have tested that reads 1 MB in one (1) read.

So are V8 and Mozilla's SpiderMonkey.

V8's d8 shell readline() tries to execute standard input, though still reads standard input until an error occurs. I use GNU Coreutils head, dd command, or qjs (QUickJS NG or QuickJS proper) to read standard input to d8, then send d8s standard input back to d8 using os.system(), https://github.com/guest271314/NativeMessagingHosts/blob/main/nm_d8.js

  const bash = ["bash", [
    "--posix",
    "-c",
    `
length=$(head -q -z --bytes=4 /proc/${pid}/fd/0 | od -An -td4 -)
message=$(head -q -z --bytes=$((length)) /proc/${pid}/fd/0)
# length=$(dd iflag=fullblock oflag=nocache conv=notrunc,fdatasync bs=4 count=1 if=/proc/${pid}/fd/0 | od -An -td4 -)
# message=$(dd iflag=fullblock oflag=nocache conv=notrunc,fdatasync bs=$((length)) count=1 if=/proc/${pid}/fd/0)
printf "$message"
`,
  ]];
  const qjs = ["/home/user/bin/qjs", [
    "--std",
    "-m",
    "-e",
    `const path = "/proc/${pid}/fd/0";
  try {
    const size = new Uint32Array(1);
    const err = { errno: 0 };
    const pipe = std.open(
      path,
      "rb",
      err,
    );
    if (err.errno !== 0) {
      throw std.strerror(err.errno);
    }
    pipe.read(size.buffer, 0, 4);
    // writeFile("len.txt", size);
    // {error: 'writeFile is not defined'
    const output = new Uint8Array(size[0]);
    pipe.read(output.buffer, 0, size[0]);
    const res = new Uint8Array([...new Uint8Array(size.buffer),...output]);
    std.out.write(res.buffer, 0, res.length);
    std.out.flush();
    std.exit(0);
  } catch (e) {
    const json = JSON.stringify({error:e.message});
    std.out.write(Uint32Array.of(json.length).buffer, 0, 4);
    std.out.puts(json);
    std.out.flush();
    std.exit(0);
  }
`,
  ]];
  while (true) {
    // Terminate current process when chrome processes close
    if (!(os.system("pgrep", ["-P", JSON.parse(ppid)]))) {
      break;
    }
    const message = getMessage(
      pid,
      qjs,
    );
// ...

SpiderMonkey's js shell readline() implementation I was able to parse that readline() implementation using a RegExp

function getMessage() {
  // Call readline() N times to catch `\r\n\r\n"` from 2d port.postMessage()
  let stdin;
  while (true) {
    stdin = readline();
    if (stdin !== null) {
      break;
    }
  }

  let data = `${stdin}`.replace(/[\r\n]+|\\x([0-9A-Fa-f]{2,4})/gu, "")
    .replace(/[^A-Za-z0-9\s\[,\]\{\}:_"]+/igu, "")
    .replace(/^"rnrn/gu, "")
    .replace(/^[#\r\n\}_]+(?=\[)/gu, "")
    .replace(/^"(?=["\{]+)|^"(?!"$)/gu, "") 
    .replace(/^\[(?=\[(?!.*\]{2}$))/gu, "")
    .replace(/^\{(?!\}|.+\}$)/gu, "")
    .replace(/^[0-9A-Z]+(?=[\[\{"])/igu, "") 
    .replace(/^[\]\}](?=\[)/i, "")
    .trimStart().trim();
  // https://stackoverflow.com/a/52434176
  // let previous = redirect("length.txt");
  // putstr(data.length);
  // redirect(previous); // restore the redirection to stdout
  // os.file.writeTypedArrayToFile("input.txt", encodeMessage(data));
  return encodeMessage(data);
}

Not standardizing standard input streams and standard output streams is a huge omission in ECMA-262.

It means every JavaScript engine, runtime, interpreter, if they implement I/O, which they do in some way, does it differently.

Circa 2024 there are far more JavaScript engines and runtimes being used outside of the browser, as child processes and exotic interfaces, e.g., "serverless", yet absolutely zero standardized way to read stdard input stream into an ArrayBuffer, nowadays we have resizable ArrayBuffers.

WHATWG Fetch is outside the scope of ECMA-262, yet everybody tries to implement fetch(), some do it successfully, some don't. llrt doesn't support streaming over HTTP/2, nor does Bun (on queue for Q3), nor txiki.js.

After all, this is what LLRT provides.

I don't see exactly what Hermes provides, or for that matter what llrt provides over QuickJS or txiki.js.

We can write an HTTP server, thus an TransferEncoding: chunked version and a WebSocket version in C https://github.com/guest271314/webserver-c/tree/quickjs-webserver that we can import into qjs engine/runtime, for under 2 MB.

You will find that all industry JS engines suffer from that. Implementing and maintaining an I/O library is orthogonal to the JS engine. After all, this is what LLRT provides.

I don't think so.

I was able to read stdin to llrt the same way I did in V8's d8, by making use of the child_process module of llrt named after the Node.js module https://github.com/guest271314/NativeMessagingHosts/blob/main/nm_llrt.js

async function getMessage([command, argv]) {
  const message = await new Promise((resolve, reject) => {
    const res = [];
    const subprocess = spawn(command, argv, { stdio: "pipe" });
    subprocess.stdout.on("data", (data) => {
      res.push(...data);
    });
    subprocess.stdout.on("close", (code) => {
      resolve(new Uint8Array(res));
    });
    subprocess.stdout.on("exit", (code) => {
      reject(encodeMessage({ code }));
    });
  }).catch((e) => e);
  const cmd = command.split(/[/-]/).pop();
  if (cmd === "bash") {
    return message;
  }
  if (cmd === "qjs") {
    const view = new DataView(message.subarray(0, 4).buffer);
    const length = view.getUint32(0, true);
    // sendMessage(encodeMessage({ length }));
    return message.subarray(4, 4 + length);
  }
}

Fair enough. If performance is satisfactory (2x slower), and the additional features of Static Hermes are unnecessary, you should definitely stick with QuickJS. As I have said multiple times, QuickJS is a very elegant minimalistic runtime.

I have no idea how you are going to test any two JavaScript engines or runtimes unless you are running the exact same codee in both.

You see QuickJS all over the place precisely because it is minimalistic with a lot of built in functionality that is useful, extendable, and unopinionated.

Hermes appears *very opinionated about including React of all things into the mix. For what? Web sites are already grossly over-engineered for no reason other than to say it was built with React, or is using this and that framework. Take a look at Bellard's home page. Just actual content, no bells and whistles dancing around with no content.

Again, the omission os standard streams in ECMA-262 is a huge omission.

There's no shortage of JavaScript engines and runtimes, and for each new entry, including Hermes, there is either a unique implementation of streams or none at all. Far more that implement I/O in some limited way that hackers have to hack to redirect, tee, and do what they want to do with the engine, rather than what somebody is trying to sell or have some grand scheme for.

guest271314 commented 3 months ago

@tmikov

Keep in mind in this day and age people can deploy "serverless" HTTP and WebSocket, and if called for HTTP/2, HTTP/3, and QUIC servers directly from their own browser using Direct Sockets in an Isolated Web App, bypassing the external engine/runtime game completely https://github.com/guest271314/direct-sockets-http-ws-server.

Re standard input and output streams, I think maintainers don't think about their programs being launched and controlled from other programs outside of their system. It's a whole system sell.

But what happens when you want to deploy through a different mechanism and control via basic standard input and output streams?

Even in the "serverless" game there is incompatibility. CloudFlares Workerd doesn't provide a means to serve one uploaded ReadableStream to a subsequent request. We can do that just fine in Deno Deploy, which is a V8 Isolate "serverless" system.

It goes on and on with incompatibilities when people other than the champions of a technology beat the grass.

So when people speak of testing speed of engines and runtimes, I could care less about a ray caster.

Show me how fast your basic standard input and output streams are. Bun has everybody beat in that regard, even QuickJS, which was recently seated at the top of that metric. That's bun running a .ts file directly and bun running the .js version of the same file.

Now, if you wanted to control llrt via a different application, look how expensive child_process implementation is. We don't know how fast llrt could be as an embedded, remotely controlled application without launching a child process to read stdin.

0   'nm_typescript' 0.07309999999962748
1   'nm_bun'    0.10180000000074506
2   'nm_qjs'    0.1018999999994412
3   'nm_d8' 0.13670000000018626
4   'nm_deno'   0.14490000000037254
5   'nm_tjs'    0.16740000000037253
6   'nm_nodejs' 0.1775
7   'nm_spidermonkey'   0.4780999999996275
8   'nm_llrt'   0.7094000000003725
guest271314 commented 3 months ago

@tmikov Hey. FWIW I'm just an individual that enjoys hacking JavaScript, as a whole. Carry on...