how does it compare to `amok`?

[serapath]

https://github.com/caspervonb/amok/issues

[omphalos]

Yeah, it looks like amok uses V8s remote debugging protocol to do hot swapping. This library makes hot swapping programmatically accessible from inside the web page, and it works by code rewriting. It doesn't depend on any special runtime-level hooks. Also amok tries to be a full-featured development tool, whereas thermite just tries to be a small, reusable library that could be used by development (or deployment) tools. Hope that answers your question. If it doesn't, please let me know!

[serapath]

hm ok, then did you check: browserify-hmr and ud? Is it more like that? https://github.com/AgentME/browserify-hmr https://github.com/AgentME/ud

[omphalos]

I hadn't seen those two projects in particular but I have checked out some other hot-swapping implementations before working on this. I really like the topic of hot swapping code and I always like to see how other people approach this.

Um, so the difference between thermite and those two is that you have to do manual management of function references with those libraries, whereas thermite (like V8) does this for you. So, let's say you are using ud and you register a callback somewhere. Then you hot swap your code. The old version of your callback will still be registered with whatever it was registered with. The same thing is true with hmr. If you want to remove your callback with hmr or ud you have to do that as a manual step when you implement the reloading portion of the interface.

There are pros to doing manual reference management. One thing is that it that hot swapping is faster. thermite needs to do a version check every time you call a function, because it hot swaps lazily. Also it injects references to eval to your code to do the hot-swapping, which isn't good for performance-critical stuff. Finally, with manual reference management you don't need much code to get it to work. This isn't an issue in Node but in the browser the user downloads the acorn JavaScript parser as part of the minified bundle. All told thermite.min.js is 131 kb long because of this.

On the other hand, thermite is a lot more generic (similar to V8 in this regard). For example if you had a code editor and you wanted to be able to hot-swap any user code arbitrarily, then you can't do this with hmr or ud. In normal JavaScript there are many ways to create dangling function references (in event listeners, Node callbacks, monkey patching or whatever). thermite hot swaps all of these whereas hmr and ud don't handle these cases for arbitrary code.

[serapath]

I'm currently using browserify-hmr and ud in my projects which i build with browserify + plugins and transforms. In my actual code, i'm using ud but I wonder if you could add an example to the readme.md how to use thermite instead. I'd love to try it out :-)

[omphalos]

Thanks for the interest, @serapath!

thermite is kind of a small, lower level library. You give it code; it returns a result, and you can tell it to update with a new version of code. It's not an integrated solution like browserify-hmr and isn't written as a plugin for WebPack or anything. I probably should have mentioned this in my last comment.

I imagine it's probably not hard to create a plugin of the sort you want - maybe by forking ud or something - and using thermite instead of the function reference swap that ud does. Alternatively you could try to push the ud maintainers to use thermite instead - there are just two methods on the thermite object to use, eval (for the initial code block) and update (for each version thereafter). I'm not sure if they would be interested in something like that or not, but you could give it a shot.

[serapath]

His pseudonym is @AgentMe and he was recently talking to @caspervonb here, so they both seem quite open to discuss matters.

I'm just an interested user who is not familiar at all with the details, but maybe you can drop them e message :-) i'd love to test "hot reloading solutions" in my code and once i get it to work, I can offer to tell you about it and create example code or an example for the readme if that's needed :yum:

[caspervonb]

Minor nits about the README while I'm here. I get your sentiment that a library can be easier to use but the bullet points there might not be entirely correct.

V8 runtimes are currently the only ones that support live code reloading (AFAIK), so if you want to do this in another browser, you're out of luck.

Supposedly its in JavaScriptCore, and heard from the IE protocol bridge guys that Chakra has support for it too.

V8 requires the user to start a debugging session and doesn't provide a way to hot swap from the program itself.

You could open a socket and talk to the runtime.

[caspervonb]

Ok. so if we only compare the actual hotpatching feature this mimics setScriptSource fairly closely. Again setScriptSource is at the core of how hotpatching works with Amok and is how I avoid doing eval in the other layers. Theoretically thermite (fork/with more work) + transform could be a fallback in amok for runtimes that do not support setScriptSource like Firefox.

I'm saying fallback because I've been down this path before, my primary reason for not going this route was the insane overhead it introduces.

To get the same behavior as you get with Amok (or just setScriptSource alone in a different context). Every single function has to be instrumented with a proxy during a compilation pass to get the same base functionality.

[caspervonb]

Question however, wouldn't this all break in strict mode? as direct eval is not allowed.

[omphalos]

@caspervonb Thanks for your comments.

Supposedly its in JavaScriptCore, and heard from the IE protocol bridge guys that Chakra has support for it too.

I'll update the README to account for this.

You could open a socket and talk to the runtime.

Interesting. That possibility hadn't occurred to me. Basically the user would have to start their browser with the remote debugging port enabled and connect over the web socket. I suppose they would also have to run the page on localhost once Chrome disables support for connecting to localhost websockets from other domains. Still, pretty neat.

I'm saying fallback because I've been down this path before, my primary reason for not going this route was the insane overhead it introduces.

Yeah, it's slower, but to be fair, thermite only calls eval when the code changes. At other times it uses a cached version of the function. Overall even this it is slower however.

Every single function has to be instrumented with a proxy during a compilation pass to get the same base functionality.

Yes, thermite does this automatically. It handles all sorts of nesting, closures, and so on during an AST rewrite pass. It handles these things as part of hotSwap operations as well.

Question however, wouldn't this all break in strict mode? as direct eval is not allowed.

eval still works in strict mode, it just has slightly different semantics (like being unable to hoist variables into the calling scope). All the tests run in strict mode, and the source is strict as well. Actually, I probably should create a non-strict version of the test suite as well to account for this.

[caspervonb]

Yeah, it's slower, but to be fair, thermite only calls eval when the code changes. At other times it uses a cached version of the function. Overall even this it is slower however.

The eval isn't going to be that big of a perf hit, but the call indirection will be and its exponential.

[omphalos]

Thanks again for your input, @caspervonb. Sorry, what do you mean by exponential?

[caspervonb]

Ups I suppose it's linear growth, if each function has a proxy there's one level of indirection for each function.

[omphalos]

Yeah, it does add a decent amount overhead though. Also I think whenever V8 sees eval it skips a bunch of optimizations unfortunately.

[omphalos]

@serapath

i'd love to test "hot reloading solutions" in my code and once i get it to work, I can offer to tell you about it and create example code or an example for the readme if that's needed

I personally don't have any plans to implement something like what you've described myself - my workflow is usually pretty bare bones - but in case someone creates a plugin using thermite, or creates some examples of its use, I'd definitely be happy to include links in the README.

[omphalos]

@caspervonb Kind of wondering what you have in mind regarding "fork/with more work" with this remark:

Theoretically thermite (fork/with more work) + transform could be a fallback in amok for runtimes that do not support setScriptSource like Firefox.

I've found your feedback pretty insightful so far and if you felt something was lacking from the current implementation I'd curious exactly what you have in mind.

[caspervonb]

I've found your feedback pretty insightful so far and if you felt something was lacking from the current implementation I'd curious exactly what you have in mind.

First thing that came to mind was source maps, which may or may not enable breakpoints depending on the browser.

[omphalos]

Yeah, I'm using falafel for the AST rewriting and traversal. There is a fork here which supports source maps which I ought to be able to try out.

[caspervonb]

Time to close this issue @omphalos? 😉

Been a long time, I've done adapters for node and all kinds of time wasting things, like making an plugin based repl that never saw life outside of my trash bin, In the end node does now support the protocol natively, but Edge and Firefox still do not implement any native hotswapping.

Thus, Amok is kinda dead and "Revaluate.js" is born, I'm approaching this the same was as I would in lets say Ruby, meta programming everywhere. Revaluate proxies functions in the same way as thermite, but also will incorporate and expand on the concepts outlined in the medium post from last year "Amokify and an Interactive Programming Workflow " (basically memoizing program statements, evaluating them once)

Meant to be consumed from Thingamajig.js, which is a node register hook / drop in script.

[omphalos]

@caspervonb Revaluate.js looks interesting. The source does look similar to thermite's overall - I see diffing and a global function reference cache - the idea looks very similar. I'm glad to see stuff in the same space. I guess the target for revaluate is functional programs without side-effects? I say this since it sounds like you plan to make use of memoization in the medium post. I may be misunderstanding what you plan to memoize however.

I will close this issue however. It may be that someone wants source maps added to this at some point, in which case we can open a new, more targeted issue for it.

[caspervonb]

The source does look similar to thermite's overall - I see diffing and a global function reference cache

For sure, the late binding function proxy idea came from you :wink:

I guess the target for revaluate is functional programs without side-effects

With side effects, top level program expressions (call expressions) are the ones that are memoized. New expressions are executed, survivors are fetched from memoization cache.

e.g, you can't do this with only swapping functions https://www.youtube.com/watch?v=XOLyM08IGhw, you need to eval new expressions.

[omphalos]

e.g, you can't do this with only swapping functions https://www.youtube.com/watch?v=XOLyM08IGhw, you need to eval new expressions.

Assuming I understand you correctly, this is what thermite is doing - each function checks its version when it's called, if it discovers it's version is out of date, it replaces its code with a new eval'd version, which grants access to the local scope.

BTW the workflow you demo there with amokify seems very similar to the Redux workflows I've seen.

With side effects, top level program expressions (call expressions) are the ones that are memoized. New expressions are executed, survivors are fetched from memoization cache.

I am kind of curious what you're doing here. So let's say you have some call expression, and this call expression, say, looks inside some directory and returns the number of files in that directory. If the user of this library invokes this function they get one number. Then they go and create another file in their directory. If this function is memoized the number returned will be the original number (not the new number). Please let me know if I've understood what you're describing correctly.

Assuming that I have, it sounds like this kind of optimization (memoization) is useful for pure call expressions, maybe less useful for stuff that does direct I/O?

[caspervonb]

Assuming I understand you correctly, this is what thermite is doing - each function checks its version when it's called, if it discovers it's version is out of date, it replaces its code with a new eval'd version, which grants access to the local scope.

Yeah thats all fine, same as chrome's swapping and by extension how amok worked. I was for the most party happy with it but sometimes you want to build things up from nothing, change the interval of a setTimeout, add an event handler while running, etc.

But, neither chrome, amok or thermite will evaluate new statements/expressions which is what I'm trying to remedy this time.

BTW the workflow you demo there with amokify seems very similar to the Redux workflows I've seen.

I'd like to see that! 😁 I could have done better demos however, it is way cooler than it seems 😅

I am kind of curious what you're doing here. So let's say you have some call expression, and this call expression, say, looks inside some directory and returns the number of files in that directory. If the user of this library invokes this function they get one number. Then they go and create another file in their directory. If this function is memoized the number returned will be the original number (not the new number). Please let me know if I've understood what you're describing correctly.

Yes, It will yield the original number of files from the first call result.

Assuming that I have, it sounds like this kind of optimization (memoization) is useful for pure call expressions, maybe less useful for stuff that does direct I/O?

Its not for optimisation sake, rather to have a deterministic program flow. If the new value was read, and other things depended on the initial value we get into a tangled mess of state corruption.

So instead of dealing with that, memorisation is used to keep old state intact as it was, and non-destructively sprinkle in newly defined state. This way, state corruption can't actually happen, it is what you said it should be. One can always refresh things by commenting and uncommenting.

To use a better example, because file reads aren't that obvious, lets say, at the top of our program this function call was something more visually disruptive, like it was creating a new electron window. Obviously one would not want 10 windows created just because something else was edited elsewhere 10times in that file.

[omphalos]

I'd like to see that! 😁 I could have done better demos however, it is way cooler than it seems

I think both your demo and Redux's demo are pretty cool. I like all the stuff that people are doing in this area. Regarding redux in particular I was recalling this - https://www.youtube.com/watch?v=xsSnOQynTHs.

But, neither chrome, amok or thermite will evaluate new statements/expressions which is what I'm trying to remedy this time.

I think I understand. It seems like kind of functionality is going to work really well/intuitively with pure, functional code (which is how the Redux stuff works - with Redux the I/O is, ideally, cleanly separated out).

If you take a look at that Redux demo and compare it to amokify I'd be curious to hear about what you consider the differences and pros and cons between them.

[caspervonb]

Oh it's Dan's talk!

The react hot module loader relies on the statelessness of react components / classes, it falls apart if your files contain anything else than stateless declarations. (e.g stick a window create in there, and boom a thousand windows, or a thousand callback handlers).

Basically HMR: Evaluate everything, you have two states, try to merge the two states (e.g for property in each exported object, set property to latest value)

My approach: Evaluate new things, leave the old things, there are no competing sets of state.

Same applies to the derivatives of Dan's loader (there have been at-least 2 implementations for browserify)

[omphalos]

@caspervonb cool, thanks for the clarification!