Allow other optimizations for dev

[ignorabilis]

Hello, we have a bit of an unconventional setup which goes like this - our app works in a Google Apps Script context, thus all the cljs that gets compiled to js needs to also be uploaded to the respective Apps Script. Uploading all the files that are dependencies is just going to be very slow and messy (and not sure if viable tbh). Instead, when a file changes, we want to wait for a build with white_space, simple or even advanced, depending on performance, optimizations. Once done we already have a process that would automatically push/deploy the resulting js to Apps Script.

Given the above - how hard would it be to enable optimizations for dev builds? Or maybe there's another way to achieve what we need? As the environment is different, we don't care about dev-tools, so if those are completely broken that's fine for us - all that matters is the resulting js.

I saw that :optimizations :none are just hardcoded here - if that gets commented out would that still produce proper javascript (as mentioned dev-tools is not a concern)?

[thheller]

If you don't care about devtools anyways, then the literal only difference between dev and release builds is that release builds apply optimizations. So you are asking for a release build? You can run shadow-cljs server and then manually trigger shadow-cljs release script, which will be reasonably fast (optimizations take time), but with the server running you'll substantially reduce the compile time.

What you are really asking for I guess is a custom :target implementation for "Google Apps Script", which would output code suitable for that environment. I have never used or looked at this though, so I'm not sure what the capabilities of that runtime/environment are? Are there some docs describing the platform details you can point me to?

[ignorabilis]

@thheller - You can put it this way I guess - we need a release build, but we want it to happen automatically when watching; it won't be as fast, but that's ok - 5-6, even 10 seconds is fine. That part of the project is not big and the JS we can upload (at least in theory) cannot grow too much. Essentially it needs to be a single file, say main.js instead of main.js + cljs-runtime folder with all the dependencies - because it would be super hard and slow and cumbersome to upload all files.

We're currently using :target :browser and that's totally fine - it's a browser-like environment.

[thheller]

But I assume it is not a browser. The cljs-runtime folder is technically not needed, it exists only for source map purposes. main.js is otherwise self-contained.

How are you uploading the files in the first place? Can you rsync the files or something like that? Most files in cljs-runtime will rarely change, and likely only need to be uploaded once. main.js however will still be multiple MB, even the baseline.

There is not support for automated "watched" release builds since the compile time will often lead to multiple wasted compiles. For example I have save-on-focus-lost in my editor, so all files get saved when I tab to view a browser. With dev watch that still leads to a "useless" compile often, with release you'd be stuck a substantial amount of time.

You could always use separate tools to get this behavior (for example chokidar-cli), but shadow-cljs will not support it for the reason mention above. It is also possible to just setup a keybinding in your editor to trigger a compilation. Many ways to do this that are much better than automated "watch" given how wasteful it can be with optimizations.

[ignorabilis]

Ok, so I'm not entirely into the details of what main.cljs contains depending on build, but here are my observations:

• in dev (:optimizations :none) the lines of code are half of what they are in any non-dev build
• copying any release build over to Apps Script works; the dev one doesn't - it's throwing

  Error: browser bootstrap used in incorrect target
  at [unknown function](Code:1316:11)
  at [unknown function](Code:1457:3)

Which actually inclined me to look it up and it lead to this; tried changing the target, but none of the others could even build

Looking at the code I now understand why you started talking about a new :target; I tried modifying the resulting build manually (stripped all the doc stuff):

var SHADOW_ENV = function() {
  var loadedFiles = {};

  var env = {};

  var scriptBase = goog.global.window.location.origin;
  if (CLOSURE_BASE_PATH[0] == '/') {
    scriptBase = scriptBase + CLOSURE_BASE_PATH;
  } else {
    // FIXME: need to handle relative paths
    scriptBase = CLOSURE_BASE_PATH;
  }

  env.scriptBase = scriptBase;

  var reportError = function(path, e) {
    // chrome displays e.stack in a usable way while firefox is just a garbled mess
    if (e.constructor.toString().indexOf("function cljs$core$ExceptionInfo") === 0 && navigator.appVersion.indexOf("Chrome") != -1) {
      console.error(e);
      console.error(e.stack);
    } else {
      console.error(e);
    }
    console.warn("The above error occurred when loading \"" + path + "\". Any additional errors after that one may be the result of that failure. In general your code cannot be trusted to execute properly after such a failure. Make sure to fix the first one before looking at others.");
  };

  env.isLoaded = function(path) {
    return loadedFiles[path] || false; // false is better than undefined
  };

  env.setLoaded = function(path) {
    loadedFiles[path] = true;
  };

  env.evalLoad = function(path, sourceMap, code) {
    loadedFiles[path] = true;
    code += ("\n//# sourceURL=" + scriptBase + path);
    if (sourceMap) {
      code += ("\n//# sourceMappingURL=" + path + ".map");
    }
    try {
      goog.globalEval(code);
    } catch (e) {
      reportError(path, e);
    }
  }

  return env;
}.call(this);

Unfortunately that just blew up with global is not defined. Investigating further seems there's a lot of logic (I assume hot reload and that kind of stuff) that relies on the browser.

Is there an easy way to create a new target that doesn't rely on browser stuff and instead just spits out main.js as-is?

[ignorabilis]

I managed to setup the node builds properly, but now getting The required JS dependency "https" is not available

[thheller]

Well, as the target names should imply :browser expects to be running in a browser, as it makes use of browser-specific APIs to support hot-reload/REPL. The node targets as such expect node and are very unlikely to work.

You are welcome to study the default target implementations, but everything will use shadow-cljs internal APIs that are not well documented. I don't expect anyone to just go and implement a new target on their own. I'm willing to assist or even write the code myself. I just need a clear definition of that is needed and what the runtime supports.

I'm guessing these are the docs for that runtime? https://developers.google.com/apps-script/guides/v8-runtime

I have never used any of Google Apps Scripts, so I don't have the slightest clue what the workflow is like. Any insights would help. Can anyone create scripts or do you need some kind of developer/privilidged account?

[ignorabilis]

@thheller - yes, that's the one. Anyone can create scripts - just create a Google Sheet and then just click on Apps Script (under Extensions).

The environment doesn't support anything you'd expect from a browser - windows, globals, etc. It also doesn't have node stuff like file access (AFAIK at least). If you want to make http requests you can do so using Google's object and methods (UrlFetcher). The idea of that environment is that you can manipulate sheets, presentations, docs, drive, etc. Much like a VBScript Excel macros.

In that sense it's a very crippled environment, but that's on purpose. I wouldn't ask for hot reload or a repl - those might be impossible given that requests longer than 5 minutes get killed and web sockets (I think) are not supported.

Thus just building the JavaScript should suffice. I'd imagine it would be simple to just strip everything related to repls, hot reload, node libs and scripts, e@thheller - yes, that's the one. Anyone can create scripts - just create a Google Sheet and then just click on Apps Script (under Extensions).

The environment doesn't support anything you'd expect from a browser - windows, globals, etc. It also doesn't have node stuff like file access (AFAIK at least). If you want to make http requests you can do so using Google's object and methods (UrlFetcher). The idea of that environment is that you can manipulate sheets, presentations, docs, drive, etc. Much like a VBScript Excel macros.

In that sense it's a very crippled environment, but that's on purpose. I wouldn't ask for hot reload or a repl - those might be impossible given that requests longer than 5 minutes get killed and web sockets (I think) are not supported.

Thus just building the JavaScript should suffice. I'd imagine it would be simple to just strip everything related to repls, hot reload, node libs and scripts, etc. - but then I haven't dived into the implementations referenced. tc. - but then I haven't dived into the implementations referenced.

[thheller]

Sorry this was left lingering for so long. I never did find the time to figure out all this Google AppsScript stuff.

As of 2.26.1 there is a working :single-file target, which only ever produces a single file. With no assumptions made about any specific runtime features and only accessing whatever the required code accesses. That might totally still blow up in the AppsScript context, I didn't test. This is however as barebones as it gets.

Example:

{:target :single-file
 :output-to "out/foo.js"
 :entries [your.ns]}

watch, compile produce a regular unoptimized but single file. It is huge. release applies regular optimizations. Still no watched release build, since I still believe manually triggering that is better.

[dcostaras]

Haha, I started a custom target for this this very weekend! Thanks @thheller I'll be taking it for a spin soon.

I think it'll work just fine, my custom target does pretty much what you're saying here... it's a very simple target.

[ignorabilis]

Wow, great stuff, thanks @thheller !

@dcostaras - would be interested to know how the setup works with the new target 🙂

[dcostaras]

@ignorabilis will let you know for sure but I suspect the answer is just fine! I took a different tack and instead of starting with the browser target I started with the Node target and just started removing stuff until it was, as Thomas describes above, a straight compilation of the CLJS source. And it worked, I was deploying the resulting JS file directly via Clasp without any changes and it worked in the AppScript runtime. I didn't do compile or watch as I most urgently needed release but it would be similar, albeit a little harder than release.

[dcostaras]

@thheller @ignorabilis the :single-file target works in Google AppsScript 👍