Using js9 on a separate server

timbeccue commented 4 years ago

I am trying to integrate a js9 instance with an existing Vue application. The Vue app is served as a static site with cloudfront on AWS (in other words, not a normal server). I would like to include server-side analysis, so I'm hoping to run js9 on a standalone ec2 instance and connect it to the frontend.

At the moment, the server is working just like https://js9.si.edu (i.e. running the provided html pages). But there's a problem when pointing to the scripts in my frontend application--the explicitly requested scripts load just fine (js9prefs, js9support, js9, js9plugins), but they are unable to call other js9 scripts (notably, astroemw.js, js9worker.js). The browser's network inspector indicates that these files are being requested from a directory relative to the frontend url (eg. http://localhost:8080/js9worker.js) instead of the server that provided the js9 scripts.

Is there some configuration I can specify to use an absolute base location for loading additional js9 scripts?

Thanks for your help!

ericmandel commented 4 years ago

We'll have to feel our way here since, I probably don't understand the setup but ... there are a bunch of files that JS9 wants to load from the directory in which JS9 is installed, including js9worker.js and astroemw.js. JS9 figures out the install directory by looking at where the js9.css file was loaded from ... but I'll assume you did not move the js9.css file (let me know if you did!)

So ... to clear up my first confusion: are you able to load any of the files that are referenced as being in the "install" dir? For example, if you go to View->show->js9logo, do you see the JS9 logo on the web page? I'm trying to figure out whether you are missing all of the install files or just a few.

I assume you're missing all of them ... in that case, what you can try is setting the parameter globalOpts.installDir in js9Prefs.json to a path (relative to your web page) that points to the actual JS9 install directory. If this property is defined, JS9 won't try to figure it out using js9.css, but will use whatever you gave it. That should allow JS9 to find everything.

But if you are only missing a few files, let me know, it's some other problem ...

ericmandel commented 4 years ago

Hmm ... re-reading your comment, I guess the problem is not relative paths on a single server, but the need to retrieve files from an entirely different server. Is that correct? If so ... and I don't exactly know what will happen ... but it looks like you just might be able to specify a URL for the globalOpts.installDir property. But more likely, I'll have to make some changes to enable retrieval of files using a separate URL. But try setting installDir to a full URL and let me know what happens ... (not sure how CORS security factors in here ...)

ericmandel commented 4 years ago

Ah ... I never tried this before ...but a little experimenting shows that you can indeed use globalOpts.installDir to load files from a remote server. My test HTML page is below. Here are some notes:

JS9.globalOpts.installDir points to the remote server install directory. I was able to load everything (e.g. the js9 logo) except ...
there appears to be a CORS problem loading the wasm file remotely, so I turned off wasm with the JS9.globalOpts.useWasm property. JS9 will fallback to using asm.js, which is a bit slower, but will work fine. I'll look into the wasm problem but it might just be a restriction for now ...
I pointed the JS9.globalOpts.helperURL to the remote helper, otherwise it tries to connect to the local helper.

Image load and remote analysis works as expected, so give this a try and let me know what you find.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
   "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=Edge;chrome=1" > 
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link type="image/x-icon" rel="shortcut icon" href="https://js9.si.edu/js9/favicon.ico">
  <link type="text/css" rel="stylesheet" href="https://js9.si.edu/js9/js9support.css">
  <link type="text/css" rel="stylesheet" href="https://js9.si.edu/js9/js9.css">
  <link rel="apple-touch-icon" href="https://js9.si.edu/js9/images/js9-apple-touch-icon.png">
  <script type="text/javascript" src="https://js9.si.edu/js9/js9prefs.js"></script>
  <script type="text/javascript" src="https://js9.si.edu/js9/js9support.min.js"></script>
  <script type="text/javascript" src="https://js9.si.edu/js9/js9.min.js"></script>
  <script type="text/javascript" src="https://js9.si.edu/js9/js9plugins.js"></script>
  <title>JS9 using remote files</title>
</head>
<body>
    <div class="JS9Menubar"></div>
    <div class="JS9"></div>
    <div style="margin-top: 2px;"><div class="JS9Colorbar"></div></div>
    <p>
    <a href='javascript:JS9.Load("js9/fits/casa.fits.gz", {scale:"log", parentFile:"fits/casa.fits"});'>load CAS-A (Chandra)</a>
    <script type="text/javascript">
      // CORS apparently prevents loading wasm ... need to check into this
      JS9.globalOpts.useWasm=false;
      // get the helper from the remote server
      JS9.globalOpts.helperURL="https://js9.si.edu";
      // change the install dir to match the remote server
      JS9.globalOpts.installDir="https://js9.si.edu/js9/";
    </script>
<p>
</body>
</html>

ericmandel commented 4 years ago

Of course, you can use js9prefs.js instead of setting the globalOpts directly in the web page. Here is my test prefs file (having removed the globalOpts property assignments from the web page):

var JS9Prefs = {
  "globalOpts": {"helperType":       "nodejs",
         "helperPort":       2718, 
         "helperCGI":        "./cgi-bin/js9/js9Helper.cgi",
         "useWasm":          false,
         "helperURL":        "https://js9.si.edu",
         "installDir":       "https://js9.si.edu/js9/",
         "fits2png":         false,
         "debug":        0,
         "loadProxy":        true,
         "workDir":      "./tmp",
         "workDirQuota":     100,
         "dataPath":         "$HOME/Desktop:$HOME/data",
         "analysisPlugins":  "./analysis-plugins",
         "analysisWrappers": "./analysis-wrappers"},
  "imageOpts":  {"colormap":         "grey",
         "scale":            "linear"}
}

timbeccue commented 4 years ago

Thank you for the detailed investigation! I was waiting to provide a more complete response, but maybe I should have shared my progress a little sooner.

Anyways, your comment on the importance of js9.css was helpful. In case someone else finds this useful: I was importing the css in my vue component's style section using @import url("<remote url>"), but changing it to a element in the header fixed the problem of where to look for everything else. I also had to enable cors on my apache server where I installed js9.

So now, almost everything works as expected. I can open files, conduct client-side analysis, wasm works, all the icons load, the backend helper connects... the only problem I can find is that I can't upload an image to enable server-side analysis. The last issue I opened had a similar problem, but I got a nice error alert in the browser, but now when I click Analysis->Upload FITS to make tasks available, nothing happens. There's no error message in the browser console, nothing in the node helper logs. The closest hint I can find is the websocket response which contains 431[{stdout: null, stderr: null, errcode: 0, encoding: "ascii"}]. Interestingly, when I run js9 from an html file served from the same server, such as the default index.html that comes with the installation, I can upload to run server-side tasks with no problem.

By the way, I ran the html from your post above, but it didn't completely work. Specifically: the window loaded and connected to the helper, the js9logo could be displayed, but I could not load the casa.fits.gz image you linked (it looked for a relative path, which wasn't found), and I was unable to load a url via cors (the dialog box opened but remained blank). I successfully opened a local fits file, but trying to upload it for server-side tasks had the same problem where nothing seems to happen.

Thanks for your help!

ericmandel commented 4 years ago

changing it to a element in the header

I'm glad we're making good progress ... if you want to serve the js9.css file from vue's component style section, you only have to set globalOpts.installDir manually ... either way should work.

when I click Analysis->Upload FITS to make tasks available, nothing happens.

I'm guessing that the worker was not initialized, since the upload call silently returns unless everything is set up. To debug: once js9 is loaded, you can look at the JS9.worker property and see if its null. Mine looks like this:

The non-null JS9.worker.worker property is the handle of the created worker. Also, when the upload is requested, you should see the following in your js9node.log:

The first line is a check to make sure the user has not exceeded the allowed amount of space. This happens before any uploadstarts ... and should happen immediately if the worker is set up correctly. The second line is the request to process the upload (I think there will be several of them, depending on file size). So if you have nothing at all in the js9node log, I'm guessing this process never got started ... cause the work was not initialized. (Or because you loaded a PNG file instead of a FITS file, I don't think PNGs can be uploaded.)

By the way, I ran the html from your post above, but it didn't completely work.

Yeah, it was just an example, with relative path on my machine etc. Sorry for not being clear ...

timbeccue commented 4 years ago

If you want to serve the js9.css file from vue's component style section, you only have to set globalOpts.installDir manually ... either way should work.

Ahh, of course. It's sinking in now; thanks for reiterating.

I'm guessing that the worker was not initialized, since the upload call silently returns unless everything is set up. To debug: once js9 is loaded, you can look at the JS9.worker property and see if its null.

Hmm, ok. Let me contrast the two places I'm testing js9: my vue app ("the vue page"), the frontend site that is hosted separately from js9, and the js9 index.html ("the js9 page"), served from the same server that has the js9 node helper. To reiterate, the js9 page so far can do everything found on https://js9.si.edu, ie, it works.

On firefox, both the js9 page and the vue page have a non-null JS9.worker.worker property. On chrome, the js9 page has a non-null JS9.worker.worker, but the vue page returns JS9.worker as undefined. However, on both browsers, the js9 page uploads just fine and the vue page returns silent.

Also, when the upload is requested, you should see the following in your js9node.log:

Oops, I failed to notice last time, but a request to upload the fits file actually does prompt a quotacheck. However, only on the js9 page does it continue with the desired uploadfits execution. I had set the JS9.globalOpts.workDirQuota to a generous 100000 too, so that shouldn't be the source of the issue (is there a good way to verify that this setting has been applied to the instance?).

(Or because you loaded a PNG file instead of a FITS file, I don't think PNGs can be uploaded.)

Nope, but it didn't hurt to check.

Seems like a big hint is that the backend is working just fine when I'm serving html from the same server. I'm just not quite sure what possibilities that should eliminate, and which ones to look into further.

ericmandel commented 4 years ago

Thanks for the clear and patient explanation, it can be pretty hard to understand a setup I've never seen ...

I think I have your setup reproduced and I see the same problems ...

It doesn't look like the worker is acting on the postMessage sent to it by js9 ... I don't know whether that is just a bug in my code or a limitation of this configuration re: workers (e.g. a CORS problem). The fact that Chrome returns an undefined worker from the vue page might be a bad sign ... but let me look into it and get back to you ... might take a bit of time ...

ericmandel commented 4 years ago

This is starting to look like a CORS problem. In my test setup, which loads JS9 files from js9.si.edu, I can't create a worker in Safari or Chrome by supplying the js9worker.js remote URL (this error is being swallowed by JS9's init routine):

Chrome:

Safari:

It appears to work in Firefox, but then postMessage does not actually send messages from JS9 to the worker, so the created worker is useless.

My main web site does not have CORS configured, but perhaps you can check your vue server to make sure it has CORS enabled for this type of asset?

I'll keep looking around ...

timbeccue commented 4 years ago

I can confirm that this seems to be the problem for me too. The browser inspector shows that the js9worker script loaded on my js9 page where everything works, and did not load on my vue page (note: it is absent from the browser network inspector as if there was no request made for the script at all. Is that what you meant by the JS9 init hiding the error?). I'll take a closer look into CORS settings on the frontend.

ericmandel commented 4 years ago

Yes, I wrap the worker startup in a try/catch and just hide the error. We have a lot of unsophisticated users and I didn't want them to get an obscure error right off the bat.

But I think we can fix your problem by retrieving the worker in a special way if the url is not local to the web page. Assuming you can edit js9.js and install a modified version, try replacing this routine around line 11184:

// create new web worker
JS9.WebWorker = function(url){
    if( url.match(JS9.URLEXP) ){
    // avoid cross-origin problems if the webworker is being retrieved
    // from somewhere other than the local host
    JS9.fetchURL(null, url, null, (blob) => {
        this.worker = new Worker(URL.createObjectURL(blob));
    });
    } else {
    // ordinary retrieval of a local file
    this.worker = new Worker(url);
    }
    this.worker.onmessage = JS9.WebWorker.prototype.msgHandler.bind(this);
    this.handlers = [];
};

You'll still need to have CORS capability on the "vue page" server, but this should fetch the URL (with CORS permission) into a blob and then call create worker on the blob.

If it works in your case, I'll make double sure it works in the ordinary case, and then update github ... tonight or tomorrow.

ericmandel commented 4 years ago

@timbeccue I got this working ... and updated into GitHub ...

The code above was not quite correct, it has to be:

JS9.WebWorker = function(url){
    const finishup = () => {
    this.worker.onmessage = JS9.WebWorker.prototype.msgHandler.bind(this);
    this.handlers = [];
    };
    if( url.match(JS9.URLEXP) ){
    // avoid cross-origin problems if the webworker is being retrieved
    // from somewhere other than the local host
    // this leaks a small bit of memory (no revokeObjectURL call)
    JS9.fetchURL(null, url, null, (blob) => {
        this.worker = new Worker(URL.createObjectURL(blob));
        finishup();
    });
    } else {
    // ordinary retrieval of a local file
    this.worker = new Worker(url);
    finishup();
    }
};

With that code and CORS turned on in the main JS9 web site, I can now use the main site's files within a local web page, upload FITS etc. The wasm file now works as well, so there is no need to turn off wasm. Here is my test web page, which should work for you out of the box. Note that there is now only one global parameter that has to be set: installDir:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
   "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=Edge;chrome=1" > 
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link type="image/x-icon" rel="shortcut icon" href="https://js9.si.edu/js9/favicon.ico">
  <link type="text/css" rel="stylesheet" href="https://js9.si.edu/js9/js9support.css">
  <link type="text/css" rel="stylesheet" href="https://js9.si.edu/js9/js9.css">
  <link rel="apple-touch-icon" href="https://js9.si.edu/js9/images/js9-apple-touch-icon.png">
  <script type="text/javascript" src="https://js9.si.edu/js9/js9prefs.js"></script>
  <script type="text/javascript" src="https://js9.si.edu/js9/js9support.min.js"></script>
  <script type="text/javascript" src="https://js9.si.edu/js9/js9.min.js"></script>
  <script type="text/javascript" src="https://js9.si.edu/js9/js9plugins.js"></script>
  <title>JS9 using remote files</title>
</head>
<body>
    <div class="JS9Menubar"></div>
    <div class="JS9"></div>
    <div style="margin-top: 2px;"><div class="JS9Colorbar"></div></div>
    <script>
      JS9.globalOpts.installDir = "https://js9.si.edu/js9/";
    </script>
</body>
</html>

Let me know if this works for you ...

timbeccue commented 4 years ago

Awesome, thank you!

I was actually mid response with not-so-promising results, but I'm glad you beat me to the punch.

Initial results with your latest post look great, and I'll fiddle around a bit more to make sure that it all works as expected.

timbeccue commented 4 years ago

I am getting an error in chrome (not firefox) as soon as I load the page:

Uncaught (in promise) RuntimeError: memory access out of bounds
    at wasm-function[389]:662
    at wasm-function[152]:24
    at wasm-function[560]:22
    at Module._maxFITSMemory (*/js9/astroemw.js:4:116678)
    at ccall (*/js9/astroemw.js:4:7038)
    at Object.Module.maxFITSMemory (*/js9/astroemw.js:4:138308)
    at Object.JS9.fitsLibrary (*/js9/js9.js:18534:15)
    at Object.JS9.initFITS (*/js9/js9.js:20726:6)
    at HTMLDocument.<anonymous> (*/js9/js9.js:24185:6)
    at HTMLDocument.dispatch (*/js9/js9support.js:5222:27)
(anonymous) @ wasm-005c7a5e-389:313
(anonymous) @ wasm-005c7a5e-152:14
(anonymous) @ wasm-005c7a5e-560:13
Module._maxFITSMemory @ astroemw.js:formatted:5591
ccall @ astroemw.js:formatted:346
Module.maxFITSMemory @ astroemw.js:formatted:6640
JS9.fitsLibrary @ js9.js:18534
JS9.initFITS @ js9.js:20726
(anonymous) @ js9.js:24185
dispatch @ js9support.js:5222
elemData.handle @ js9support.js:5030
trigger @ js9support.js:8217
(anonymous) @ js9support.js:8285
each @ js9support.js:378
each @ js9support.js:173
trigger @ js9support.js:8284
Module.onRuntimeInitialized @ astroemw.js:formatted:12
doRun @ astroemw.js:formatted:5747
run @ astroemw.js:formatted:5759
runCaller @ astroemw.js:formatted:5723
removeRunDependency @ astroemw.js:formatted:707
receiveInstance @ astroemw.js:formatted:766
receiveInstantiatedSource @ astroemw.js:formatted:770
Promise.then (async)
instantiateArrayBuffer @ astroemw.js:formatted:775
instantiateAsync @ astroemw.js:formatted:792
createWasm @ astroemw.js:formatted:803
Module.asm @ astroemw.js:formatted:815
(anonymous) @ astroemw.js:formatted:5276

The result is that I'm unable to open files, instead receiving the alert JS9 ERROR: can't bunzip2 to virtual file: <filename>, which I assume is due to exceeding some chrome memory safeguards.

This has only appeared since pulling your recent changes, so perhaps it's related? Let me know if you prefer we open this in a new github issue instead.

ericmandel commented 4 years ago

Try turning off wasm again:

useWasm: false

ericmandel commented 4 years ago

P.S. Have you been keeping your code updated, so that you were testing with all but the most recent changes?

P.P.S. We don't need a new issue, as this is almost certainly related. If turning off wasm works, I'll have to think about how to isolate this enough to be able to make a report to the Emscripten/wasm developers.

ericmandel commented 4 years ago

And finally ... there also might be a caching problem, if your previous version is a bit old, between the astroemw.js and astroemw.wasm file. So you can try emptying the cache (with and without using wasm).

timbeccue commented 4 years ago

Well of course, clearing the cache did the trick. :roll_eyes: No need to touch wasm.

Many thanks again for the prompt and thorough assistance with everything!

timbeccue commented 4 years ago

I found another issue that I think is related. In short: the js9worker script can load more than once, and when it does, the js9 instance won't open fits images, alerting JS9 ERROR: can't bunzip2 to virtual file and/or JS9 ERROR: virtual FITS file is missing for getFITSImage().

I'll admit up front that, since I'm working on a single page app (SPA), I'm probably not loading js9 the way it was designed. Could you describe what exactly triggers the js9worker script to load? If I can design the page so that it only does that once, I think the problem will be solved.

The way I'm currently doing things in Vue, js9 sits inside a component that might be added or removed from the page multiple times without a full page reload (eg. opening/closing a modal 'popup' window to do image analysis in js9). For this component, I have the four js9 scripts and two css files programmatically appended to the document head when the component loads, and removed from the head when the component is removed. This works well for the first load/unload cycle, but after n loads, the browser network tab shows the js9worker script loaded n times, producing the aforementioned problem.

I also did a quick test of loading the js9 scripts in the head "normally" (ie. hardcoded in Vue's main index.html file), and running JS9.init() whenever the component with the <div class="JS9" /> element is loaded, but that had similar results.

Any ideas?

ericmandel commented 4 years ago

JS9.init() is called when the document is "ready" (in the jquery sense), so I'm guessing that its somehow being triggered by your partial page reload. I also expect the trouble is that the Emscripten module is being initialized more than once. This is where CFITSIO lives and I have no idea what happens when its gets reinitialized.

You should be able to skip the Js9.init() call if you add data-js9init="false" to the JS9 display div (well, any div actually):

<div class="JS9" data-js9init="false"></div>

Let me know if this works ... I added the capability after a similar request, but it doesn't get exercised in the course of ordinary testing ...

timbeccue commented 4 years ago

The feature works, but I'm not sure it helps. The problem is that whenever the component with js9 closes and loads again, it needs to also run the JS9.init() sequence again.

Do you know if there's a way I could hard-reload the emscripten module whenever I close the js9 component, so it works as if the page did a complete reload?

ericmandel commented 4 years ago

Sorry I don't understand why JS9.init() needs to be run again. Is the JS9 global var somehow deleted? Or not valid? And if so, why does the global JS9 var go away, but not the Emscripten ones (Module and my derivative Astroem). Can you please investigate and let me know if any of the following objects still exist and are non-null, i.e. with lots of good stuff in them: JS9, Module, Astroem. And, if JS9 still is available, then does JS9.worker exist and have a worker property?

It sounds like you need a partial execution of JS9.init() but I'll need to understand better the state of those global objects. I could, for example, sense the presence of Astroem and its properties, skip the Emscripten init, and sense the existence of the JS9 worker and skip the worker init ...

timbeccue commented 4 years ago

Sorry, I should have been clearer: I believe JS9.init() needs to be run again because running JS9.load after the component has been removed and added again gives the error JS9 ERROR: can't find JS9 display with id: <the ID I use>.

After the first JS9.init, the objects JS9, Module, and Astroem are present even when the JS9 component is removed or is recreated. However, the JS9.worker is undefined whenever the component is removed, and it comes back whenever JS9.init is run again. Perhaps this is a helpful clue?

I also tried creating a new instance (as demoed in js9create.html) whenever the component reloads... but I think without running init(), the worker stays undefined.

Thanks for your help!

ericmandel commented 4 years ago

Are you re-using display id when bringing up a new component? I'm not a DOM expert, but I wonder if that could cause some confusion between the new and deleted divs. Because the JS9 display id is just the id of the div, and those are supposed to be unique, right? So not sure about reuse ... a quick look on the net doesn't make it look promising but again, I'm not an expert.

But if it is possible to use a different JS9 id every time you bring up a component, you would simply add the {display: newid} arg at the end of load:

JS9.Load("foo.fits", {display: newid})

And if that works -- except for the absence of the worker -- you could simply re-init the worker each time, something like:

    if( window.Worker and !JS9.worker){
    try{ JS9.worker = new JS9.WebWorker(JS9.InstallDir(JS9.WORKERFILE)); }
    catch(e){ /* empty */ }
    }

I don't know why the worker disappears each time, I wasn't aware that it was tied to the calling div or anything subtle like that ...

ericmandel commented 4 years ago

Just FYI, I called JS9.LoadWindow() using the same id repeatedly and it works as expected:

JS9.LoadWindow("fits/casa.fits.gz", {id: "foo"}, "light")

... including the fact that the worker does not go away ...

timbeccue commented 4 years ago

Thanks, you've given me plenty of ideas to play with. Hopefully it's something I can figure out without needing to modify the source code. I'll work more on it and report back if anything interesting pops up.

ericmandel commented 4 years ago

Final suggestion: is it possible that JS9 is telling the truth and that the div is not ready when you call JS9.Load()? Wrapping it in a setTimeout() will tell you in a jiffy. If that turns out to be the case, you can use a mutation observer to wait for the DOM. JS9 uses the jquery-based arrive. So when I am awaiting a light window:

$("#dhtmlwindowholder").arrive(`#${uwin}`, {onceOnly: true}, () => {
    uim.moveToDisplay(uwin);
});

timbeccue commented 4 years ago

I ended up finding a working solution by leaving JS9 constantly running, and dynamically hiding and showing it as needed. If performance becomes an issue, I think I could find a way to duplicate the functionality in js9create.html. Thanks for the help!

ericmandel commented 4 years ago

That sounds like a good plan, too. Let me know if you run into other problems ...

ericmandel / js9

Using js9 on a separate server #65