implement dual-window zoom-pan JS9 display

[o-smirnov]

As discussed in https://github.com/ratt-ru/radiopadre-devel/issues/2#issuecomment-403048918

@ericmandel: I've given it a stab, just to see what it looks like, and to get some feel for events in JS9. Haven't got the panning in it yet because I wanted to start easy, just sync the colormaps, as discussed at the bottom of https://js9.si.edu/js9/help/localtasks.html.

    <div id="centerdiv">
    <TABLE>
    <TR>
        <TD id="rebin-top">
            <div class="JS9Menubar" id="rebin-JS9_Menubar"></div>
            <div class="JS9" data-height="600" id="rebin-JS9"></div>
            <div style="margin-top: 2px;"><div class="JS9Colorbar" id="rebin-JS9_Colorbar"></div></div>
        </TD>
        <TD id="zoom-top">
            <div class="JS9Menubar" id="zoom-JS9_Menubar"></div>
            <div class="JS9" data-height="600" id="zoom-JS9"></div>
            <div style="margin-top: 2px;"><div class="JS9Colorbar" id="zoom-JS9_Colorbar"></div></div>
        </TD>
        <TD>
            <TABLE>
                <TR><TD><div class="JS9Panner" data-width="200" data-height="200" id="rebin-JS9_Panner"></div></TD></TR>
                <TR><TD><div class="JS9Panner" data-width="200" data-height="200" id="zoom-JS9_Panner"></div></TD></TR>
                <TR><TD><div class="JS9Magnifier" data-width="200" data-height="200" id="zoom-JS9_Magnifier"></div></TD></TR>
            </TABLE>
        </TD>
    </TR>
    </TABLE>
    </div>

    <script type="text/javascript">
      $(document).ready(function(){
          $("#centerdiv").draggable({
            handle: "#JS9Menubar",
            opacity: 0.35
          });
        JS9.AddDivs("zoom-JS9", "rebin-JS9")
        JS9.Preload("/./combined-4M5S-robust0_5.fits", {fits2fits:true,xcen:2048,ycen:2048,xdim:4096,ydim:4096,bin:4}, {display: "rebin-JS9"});
        JS9.Preload("/./combined-4M5S-robust0_5.fits", {fits2fits:true,xcen:2048,ycen:2048,xdim:1024,ydim:1024,bin:1}, {display: "zoom-JS9"});

        var sync_colormaps = function (im)
        {
            var i, tim, obj;
            console.log("syncing colormap from ", im)
            JS9.globalOpts.extendedPlugins = false;
            obj = JS9.GetColormap({display: im});
            console.log(obj)
            for(i=0; i<JS9.displays.length; i++){
                tim = JS9.displays[i].image;
                if( tim && (tim !== im) ){
                JS9.SetColormap(obj.colormap, obj.contrast, obj.bias, {display: tim});
                }
            }
            JS9.globalOpts.extendedPlugins = true;
        }

        JS9.RegisterPlugin("MyPlugins", "allconstrastbias",
                   function(){return;},
                   {onchangecontrastbias: sync_colormaps,
                     winDims: [0, 0]});

        JS9.RegisterPlugin("MyPlugins", "allcolormap",
                   function(){return;},
                   {onsetcolormap: sync_colormaps,
                     winDims: [0, 0]});

        JS9.RegisterPlugin("MyPlugins", "allscale",
                   function(){return;},
                   {onsetscale: function(im){
                       var i, tim, obj;
                       console.log("syncing scale from ", im)
                       JS9.globalOpts.extendedPlugins = false;
                       obj = JS9.GetScale({display: im});
                       console.log(obj)
                       for(i=0; i<JS9.displays.length; i++){
                           tim = JS9.displays[i].image;
                           if( tim && (tim !== im) ){
                               JS9.SetScale(obj.scale, obj.smin, obj.smax, {display: tim});
                           }
                       }
                       JS9.globalOpts.extendedPlugins = true;
                   },
                   winDims: [0, 0]});

      });
    </script>

First problem is that the syncing doesn't happen. I get this on the console from inside the plugin:

syncing colormap from  b.Image {type: "image", display: b.Display, params: {…}, tmp: {…}, cmapObj: b.Colormap}
js9.min.js:419 error in onsetcolormap: MyPluginsallcolormap [can't find JS9 display with id: [object Object]]

Looks like JS9.GetColormap({display: im}); call is not happy with its argument.

Also, it looks like the fits2fits:true argument is being ignored, as I'm seeing full images being requested and transferred -- no transforms are requested on the helper. When I go back to using fits2fits on a single display, the helper kicks in and small images are served as appropriate.

[o-smirnov]

Also, it looks like the fits2fits:true argument is being ignored

OK, this was due to the JS9.AddDivs() call, which (as you explained elsewhere) is unnecessary in this scenario. The first problem remains though.

[o-smirnov]

For our images, https://github.com/ericmandel/js9/issues/40 is also going to be relevant.... no point in syncing scales when binning changes the underlying scale of the first image....

[ericmandel]

I'll look into this a bit today, not obvious what is going on ...

[ericmandel]

Oh dear ... looks like I committed the cardinal sin of calling this.setColormap() inside the JS9.Image constructor, before the new image was ready. That is fixed and updated in GitHub.

I also added a JS9.globalOpts.xeqPlugins property to turn on/off plugin callbacks, to be used instead of JS9.globalOpts.extendedPlugins, the latter having been meant for a slightly different purpose. So the code above probably should be changed (thought using extendedPlugins will still work, in this particular case). I've changed the documentation you read and copied to reflect this ...

Finally, you might need to remove the AddDivs before the JS9.Preloads(), depending on how this is actually being called. My test divs were added directly to the web page, not injected, so the AddDivs was redundant and resulted in grey displays.

[o-smirnov]

Sorry, still doesn't work. (I have removed the AddDivs already, as a result of our previous conversation.)

Getting this on the console:

From this bit of code:

        var sync_colormaps = function (im)
        {{
            var i, tim, obj;
            JS9.globalOpts.xeqPlugins = false;
            console.log("syncing colormap from ", im)
            obj = JS9.GetColormap({{display: im}});
            console.log("colormap is", obj);
            for(i=0; i<JS9.displays.length; i++){{
                tim = JS9.displays[i].image;
                if( tim && (tim !== im) ){{
                JS9.SetColormap(obj.colormap, obj.contrast, obj.bias, {{display: tim}});
                }}
            }}
            JS9.globalOpts.xeqPlugins = true;
        }}

So pretty sure GetColormap() is refusing to recognize my im argument.

[ericmandel]

I'll debug this again, but I thought it was working and well understood. No chance you're having a caching problem, picking up the old code ...

[ericmandel]

Hmmm ... your code works for me, without error:

and colormaps change together as expected. Gotta be picking up the old code ...

[ericmandel]

Here is my jup.html file, used in the test above:

<html>
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=Edge;chrome=1" > 
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link type="image/x-icon" rel="shortcut icon" href="./favicon.ico">
  <link type="text/css" rel="stylesheet" href="js9support.css">
  <link type="text/css" rel="stylesheet" href="js9.css">
  <script type="text/javascript" src="js9prefs.js"></script>
  <script type="text/javascript" src="js9support.min.js"></script>
  <script type="text/javascript" src="js9.js"></script>
  <script type="text/javascript" src="js9plugins.js"></script>
  <title>JS9 testing</title>
</head>
<body>
    <div id="centerdiv">
    <TABLE>
    <TR>
        <TD id="rebin-top">
            <div class="JS9Menubar" id="rebin-JS9_Menubar"></div>
            <div class="JS9" data-height="600" id="rebin-JS9"></div>
            <div style="margin-top: 2px;"><div class="JS9Colorbar" id="rebin-JS9_Colorbar"></div></div>
        </TD>
        <TD id="zoom-top">
            <div class="JS9Menubar" id="zoom-JS9_Menubar"></div>
            <div class="JS9" data-height="600" id="zoom-JS9"></div>
            <div style="margin-top: 2px;"><div class="JS9Colorbar" id="zoom-JS9_Colorbar"></div></div>
        </TD>
        <TD>
            <TABLE>
                <TR><TD><div class="JS9Panner" data-width="200" data-height="200" id="rebin-JS9_Panner"></div></TD></TR>
                <TR><TD><div class="JS9Panner" data-width="200" data-height="200" id="zoom-JS9_Panner"></div></TD></TR>
                <TR><TD><div class="JS9Magnifier" data-width="200" data-height="200" id="zoom-JS9_Magnifier"></div></TD></TR>
            </TABLE>
        </TD>
    </TR>
    </TABLE>
    </div>

    <script type="text/javascript">
      $(document).ready(function(){
          $("#centerdiv").draggable({
            handle: "#JS9Menubar",
            opacity: 0.35
          });

        JS9.Preload("/./combined-4M5S-robust0_5.fits", {fits2fits:true,xcen:2048,ycen:2048,xdim:4096,ydim:4096,bin:4}, {display: "rebin-JS9"});
        JS9.Preload("/./combined-4M5S-robust0_5.fits", {fits2fits:true,xcen:2048,ycen:2048,xdim:1024,ydim:1024,bin:1}, {display: "zoom-JS9"});

        var sync_colormaps = function (im)
        {
            var i, tim, obj;
            console.log("syncing colormap from ", im)
            JS9.globalOpts.extendedPlugins = false;
            obj = JS9.GetColormap({display: im});
            console.log(obj)
            for(i=0; i<JS9.displays.length; i++){
                tim = JS9.displays[i].image;
                if( tim && (tim !== im) ){
                JS9.SetColormap(obj.colormap, obj.contrast, obj.bias, {display: tim});
                }
            }
            JS9.globalOpts.extendedPlugins = true;
        }

        JS9.RegisterPlugin("MyPlugins", "allconstrastbias",
                   function(){return;},
                   {onchangecontrastbias: sync_colormaps,
                     winDims: [0, 0]});

        JS9.RegisterPlugin("MyPlugins", "allcolormap",
                   function(){return;},
                   {onsetcolormap: sync_colormaps,
                     winDims: [0, 0]});

        JS9.RegisterPlugin("MyPlugins", "allscale",
                   function(){return;},
                   {onsetscale: function(im){
                       var i, tim, obj;
                       console.log("syncing scale from ", im)
                       JS9.globalOpts.extendedPlugins = false;
                       obj = JS9.GetScale({display: im});
                       console.log(obj)
                       for(i=0; i<JS9.displays.length; i++){
                           tim = JS9.displays[i].image;
                           if( tim && (tim !== im) ){
                               JS9.SetScale(obj.scale, obj.smin, obj.smax, {display: tim});
                           }
                       }
                       JS9.globalOpts.extendedPlugins = true;
                   },
                   winDims: [0, 0]});

      });
    </script>
</body>
</html>

[o-smirnov]

You're absolutely right. My bad. It did use the old code somehow. Reinstalling/restarting took care of this.

Does scale syncing work right for you in my example? I get odd results -- changing the min and max doesn't propagate, but changing linear to log and back, I get NaNs for min and max in the other window. Anyway, that's a problem for tomorrow...

[ericmandel]

I'll try to look tomorrow, though I'm going to be away from computers for 1/2 day. And right now I'm in the "mean binning" branch and everything, I mean everything, is in pieces on the floor.

Have a good evening, what's left of it for you ...

[o-smirnov]

Well I've made decent progress, but am now stuck on a few persistent problems. My current code is https://gist.github.com/o-smirnov/978db3f950ae93c53bf1d22f8456b534.

• The colormaps are now correctly synced, but the colorbar of the other image is not updated to reflect the new colors.

• Changing the clipping limits does not invoke the onsetscale event. I have to actually change the scale type (e.g. reselect "linear" in the menu) to get the event to fire.

• ~This problem has reared its head again. It's still doing full downloads, and cheerfully ignoring my fits2fits:true setting. I can see the helper reports it's connecting to it -- it just doesn't get invoked! I thought it got solved by removing AddDivs(), but that was obviously a red herring.~ Edit: nevermind, found it, it was caused by an incorrect path in LoadImage (gave full URL rather than relative pat).

• The off-centre problem of https://github.com/gijzelaerr/js9notebook/issues/5#issuecomment-403961983 remains. I try loading the images with zoom:'T' to mitigate, as soon as I try to zoom in, it zooms into an off-centre region, rather than the centre of the image.

• Still something flaky going on with the SetScale() calls. You can see I've added code to reapply the saved scale and colormap every time a new representation image is loaded into the right display (otherwise it resets to defaults). After moving the region around a few times, the clipping limits reset to NaNs, despite the console indicating I'm setting them correctly (compare with console.log() statements in my code):

resetting colormap 
{colormap: "heat", contrast: 1, bias: 0.5}
resetting scale 
{scale: "linear", scalemin: -0.00003460508014541119, scalemax: 0.0001, scaleclipping: "user"}

• Similarly, playing with clipping limits in one image tends to get them into NaNs in the other image. Sounds like this is the same problem.

Apart from that, it's shaping up very nicely!

[ericmandel]

Seems like I need to spend some time getting these various problems understood and working before you waste more time on it. I strongly suspect there will be some changes needed to support changing the scaling updates. Though the off-center issue is the most worrying to me, as that has no obvious explanation.

That said, I am soon leaving my computer for most of the rest of today ... but more importantly, I am more than half-way done with adding support for averaging binned pixels. I want to concentrate on finishing that work before I look into this problem, so give me a few days ...

[o-smirnov]

No rush at all! I'm going to down tools for a few days anyway as we've got a few distractions down here (of the telescope inauguration variety)...

One problem above I solved myself. My latest code is here and seems to work well, apart from the SetScale() issues: https://gist.github.com/o-smirnov/07426ed03663c6ec229bbc97b8c9d33f

Another thought for the future: this mode is not exactly snappy. I suspect the helper is effectively re-reading the original image each time? One thing to consider for a Python helper implementation is just holding onto the last-requested image in memory, so that subsequent requests for different sections of the same image are served faster. Could also think about caching them...

[ericmandel]

I worked on the partner plugin and have it working on my desktop. To allow me to run eslint and also debug it in Chrome, I split your code up into two pieces, a test html page and javascript plugin, both of which are appended below (with .txt extensions to make GitHub happy). You'll have to fiddle with the pathnames again, sorry ...

Some notes:

1. the scale limits were not being updated because the Scale menu was not using setScale(). This is now fixed and updated.

2. the technique of turning on/off xeqPlugins to avoid recursion is not appropriate for a sophisticated plugin like this one. For example, the colorbar updates in responce to plugin events, and therefore was not being updated when we turned off these events. So I simply kept track (in each image's tmp object) of which plugin callback was being executed and avoiding recursion on that callback only.

3. I ran eslint over the code and found a bunch of variables that were undeclared and were therefore made global. This might have caused some problems. I also found a typo that probably caused the NaNs when updating scaling.

4. The partner.hml file sets the new JS9.globalOpts.binMode to 'a', which utilizes averaging instead of summing when binning. Support for averaging is updated in GitHub as well.

I still am unable to reproduce the off-center problem. We'll need to concentrate on that at some point. I have no idea right now how to think about that!

partner.html.txt

partner.js.txt

[o-smirnov]

Thanks a lot! I'll take a look as soon as I can. But, I think you forgot to add the Javascript code -- I only see two copies of the HTML.

[ericmandel]

Oops ... fixed.

[o-smirnov]

Excellent stuff, in standalone mode everything works like a charm now! As for the off-center problem, it appears to have gone away...

Getting a new failure mode when running it under Jupyter though. When loading into the partner displays, I get one of these two errors, and my mouse becomes an ever-spinning wheel:

The first case is for the 4k x 4k image, the second case is for a 513x513 image (thus, smaller than the requested fits2fits dimensions). In both cases, the network console shows a full image being downloaded.

Note that if I load an image under Jupyter with fits2fits:false, these errors do not occur.

What's different between the Jupyter case and the standalone case? My only guess is, in the former case the image is loaded as /files/name.fits, in the latter case, as ./name.fits.

Two other minor niggles:

• In standalone mode, when the low-res image is first loaded and the zoom_region is added, it proceeds to load the zoomed image in the partner display as intended (as I can tell from the console messages, and the helper), but the partner window remains blank, without the usual "spinning wheel" on it. When the helper finishes serving up the image section, the loaded image appears. This is a bit misleading to the user, since there's no indication that things are still being loaded.

• Not a bug, rather some musing on what the correct behaviour should be: when the partner displays are first loaded up, their scales are not synced. When I drag the zoom region around, the scale of the right-hand display follows the min/max values of the selected image section, while the scale of the left-hand display stays at the min/max of the binned image. This is perfectly understandable: onsetscale has not fired at this point, so the scales remain unsynced. But is this what we want? Should we, perhaps, be updating the scale of the left-hand display to match the scale of the right-hand display?

[ericmandel]

Is this a new error? It's a little hard to imagine that the changes I made in the past week would cause such errors.

The first error indicates that the section information was not set up yet, hence a width of 0. The second error indicates that CFITSIO could not read the blob passed to it as a FITS file.

Could the errors be due to trying to process a FITS file before everything is ready?? Is the partner code kicking over as soon as registration happens? You can try changing:

$(document).ready(function(){
  ... JS9.Preload()

to:

$(document).on("JS9:ready", function(){
  ... JS9.Load()

In "real life", i.e. outside Jupyter, JS9:ready only gets fired when JS9 really is ready.

Also, with regard to the first error (4k x 4k), you can try setting the bin to 1 (bin 1 will skip the stretch of code that is failing) and see what the next error is ...

[o-smirnov]

Could the errors be due to trying to process a FITS file before everything is ready?? Is the partner code kicking over as soon as registration happens?

No, can't be. I have integrated JS9 initialization into the startup of the Jupyter kernel, so it's loaded as the interface page loads, long before I execute any Jupyter cell.

Anyway, it's definitely something to do with binning. The following cell produces the "source width is 0" error:

%%html
<div id='foo'>
<div class='JS9Menubar' id='foo_JS9Menubar'></div>
<div class='JS9' id='foo_JS9'></div>
<div style='margin-top: 2px;'><div class='JS9Colorbar' id='foo_JS9Colorbar'></div></div>
</div>
<script type="text/javascript">
    JS9.AddDivs('foo_JS9');
    JS9.Load("/files/combined-4M5S-robust0_25.fits",
                     { fits2fits:true,xcen:2048,ycen:2048,xdim:4096,ydim:4096,bin:'4a' },
                     {display:"foo_JS9"})
</script> 

...but works just fine if I change it to say bin:4.

(To reproduce on your end, you're going to have to paste the JS9 loading code at the start of the cell, and change Load() to Preload()...)

[ericmandel]

If bin:4 works but bin:"4a" does not work, one explanation is that you are picking up the old version of the js9helper program, which does not recognize the "a" suffix. The other explanation is that I didn't test the js9helper program and it does not work ... I thought I did.

I'll re-test today, but please check for an old version of the js9helper program.

BTW, the resulting FITS file is in the workDir directory and you can look at it directly. It probably is not a valid FITS file.

[ericmandel]

Whoah ... hang on, I can reproduce these problems ... let me look into it.

[o-smirnov]

OK glad I'm not going crazy. :) For the record, it is the new js9helper I'm running -- and it works fine with the standalone HTML pages. Anyway, no rush, I'm off on a trip so will only be developing this sporadically over the next few days. And I've got plenty to work with already!

[ericmandel]

No, you're not crazy, but I might be crazy to have changed the bin factor from an integer to either an integer or a string containing the avg/sum specifier. There were a couple of places where I forgot to check for a string, and that caused problems. Also a 5-year old bug in how js9helper parsed one of the five cases of image section specifier.

But perhaps most importantly, I found and fixed the off-center problem! It had to do with specifying xcen, ycen, and a bin factor, blah blah. JS9 tries to pan to the specified center (which is in file coords), but in doing so needed to take the bin factor into account (since the pan position is in image coords).

Since all of these problems were found outside Jupyter, I did not do anything inside Jupyter ...

It's all in GitHub and while I tested 16 combinations of fits2fits and bin specifier, maybe you can find another case or two ...

That said, I'm working on some UI issues this week, so we can pick things up when you get back .... have a good trip!

[o-smirnov]

Excellent, thanks, that makes all the errors go away!

I've got one remaining problem under Jupyter (and only under Jupyter). The partner displays are still downloading full images, rather than asking the helper for sections. I'm pretty sure it's to do with paths:

Let's say I start js9helper and Jupyter under ~/RP-3C147/. If I now want to load ~/RP-3C147/combined-4M5S-robust0_5.fits into JS9, I must pass its URL as /notebooks/./combined-4M5S-robust0_5.fits. /notebooks is the path prefix under which Jupter's built-in web server makes local files visible.

I think the helper has no way of knowing that /notebooks/./combined-4M5S-robust0_5.fits and ~/RP-3C147/combined-4M5S-robust0_5.fits are the same file, right? So JS9 ends up bypassing the helper.

Is there any way to hack such a prefix into the helper?

[ericmandel]

I'm relieved that the errors went away and apologize for having let some problems slip through unnoticed. I try not to do that in general ...

There are two reasons fits2fits == "always" would ask the browser to load the full file instead of performing the imsection:

1. the node.js helper could not find the js9helper C program that does the image section. This should be checked first by looking at the JS9.helper.js9helper property in the browser after an image is loaded. If it's null, the js9helper program was not found and all requests for fits2fits will be ignored. That's a *nix path problem.

2. the node.js helper, as you say, could not find the data file. Are you saying that you are sending it the pathname /notebooks/./combined-4M5S-robust0_5.fits? And that this should be translated into `./combined-4M5S-robust0_5.fits? If that is the case (and JS9.helper.js9helper exists), let me think about this for a bit, I'm sure we can hack something easily -- my specialty has always been to break the rules to make things work.

[o-smirnov]

I'm relieved that the errors went away and apologize for having let some problems slip through unnoticed. I try not to do that in general ...

No need to apologize -- we are pushing this thing forward rather rapidly, bugs are bound to slip through...

1. Indeed js9helper is undefined:

But why? It works fine when I'm connecting to it from standalone HTML:

Are you saying that you are sending it the pathname /notebooks/./combined-4M5S-robust0_5.fits? And that this should be translated into `./combined-4M5S-robust0_5.fits?

Indeed. But let's get it to find the helper first!

[ericmandel]

The check on js9helper just prepends files from the OS PATH and checks for existence:

    // can we find the helper program?
    jpath = !!getFilePath(globalOpts.cmd, process.env.PATH, process.env);

so the first thing is to display the file checks. You can add a console.log statement to js9Helper.js at around line 541:

    // make up pathnames to check
    s1 = path.join(s, froot1);
console.log("checking: %s", s1); 
    if( fs.existsSync(s1) ){

and see what you get ...

[o-smirnov]

But that can't be the problem, surely. I can connect to the exact same helper process from a standalone HTML page, and everything works. It's only under Jupyter that we fail. I think we rather need to be looking at what's going on with the socket or something...

[ericmandel]

But the helper inherits the PATH from the process that starts it up, and won't that, in principle, be different when started inside and outside Jupyter?

[o-smirnov]

In this case (in all cases, actually), I'm starting the helper from a wrapper script outside Jupyter though.

On Mon, 16 Jul 2018, 15:12 Eric Mandel, notifications@github.com wrote:

But the helper inherits the PATH from the process that starts it up, and won't that, in principle, be different when started inside and outside Jupyter?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ratt-ru/radiopadre-devel/issues/4#issuecomment-405242648, or mute the thread https://github.com/notifications/unsubscribe-auth/AGK5vxX8NmJzWuWh2sYhWSX4r8Ns0ER_ks5uHJFFgaJpZM4VHnHQ .

[ericmandel]

Oh ... well ... I need to change my assumptions! And ... now that I have, you are correct ... looking more closely at the output from JS9.helper, I see that helper and connected are false, which means that a connection was not made at all between the browser and the helper. I've been under the false understanding that the connection was made successfully, so we do have a problem here ...

Let me think for a minute ....

[ericmandel]

OK, so I am a little confused, because I thought we were seeing connections to the node.js helper from within Jupyter. But perhaps I misunderstood and this was only happening from outside Jupyter? For example, do you see the connect message in the helper log from clients started inside Jupyter:

connect: 127.0.0.1 (JS9,myJS9)  [2018-7-15 23:36:20]

[ericmandel]

One thing you can do is set JS9.globalOpts.debug to 2 and you will see various debugging output, including a debugging message if the browser failed to connect to the helper.

[ericmandel]

I suspect that when you turn on debugging, you will get an error socket io object is undefined when JS9 tries to load the client socket.io.js file using an XHR call. We probably missed the fact that XHR was still not permitted by Jupyter when you made your link and got many things to work.

But thinking more about this, I believe the solution is to load the socket.io.js file explicitly when you load all of the other JS9 files, e.g. load it in a script tag after loading the other files. The URL to load will be something like this (with the port set to your chosen port) and the URL fiddled as needed:

http://localhost:2718/socket.io/socket.io.js

Jupyter should allow this, since its just an ordinary javascript file. The JS9 helper connect code will also try to load this file using XHR (and will fail again), but this should not matter, because after making the XHR call, JS9 simply looks to see whether the io object exists -- and it should exitsafter you load socket.io.js in the script tag. So everything should work from then on.

Looking at the URL above, it also occurs to me that it might not be a correct URL for your Jupyter case and that might be why XHR is failing. So if there is a different URL that is required, we can play with this as well. But the above should work.

[o-smirnov]

I suspect that when you turn on debugging, you will get an error socket io object is undefined when JS9 tries to load the client socket.io.js file using an XHR call. We probably missed the fact that XHR was still not permitted by Jupyter when you made your link and got many things to work.

You're right, I get that very error.

But thinking more about this, I believe the solution is to load the socket.io.js file explicitly when you load all of the other JS9 files, e.g. load it in a script tag after loading the other files.

I tried. Doesn't quite help. My code looks like this:

      <script type='text/javascript' src='/static/js9-www/js9prefs.js'></script>
      <script type='text/javascript'> console.log('loaded JS9 prefs 1') </script>
      <script type='text/javascript' src='/files/.radiopadre/js9prefs.js'></script>
      <script type='text/javascript'> console.log('loaded JS9 prefs 2')</script>
      <script type='text/javascript' src='http://localhost:1025/socket.io/socket.io.js'></script>
      <script type='text/javascript'> console.log('loaded socket.io')</script>

Console says:

loaded JS9 prefs 1
loaded JS9 prefs 2
loaded socket.io

...no errors, so I can only assume socket.io.js loaded correctly. But down the line I still get:

JS9 helper connect error: socket io object is undefined (or connection refused)

[o-smirnov]

Why does it have to load socket.js from the helper, BTW? Can't I place it in, e.g., the JS9 web directory, and load from there?

[o-smirnov]

OK, as a crude hack I downloaded socket.io.js from my own helper with wget, and placed a copy into Jupyter's static directory. I can now load this socket.io.js via the define/require incantations that I can still barely understand (really flying blind and drunk here), and I get JS9 helper: connect: nodejs in the console. Hurrah! Now to figure out how to do this properly rather than so crudely...

On to the next failure: as discussed in https://github.com/ratt-ru/radiopadre-devel/issues/4#issuecomment-405231643, getting "ERROR: can't find FITS file 'notebooks/combined-4M5S-robust0_25.fits'". Strangely, even making a symlink (notebooks -> .) doesn't seem to help.

[ericmandel]

Here is the problem we are up against using the simplest javascript imaginable (taken from the client api docs at https://socket.io/docs/client-api/). I can load this web page in a web browser:

<html>
<body>
<script src="http://localhost:2718/socket.io/socket.io.js"></script>
<script>
  console.log(typeof io);
  const socket = io('http://localhost:2718');
  console.log(typeof socket);
</script>
</body>
</html>

and get the expected results:

which shows that the io function is now in the environment and can be used to connect to the helper. But if I add this same code to a cell in Jupyter:

<script src="http://localhost:2718/socket.io/socket.io.js"></script>
<script>
  console.log(typeof io);
  const socket = io('http://localhost:2718');
  console.log(typeof socket);
</script>

and run it, I get this:

and the javacript console shows this:

This doesn't even use XHR, so there is something wrong with Jupyter's handling of this .js file

[ericmandel]

Why does it have to load socket.js from the helper, BTW? Can't I place it in, e.g., the JS9 web directory, and load from there?

I have to load JS9 from the helper, because the helper is not necessarily on my machine. But since you know where its located, I see no reason why you can't access it locally.

[ericmandel]

Hurrah! Now to figure out how to do this properly rather than so crudely...

Indeed! I'd still like to know why Jupyter can't succeed with my simple test case. But for now ...

[o-smirnov]

I have to load JS9 from the helper, because the helper is not necessarily on my machine.

But why couldn't you serve up socket.io.js from the same location as, e.g., js9.min.js? Just trying to understand why you're treating this bit of Javascript differently from the rest...

Indeed! I'd still like to know why Jupyter can't succeed with my simple test case. But for now ...

Probably worth it to open another Jupyter ticket on this. Might end up to be another insurmountable sandboxing problem, but at least we'll know.

Anyway, I seem to have a workaround for loading it, see above (where does socket.io.js come from, in the first place? If I make an installation script, where do I get the original?)

[o-smirnov]

Related posts:

https://stackoverflow.com/questions/30367591/how-to-add-external-javascript-file-in-ipython-notebook https://stackoverflow.com/questions/43234410/how-can-i-load-external-static-javascript-files-in-ipython-or-jupyter-notebook https://stackoverflow.com/questions/32170197/how-do-can-i-use-a-custom-js-file-under-jupyter-notebook https://stackoverflow.com/questions/16852885/ipython-adding-javascript-scripts-to-ipython-notebook

[ericmandel]

I load socket.io via XHR because:

• I don't know if a helper is available for that web page
• if it is available, I don't know if its a node.js helper (CGI is supported, with less functionality)
• if it is a node.js helper, I need to make sure the client and server are matched (since I have no idea about compatibility of versions).

where does socket.io.js come from, in the first place? If I make an installation script, where do I get the original?)

https://socket.io/

[o-smirnov]

OK, so since in my specific case I have an installation script that sets up JS9, runs the helper, etc., I'm in control of all these variables. So I guess it's perfectly legit if my script gets socket.io from cloudfare and installs it somewhere where Jupyter can serve it up to the client. Good, that's essentially solved then -- the rest is just details.

Remaining problem is with paths, as discussed here: https://github.com/ratt-ru/radiopadre-devel/issues/4#issuecomment-405618816

[ericmandel]

I'll go through some of the stackoverflow qestions along the way, but a ticket might be faster.

https://stackoverflow.com/questions/30367591/how-to-add-external-javascript-file-in-ipython-notebook

Doesn't appear to be a race condition in our case:

[ericmandel]

So I guess it's perfectly legit if my script gets socket.io from cloudfare and installs it somewhere where Jupyter can serve it up to the client. Good, that's essentially solved then -- the rest is just details.

OK, let's concentrate on the file loading problem.

[o-smirnov]

I'll go through some of the stackoverflow qestion

Don't waste too much time on that -- I pasted those up as a note to myself essentially. I'm coming around to the realization that it was a mistake to rely on HTML/JS output from Jupyter cells (at least for making persistent objects). The code is dynamically injected, and interacts with the rest of the DOM in funny ways.

Rather, any objects that need to be persistent across the document (socket.io, the JS9 namespace) should be set up in kernel.js or custom.js (which runs "up front" when setting up the Jupyter GUI). Which I've mostly figured out how to do... and I think the stackoverflow links explain some of the missing detail... so I'll come back to this later.

[ericmandel]

On to the next failure: as discussed in #4 (comment), getting "ERROR: can't find FITS file 'notebooks/combined-4M5S-robust0_25.fits'". Strangely, even making a symlink (notebooks -> .) doesn't seem to help.

Did you ever add the recommended console.log statement to js9Helper.js and see how the helper is checking the file against directories in the PATH? That might be useful before make a change to remove /notebook/ from the filename.

As described above:

The check on js9helper just prepends files from the OS PATH and checks for existence:

    // can we find the helper program?
    jpath = !!getFilePath(globalOpts.cmd, process.env.PATH, process.env);

so the first thing is to display the file checks. You can add a console.log statement to js9Helper.js at around line 541:

    // make up pathnames to check
    s1 = path.join(s, froot1);
console.log("checking: %s", s1); 
    if( fs.existsSync(s1) ){

[ericmandel]

Also, are you always adding a leading "/" to the pathname? You should definitely try removing that leading "/". Because JS9 works with the file URI, a leading slash is treated as an absolute pathname, not the top directory containing the web page. (It may be that I will need to fiddle this a bit to distinguish the two cases, but it gets messy quickly.)

[o-smirnov]

Also, are you always adding a leading "/" to the pathname?

Well I have to. If I run everything under ~/RP-3C147, then for Jupyter to serve me ~/RP-3C147/foo.fits, I have to ask it for a URL like /notebooks/foo.fits. But for the helper to help me, I have to ask for foo.fits or ./foo.fits.

And as I write this, I realize that's exactly how I can work around it. I just need to construct the URL differently depending on whether I use fits2fits or not...