Loading files from the server with the node helper

[timbeccue]

Hi Eric,

I'm wondering if you have any advice... I'm having a problem with loading images from the server when using the node helper.

Without the node helper, the default index.html will load casa.fits.gz with no problem.

With the helper enabled, the image does not load and I get the following alert:

JS9 Error: js9 helper:error while loading shared libraries: libcfitsio.so.8: cannot open shared object file: No such file or directory. 

The node helper logs report: exec: /var/www/js9/analysis-wrappers/js9Xeq [imsection,-parent,../js9/fits/casa.fits.gz,4096,4096,1] [11/14/2019, 18:03:10] , which appears to be the correct path relative to the node helper.

With other demo pages, I get the same error with the helper log listing an absolute path to the image on the server: exec: /var/www/js9/analysis-wrappers/js9Xeq [imsection,-parent,/var/www/js9/fits/snr.fits,4096,4096,1] [11/14/2019, 18:08:17]

I have no problems when loading an image into the browser first and then uploading to the server, and server analysis tasks work without issue.

The motivation behind this is to open urls via proxy, and have the file already available on the server. But currently this produces the same error alert.

Thanks for taking a look!

[ericmandel]

Hi Tim,

That error means that the cfitio library is either not on the server or else is not in a standard location. Assuming the latter, note that programs look for shared libraries in /usr/lib, /usr/local/lib and maybe some others standard locations, and if you installed cfitsio elsewhere, it won't be found at runtime.

On Linux, the easy solution is to set the environment variable LD_LIBRARY_PATH to include the directory in which the shared library is located. For example, on one of my Linux boxes I install libraries in a non-standard /proj/rd/linux64/lib64 directory, and so I have this set:

eric:~$ echo $LD_LIBRARY_PATH
/proj/rd/linux64/lib64

I think you can also pass the -R[path] to the link command so that it remember where to look at runtime but I forget the syntax: I generally use LD_LIBRARY_PATH since all of my shared libraries are in the same place.

The final option is to link against unshared libraries, so that the library becomes part of the executable. You would do this at link time by specifying the .a library file explicitly in the link path, because the linker will generally prefer to link the shared library above the unshared, if both are present at link time.

I am surprised that imsection is being run at all. Are you passing the parentFile property on load?

[ericmandel]

huh, I can't find any discussion of -R on the link line, so forget it, just use LD_LIBRARY_PATH ... which, in case it wasn't obvious, needs to be in the environment before the helper is started up ...

[timbeccue]

The LD_LIBRARY_PATH fix doesn't seem to work (I included my steps below). However, I noticed that the problem goes away when I set globalOpts.fits2fits=false in the js9prefs files, and likewise reappears when set to true. Maybe this helps to pinpoint the issue?

# stop the current node helper

sudo su
mkdir /cfitsio
cd /home/ubuntu
git clone https://github.com/healpy/cfitsio
cd cfitsio
./configure --prefix=/cfitsio
make
make install
chown -R ubuntu:ubuntu /cfitsio

exit #back to normal user
export LD_LIBRARY_PATH='/cfitsio'
node /var/www/js9/js9Helper.js

Each of these steps appeared to work as expected, but I get the same error alert when I load index.html.

[ericmandel]

Check the actual location of the libcfitsio.so library. You have to use something like this:

export LD_LIBRARY_PATH='/cfitsio/lib'

since the .so won't be in /cfitsio but in a subdirectory like lib or lib64.

Are you using fits2fits for a reason? It's meant for really large files that you don't want to download entirely into the browser. But unless you have that situation, you can leave it turned off, in which case the browser will download the entire FITS file, keep it in memory, and extract sections from the in-memory virtual file. But I don't know how big your files are ...

[timbeccue]

This fixes the problem loading files from the default ./fits directory. Now (without fits2fits), the index.html page loads as expected and server analysis works. Thanks!

When I load with fits2fits enabled, I get this alert: JS9 ERROR: could not find /var/www/js9/tmp/917e2ae7-22a5-4f30-aa11-fcfb05cebb80/snr_4096_4096_1.fits.gz

I checked the directory and the file is actually there (and is a valid fits file).

I get a similar error when proxy loading from a url. Again, the file is saved correctly but throws a 'doesn't exist' error when the browser attempts to retrieve it.

If I open a local file or open a link directly in the browser via CORS, and then upload it to the server, there are no problems and server analysis works great.

So it seems like the general problem description is roughly: files cannot be found when they are saved in the temporary directory before being read by the browser client, but files that go through the browser first have no problem working in the temporary directory.

[timbeccue]

I wonder if it has anything to do with how I specify the temporary directory. I had been using an absolute path, but I switched to ./tmp and from a brief look, fits2fits appears to be working. Proxy loading also opens the image in the js9 instance... except I need to upload it again before server tasks are available. I'll keep looking...

[ericmandel]

Please remind me: is the web page server a different machine from the JS9 helper machine? Because that would cause this problem: the retrieved image is stored on the JS9 helper machine and then the FITS retrieval is requested from the web page server ... where as (somehow) it should be requested from a different machine??

[timbeccue]

The helper is run on the same machine as the server.

[ericmandel]

The helper is run on the same machine as the server.

OK, that's good!

Hmmm ... absolute paths ... I guess you probably can't do that because the web server will not allow you to access a file outside its directory domain -- you can't load any URL resolving to a path that is outside a server domain, right? So probably you have to make a link inside the web directory to store files outside the directory (and have the follow symlinks capability turned on in the web server).

[ericmandel]

I see there is confusion here because when you upload a local file, there is no server-based retrieval to fail when the path is outside the web server directory domain. Hmmm ... I probably need to document this mich more carefully and clearly than is the case ... no doubt I got confused myself!

[ericmandel]

(The complication is absolute paths do work with the file:// URI, i.e., with desktop JS9. So I have to look at all of that carefully and try to explain it.)

[ericmandel]

As it turns out, the serverside.html documentation does say to use relative paths for the tmp directory. I added some more explanation:

I'll be in limited email contact until late Sunday.

[github-actions[bot]]

This issue is stale because it has been open for 365 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.