missing HTML thumbnails

[o-smirnov]

@Athanaseus please check if you still have the problem with rc12 images.

settings.html should show puppeteer by default. Also, check radiopadre.display_log(True) for more info.

[Athanaseus]

$ pip list
Package           Version Location
----------------- ------- --------------------------------------------------------
pip               20.1
psutil            5.7.0
radiopadre-client 1.0rc12 /home/athanaseus/Documents/gitPackages/radiopadre-client
setuptools        46.0.0
six               1.14.0
wheel             0.34.2

$ run-radiopadre ramaila@stills.ru.ac.za:/home/ramaila/working/PicA/output/ --auto-init --singularity
radiopadre.client [0.02s]: Welcome to the radiopadre client!
radiopadre.client [0.03s]:   command line specifies AUTO_INIT = True
radiopadre.client [0.03s]:   command line specifies BACKEND = singularity
radiopadre.client: Opening ssh connection to ramaila@stills.ru.ac.za. You may be prompted for your password.
radiopadre.client: No remote virtualenv found at ramaila@stills.ru.ac.za:/home/athanaseus/.radiopadre/venv
radiopadre.client: No remote client script run-radiopadre found
radiopadre.client: No run-radiopadre script found on ramaila@stills.ru.ac.za
radiopadre.client: Trying to --auto-init an installation for you...
radiopadre.client: Creating virtualenv ramaila@stills.ru.ac.za:/home/athanaseus/.radiopadre/venv
Traceback (most recent call last):
  File "/usr/bin/virtualenv", line 11, in <module>
    load_entry_point('virtualenv==15.1.0', 'console_scripts', 'virtualenv')()
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 724, in main
    symlink=options.symlink)
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 946, in create_environment
    site_packages=site_packages, clear=clear, symlink=symlink))
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 1157, in install_python
    mkdir(lib_dir)
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 329, in mkdir
    os.makedirs(path)
  File "/usr/lib/python3.6/os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  File "/usr/lib/python3.6/os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  File "/usr/lib/python3.6/os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  [Previous line repeated 1 more time]
  File "/usr/lib/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/athanaseus'
radiopadre.client: ssh virtualenv -p python3 /home/athanaseus/.radiopadre/venv failed with exit code 1
Traceback (most recent call last):
  File "/home/athanaseus/.virtualenvs/radiopadre-client3/bin/run-radiopadre", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/athanaseus/Documents/gitPackages/radiopadre-client/bin/run-radiopadre", line 311, in <module>
    retcode = radiopadre_client.remote.run_remote_session(command, copy_initial_notebook, notebook_path, arguments)
  File "/home/athanaseus/Documents/gitPackages/radiopadre-client/radiopadre_client/remote.py", line 221, in run_remote_session
    ssh_remote_v(ff("virtualenv -p python3 {config.RADIOPADRE_VENV}"))
  File "/home/athanaseus/Documents/gitPackages/radiopadre-client/radiopadre_client/remote.py", line 56, in ssh_remote_v
    return ssh_remote(command, fail_retcode, stderr=sys.stderr)
  File "/home/athanaseus/Documents/gitPackages/radiopadre-client/radiopadre_client/remote.py", line 48, in ssh_remote
    return subprocess.check_output(SSH_OPTS + [command], stderr=stderr).decode('utf-8')
  File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ssh', '-o', 'ControlPath=/tmp/ssh_mux_radiopadre_%C', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=1h', 'ramaila@stills.ru.ac.za', 'virtualenv -p python3 /home/athanaseus/.radiopadre/venv']' returned non-zero exit status 1.

I removed my old .radiopadre on the remote machine hoping that the above will initiate a new one. Now I'm getting some permission error. Did I miss something maybe?

[o-smirnov]

Ah! I know exactly what's wrong. Gimme a couple of hours...

Problem is, it does os.expanduser("~/.radiopadre/venv") too early in the game. It needs to be passed to the remote unexpanded, because the username is different.

[o-smirnov]

See https://github.com/ratt-ru/radiopadre-client/issues/29. The fix is on the b1.0-pre12 branch, could you try with that? The change should only affect your local runscript.

[Athanaseus]

ok this kinda does it, but i have an outcast (*B0.html), both in the std and detail. Might be my internet, but i ran it a couple of times and still gives the same results. The file itself does open though.

[o-smirnov]

Hmmm, B0 comes before B1... Were you running a previous version of radiopadre on the same output dir per chance? Maybe it has a stale thumbnail cached from that time. Before you do anything else, could you please check the timestamp of the thumbnail image to see if this the case?

If that's the case, then just remove the thumbnail image and re-render.

[Athanaseus]

http://localhost:13003/25e8eebee3264abebb9fcb1555758d93/home/ramaila/working/PicA/output/diagnostic_plots/crosscal/.radiopadre/html-render/.radiopadre/thumbs/474.picaq-PicA-1gc1_primary.B0.width-1920.height-1024.png?mtime=1590068149.3755832

http://localhost:13003/25e8eebee3264abebb9fcb1555758d93/home/ramaila/working/PicA/output/diagnostic_plots/crosscal/.radiopadre/html-render/.radiopadre/thumbs/474.picaq-PicA-1gc1_primary.B1.width-1920.height-1024.png?mtime=1590068136.1314557

/home/ramaila/working/PicA/output/diagnostic_plots/crosscal/.radiopadre/html-render

-rw-r--r-- 1 ramaila ramaila     16 May 21 15:35 picaq-PicA-1gc1_secondary.G1.width-1920.height-1024.png.md5
-rw-r--r-- 1 ramaila ramaila 148843 May 21 15:35 picaq-PicA-1gc1_secondary.G1.width-1920.height-1024.png
-rw-r--r-- 1 ramaila ramaila     16 May 21 15:35 picaq-PicA-1gc1_secondary.K1.width-1920.height-1024.png.md5
-rw-r--r-- 1 ramaila ramaila 110973 May 21 15:35 picaq-PicA-1gc1_secondary.K1.width-1920.height-1024.png
-rw-r--r-- 1 ramaila ramaila     16 May 21 15:35 picaq-PicA-1gc1_secondary.G0.width-1920.height-1024.png.md5
-rw-r--r-- 1 ramaila ramaila 156788 May 21 15:35 picaq-PicA-1gc1_secondary.G0.width-1920.height-1024.png
-rw-r--r-- 1 ramaila ramaila     16 May 21 15:35 picaq-PicA-1gc1_secondary.K0.width-1920.height-1024.png.md5
-rw-r--r-- 1 ramaila ramaila 111759 May 21 15:35 picaq-PicA-1gc1_secondary.K0.width-1920.height-1024.png
-rw-r--r-- 1 ramaila ramaila     16 May 21 15:35 picaq-PicA-1gc1_primary.B1.width-1920.height-1024.png.md5
-rw-r--r-- 1 ramaila ramaila 417754 May 21 15:35 picaq-PicA-1gc1_primary.B1.width-1920.height-1024.png
-rw-r--r-- 1 ramaila ramaila     16 May 21 15:35 picaq-PicA-1gc1_primary.G1.width-1920.height-1024.png.md5
-rw-r--r-- 1 ramaila ramaila 133281 May 21 15:35 picaq-PicA-1gc1_primary.G1.width-1920.height-1024.png
-rw-r--r-- 1 ramaila ramaila     16 May 21 15:35 picaq-PicA-1gc1_primary.K1.width-1920.height-1024.png.md5
-rw-r--r-- 1 ramaila ramaila 115586 May 21 15:35 picaq-PicA-1gc1_primary.K1.width-1920.height-1024.png
-rw-r--r-- 1 ramaila ramaila     16 May 21 15:35 picaq-PicA-1gc1_primary.B0.width-1920.height-1024.png.md5
-rw-r--r-- 1 ramaila ramaila  10053 May 21 15:35 picaq-PicA-1gc1_primary.B0.width-1920.height-1024.png
-rw-r--r-- 1 ramaila ramaila     16 May 21 15:35 picaq-PicA-1gc1_primary.G0.width-1920.height-1024.png.md5
-rw-r--r-- 1 ramaila ramaila  97725 May 21 15:35 picaq-PicA-1gc1_primary.G0.width-1920.height-1024.png
-rw-r--r-- 1 ramaila ramaila     16 May 21 15:35 picaq-PicA-1gc1_primary.K0.width-1920.height-1024.png.md5
-rw-r--r-- 1 ramaila ramaila  81541 May 21 15:35 picaq-PicA-1gc1_primary.K0.width-1920.height-1024.png

/home/ramaila/working/PicA/output/diagnostic_plots/crosscal/.radiopadre/html-render/.radiopadre/thumbs

-rw-r--r-- 1 ramaila ramaila 54429 May 21 15:35 474.picaq-PicA-1gc1_secondary.G1.width-1920.height-1024.png
-rw-r--r-- 1 ramaila ramaila 34887 May 21 15:35 474.picaq-PicA-1gc1_secondary.K1.width-1920.height-1024.png
-rw-r--r-- 1 ramaila ramaila 54023 May 21 15:35 474.picaq-PicA-1gc1_secondary.G0.width-1920.height-1024.png
-rw-r--r-- 1 ramaila ramaila 32636 May 21 15:35 474.picaq-PicA-1gc1_secondary.K0.width-1920.height-1024.png
-rw-r--r-- 1 ramaila ramaila 75123 May 21 15:35 474.picaq-PicA-1gc1_primary.B1.width-1920.height-1024.png
-rw-r--r-- 1 ramaila ramaila 39576 May 21 15:35 474.picaq-PicA-1gc1_primary.G1.width-1920.height-1024.png
-rw-r--r-- 1 ramaila ramaila 30441 May 21 15:35 474.picaq-PicA-1gc1_primary.K1.width-1920.height-1024.png
-rw-r--r-- 1 ramaila ramaila  1068 May 21 15:35 474.picaq-PicA-1gc1_primary.B0.width-1920.height-1024.png
-rw-r--r-- 1 ramaila ramaila 32359 May 21 15:35 474.picaq-PicA-1gc1_primary.G0.width-1920.height-1024.png
-rw-r--r-- 1 ramaila ramaila 24249 May 21 15:35 474.picaq-PicA-1gc1_primary.K0.width-1920.height-1024.png

I removed output/diagnostic_plots/crosscal/.radiopadre and re-rendered, and i still get the same.

[o-smirnov]

OK, let me check... this is on stills?

[Athanaseus]

yes, on stills.

PS: /home/ramaila/working/PicA

[o-smirnov]

@Athanaseus I really don't know what to do about your outcast. Maybe @Mulan-94 can help. Or maybe some bright young person can google some fancy Javascript knowledge for me, because it is beyond my ken.

The basic problem is this. Your B0 HTML plot is huge, 257MB in size (while B1 is smaller, only 50-odd). Why it's so big is another question for @Mulan-94, but that's beside the point.

If I just load that file into Chrome, it takes about ~20 seconds of looking at a blank page, before any plots show up. The problem is that inside radiopadre, there's a bit of Javascript that drives puppeteer, doing something like this:

  const page = await browser.newPage();
  page.setViewport({width: width, height:height})
  await page.goto(url);
  await page.waitFor(delay);
  await page.screenshot({path: output});

The default delay is 0.2s, which is fine for pretty much any HTML. But these Boken plots are different. They barely have any static HTML in them -- it's all a bunch of Javascript code which spools up and renders HTML elements dynamically. And in case of the B0 plot, this JS becomes very slow, for reasons unknown.

Problem is, on the outside, there's no robust way I can find for the thumbnail script to know that the JS code is done rendering. It runs in the background and doesn't tell me when it's done. There are some proposals on https://stackoverflow.com/questions/52497252/puppeteer-wait-until-page-is-completely-loaded, but none of them seem to apply with this plot. Even putting in a 60s timeout doesn't help, strangely... and in any case, putting in a long fixed delay is not an acceptable solution, as it would delay all HTML thumbnails.

So I'm going to write this one off as "sorry, we just can't reliably do thumbnails of pathologically large HTML files".

In the future, we could maybe think of a bespoke solution. If @Mulan-94 could figure out how to have Boken emit some kind of signal when the plot is done rendering, then we could have the radiopadre thumbnail rendering script wait for that signal. Of course, it would then also need to know for sure that a ragavi plot is being rendered. Something along those lines could be feasible.

[o-smirnov]

Thanks for the update though, it did lead to a bugfix in phantomjs mode...

[Mulan-94]

@o-smirnov @Athanaseus The plots become slow because they are really large. And they are really large because in that case, they contain too much data (and nowadays, ragavi plots the bandpass for all available times). You are right, it will take a whole lot more than 0.2s for a plot that large to render. I have been thinking about this problem and was thinking that maybe I will have to write out some data. Will have some conclusive answer soon.