Jsvis prototype

[francoismg]

Prototype for js visualization

Still to do :

• [x] Add error bars processing for D3
• [x] More robust error bar processing for Bokeh
• [ ] Code from data processor and wrappers needs to be improved
• [ ] Some inconsistencies to fix and inheritance to lessen code duplication
• [x] Some clean up
• [x] make an npm package
• [x] custom columns on the error bar
• [x] delete column cross does not work
• [x] make bokeh a dependency
• [x] add arbitrary digits processing to arithmetic columns

@volodymyrss @dsavchenko

[github-actions[bot]]

PR Preview Action v1.4.8 :---: :rocket: Deployed preview to https://esg-epfl-apc.github.io/astrojsvis/pr-preview/pr-12/ on branch gh-pages at 2024-10-02 07:51 UTC

[volodymyrss]

Looks interesting, shows in the preview! How do I load file in there?

[francoismg]

Looks interesting, shows in the preview! How do I load file in there?

Right now it works like a galaxy plugin so file cannot be changed in the interface, if you want to try another file you have to add it to "test_files" and point to it in main.js

I could add a new box that is populated with a list of accessible files and let the user choose between them if you want.

It wasn't clear if the use case for a standalone version was a file or list of files or a specific endpoint is set in the code like with other js library and you instantiate the interface for a specific file or if it was up to the user to provide the files himself. Since there are probably way better tool to do that locally with local files I thought the main use would be that files are remote

[volodymyrss]

Right now it works like a galaxy plugin so file cannot be changed in the interface, if you want to try another file you have to add it to "test_files" and point to it in main.js

It makes sense if it works like AladinLite, with an option to load local file or by URL.

I could add a new box that is populated with a list of accessible files and let the user choose between them if you want.

This would be useful. Else the demonstration does not actually demonstrate.

It wasn't clear if the use case for a standalone version was a file or list of files or a specific endpoint is set in the code like with other js library and you instantiate the interface for a specific file or if it was up to the user to provide the files himself.

We wanted it to be like AladinLite in the file load process. Presumably it's not actually difficult to select the source from upload and URL, right?

Since there are probably way better tool to do that locally with local files I thought the main use would be that files are remote

There is really not actually. UIs largely moved to the Web/Browser experience, but this sort of tool, similar to what Andrii was mentioning (called FV), was not re-implemented for Web. This is why we kind of have to do it from scratch. But I think it starts to look very interesting!

[francoismg]

Right now it works like a galaxy plugin so file cannot be changed in the interface, if you want to try another file you have to add it to "test_files" and point to it in main.js

It makes sense if it works like AladinLite, with an option to load local file or by URL.

I could add a new box that is populated with a list of accessible files and let the user choose between them if you want.

This would be useful. Else the demonstration does not actually demonstrate.

It wasn't clear if the use case for a standalone version was a file or list of files or a specific endpoint is set in the code like with other js library and you instantiate the interface for a specific file or if it was up to the user to provide the files himself.

We wanted it to be like AladinLite in the file load process. Presumably it's not actually difficult to select the source from upload and URL, right?

Since there are probably way better tool to do that locally with local files I thought the main use would be that files are remote

There is really not actually. UIs largely moved to the Web/Browser experience, but this sort of tool, similar to what Andrii was mentioning (called FV), was not re-implemented for Web. This is why we kind of have to do it from scratch. But I think it starts to look very interesting!

Ok no problem I will add a new box with an input that lets the user specify a file (url is the easiest right now but needs to come from a cors compliant server (or we can use an open cors proxy server), for local files uploaded directly I will just have to check fits-reader cause there was some specific things when dealing with local files) and once it's loaded it will be added to a list of available files that the user can select

[volodymyrss]

Would be really cool if there was something for the demo today!

[francoismg]

Would be really cool if there was something for the demo today!

I just finished error bars for d3 so not sure I will have the time but will try

[francoismg]

For reference :

• add the possibility for user to download and change current file (investigate possible way to do cross origin download)
• synchronize hdu choice between header and data components
• add extname to hdu name
• add arithmetic and mathematical operation for column choice

[francoismg]

• Bugs with bokeh graphs have been fixed and error bars added to D3

• I was able to plot the file from mmoda without changes to the code, only problem is the fact that timedel is from the header and changes regarding how this is handled are ongoing

• For url file upload I looked at bypassing cors policies but I can't find a way to do that in client side js only without relying on some server config or code, using a proxy server or local proxy, or routing the requests through some backend. If you still have the code from that data challenge you talked about I could have a look at it maybe I have missed something

Regarding file upload, changes to the code and interface are still in progress and it's not testable right now but an usable minimal version should be ready during the week

Local file upload works with fits reader you just have to do things a bit differently that you would normally do but it works fine. However there were other things to take into consideration since users will be able to upload file directly, that csv support should be added, plus arithmetic column and multiple datasets support.

Ongoing development :

• User will have to specify a file type before upload (fits and csv right now)
• User will be able to upload any number of local files that will be displayed in an available files list
• Files in the available list can be selected and parametrized to some extent (custom column names, header cards value change.....) + some general information about the files are displayed
• User can move files from the available list to a "current" files list that will be used for generating visualization
• User can remove files from the current list when necessary
• Settings let the user choose between any column of any hdu or files that are in the current files list (every column is prefixed by a file id (and an hdu index when applicable)). That will let the user be able to plot things using data from any number of files together and do column arithmetic with columns from any files
• Multiple datasets can be plotted on the same graph
• Header and data component will still be synchronized together but will be able to show data from any files in the current list

[dsavchenko]

@francoismg I made a small change so that the file that's in the repo is accessible in preview But now there are two files in "available" and only the second one is really working. But I hope you can debug this easily

[francoismg]

@francoismg I made a small change so that the file that's in the repo is accessible in preview But now there are two files in "available" and only the second one is really working. But I hope you can debug this easily

yes I have the same thing locally it was just some test I made, I will remove that so it's not confusing

[volodymyrss]

check how it's done here https://github.com/oda-hub/dispatcher-app/blob/master/cdci_data_analysis/analysis/plot_tools.py

add range-zoom UI

[volodymyrss]

How is it going, @francoismg ?

[francoismg]

How is it going, @francoismg ?

Custom range is working for D3 still a few bugs but seems to work, I will push it so you can test that, just need to remove things that are not working. If you want to test it you need to select D3, x range is working fine but y range needs x range to be set to work (one of the bugs I'm working on).

Bokeh part should be good soon and hopefully the plugin part and the other unrelated bugs I have spotted could be ready for friday

[francoismg]

Custom range for bokeh has been added + most of the bugs have been fixed (still an issue with tick values on log scale with bokeh). Let me know if it looks ok to you.

I will try to fix the bokeh tick problem and clean everything for the plugin

[volodymyrss]

Ok thanks, but in the preview it does not really work, I do not see the to select values to plot, in the console it is HDUNotTabularError: Selected HDU is not tabular.

Did you check that the right version is pushed?

[volodymyrss]

this one is not working

ibis.zip

[francoismg]

this one is not working

ibis.zip

looks like one of the header from the arf file is corrupted or at least it doesn't respect fits standard for header block size, same problem when trying to use the file with astropy

the two other files are working fine, were you able to use the arf file for other things?

[volodymyrss]

this one is not working ibis.zip

looks like one of the header from the arf file is corrupted or at least it doesn't respect fits standard for header block size, same problem when trying to use the file with astropy

Apparently it is an archive. I removed it, I will add another one.

the two other files are working fine, were you able to use the arf file for other things?

The two others are still not plotting for me, like I showed. Do they plot for you?

[francoismg]

The two others are still not plotting for me, like I showed. Do they plot for you?

It should be good now I just pushed a small workaround so it will not try to fetch the arf file, but you have to upload the matrix file before adding the main file to the "current files list" just in case

[volodymyrss]

The two others are still not plotting for me, like I showed. Do they plot for you?

It should be good now I just pushed a small workaround so it will not try to fetch the arf file, but you have to upload the matrix file before adding the main file to the "current files list" just in case

It works, thanks! The order requirements is a bit inconvenient.

One thing missing is some sort of "D_E", I thought it was there before?

[francoismg]

The two others are still not plotting for me, like I showed. Do they plot for you?

It should be good now I just pushed a small workaround so it will not try to fetch the arf file, but you have to upload the matrix file before adding the main file to the "current files list" just in case

It works, thanks! The order requirements is a bit inconvenient.

One thing missing is some sort of "D_E", I thought it was there before?

Ok cool no problem, regarding the order thing I will check if I can make it more convenient like maybe reprocess files when a new one is loaded so it doesn't care about the order.

For D_E I named it half width but I guess it's not the right name then, do you want me to just call the column D_E ?

[francoismg]

Galaxy plugin is working just 2 issues with the build process to fix

Display bugs in galaxy are fixed

Clean up mostly ok but had to remove some refactors that were not working will add them back after

One drawing out of bounds bug with error bars but I think I have found the problem so should be easy to fix

Add option to link files in file settings + possibility to choose only one error bar

[francoismg]

@volodymyrss @dsavchenko

Had issues fixing build problems with parcel so I turned everything into a module and remade the build process with webpack and it seems ok now. There might be an issue with the automatic rebuild on changes but I think that might be a local issue so would be nice to see if it works on the test galaxy instance

Plugin has been pushed to this branch https://github.com/esg-epfl-apc/galaxy/tree/fits-graph-visualization-plugin

Galaxy gulp file outside plugins directory has been modified too so the plugin is added to the gulp process

Dependencies should be loaded automatically when you trigger a client rebuild but you can run yarn or npm install if there's some issues

If plugin is not rebuilding after first build you can delete the plugin build hash file from plugin static so the plugin will be rebuild

[dsavchenko]

Trying to upgrade the available galaxy instance to this branch, there is an issue in a database migration (unrelated to the visualisation plugin). I will explore further

[dsavchenko]

Seems I managed to fix the update process. Our deployment is now 24.0 branch @francoismg could you rebase your branch on top of it? I would prefer to have a stable release synchronised with usegalaxy.eu in the preview instance, not a development branch.

[francoismg]

Seems I managed to fix the update process. Our deployment is now 24.0 branch @francoismg could you rebase your branch on top of it? I would prefer to have a stable release synchronised with usegalaxy.eu in the preview instance, not a development branch.

ok thanks i will try to rebase it no problem

[dsavchenko]

I tried to update galaxy.odahub.fr to the fits-graph-visualization-plugin, but the plugin is not among the visualisations.

@francoismg do you have a suggestion, why could it be?

[francoismg]

I tried to update galaxy.odahub.fr to the fits-graph-visualization-plugin, but the plugin is not among the visualisations.

@francoismg do you have a suggestion, why could it be?

Is the plugin in the build list when the client is built and is there some errors or doest it just say no changes detected for plugin graph viewer? Or maybe the client build is skipped completely?

I will check if there's some 'registration' somewhere I forgot to add when moving to that branch

[francoismg]

I got the same issue when rebuilding everything so the problem is not on your side @dsavchenko

I will check and get back to you when I found the issue

[francoismg]

Problem seems to be that the plugin is not part of the build process

[francoismg]

@dsavchenko I think I found the problem, somehow the plugin files were completely messed up there was some config files missing and gulp file not updated (hence why the plugin was not in the build process) and there was no src to build too.

I was able to rebuild the thing and that time the client build was working but now the whole instance startup is failing (database version problem, maybe same 'migration' issues you had)

Maybe that particular dev branch state is not functional or something.

I tried to rebase it the other day but there were too many conflicts so I created a new branch based on stable release, I will move plugin files to that branch now and hopefully it will fix the remaining issues

[francoismg]

@dsavchenko ok it seems to be good now, will push to the new branch and will put the link to new stable branch here

[francoismg]

@dsavchenko you can try with this branch https://github.com/esg-epfl-apc/galaxy/tree/galaxy-stable-24.0 should be working now

[dsavchenko]

@dsavchenko you can try with this branch https://github.com/esg-epfl-apc/galaxy/tree/galaxy-stable-24.0 should be working now

Still not working. What should I check in the installation to debug the build problem? Maybe I need to cleanup some leftovers to ensure the plugin rebuild? (I'm updating in-place using ansible following https://training.galaxyproject.org/training-material/topics/admin/tutorials/ansible-galaxy/tutorial.html)

[francoismg]

@dsavchenko you can try with this branch https://github.com/esg-epfl-apc/galaxy/tree/galaxy-stable-24.0 should be working now

Still not working. What should I check in the installation to debug the build problem? Maybe I need to cleanup some leftovers to ensure the plugin rebuild? (I'm updating in-place using ansible following https://training.galaxyproject.org/training-material/topics/admin/tutorials/ansible-galaxy/tutorial.html)

I'm not sure if using ansible changes things regarding that issue, when having a real instance is it still launched using run.sh or is using another way?

if it's using run.sh you can check during startup that "fits-graph-viewer" is listed when the script is building plugins -> if it's not, check that the plugin name is listed in gulpfile.js and that there's a 'plugin-name'.xml file in the plugin config folder

if there are some webpack or other build errors there should be listed here

If the plugin is listed with "no changes detected" you can remove the hash-build.txt files and it should trigger a rebuild at startup

Also make sure that there's no "no-client-rebuild" flag (not sure about the exact name) or things like that that would keep the plugins from being built

If these things are ok and there are no error during plugin build the plugin should be available in the viualization list in the ui

if it's listed in the visualization list but not working you can check that the plugin static folder is there and populated and also you can check that the main static folder at the root /static/visualization/plugins/plugin_name is populated too

[dsavchenko]

when having a real instance is it still launched using run.sh or is using another way?

The "real" instance is deployed with ansible. Locally, when running with run.sh everything works, the plugin is available. In staging installation with ansible, it doesn't work, I'm still exploring why, it looks like the needed files are in place

[francoismg]

when having a real instance is it still launched using run.sh or is using another way?

The "real" instance is deployed with ansible. Locally, when running with run.sh everything works, the plugin is available. In staging installation with ansible, it doesn't work, I'm still exploring why, it looks like the needed files are in place

Did you install the aladin lite viewer in this instance? The main structure (config + mako) is more or less the same so if this plugin is working it might give some insights

[dsavchenko]

Well, after changing the branch back-and-forth, the plugin finally appears in https://galaxy.odahub.fr Not sure, what was the problem...

@volodymyrss @andriineronov you can now test it as galaxy plugin and report issues

[volodymyrss]

Well, after changing the branch back-and-forth, the plugin finally appears in https://galaxy.odahub.fr Not sure, what was the problem...

@volodymyrss @andriineronov you can now test it as galaxy plugin and report issues

Are we nearly happy with the implementation? Is further work needed on @francoismg side give what we already discussed?

[francoismg]

Well, after changing the branch back-and-forth, the plugin finally appears in https://galaxy.odahub.fr Not sure, what was the problem... @volodymyrss @andriineronov you can now test it as galaxy plugin and report issues

Are we nearly happy with the implementation? Is further work needed on @francoismg side give what we already discussed?

on my side it's now possible to have only one error bar and it works with bokeh and d3 like denys requested, I have also added the possibility to choose product type and link arf and rmf files directly in the file 'settings' but there's still a few bugs, also have been working on arithmetic columns creation and multiple plot by graph

[volodymyrss]

Well, after changing the branch back-and-forth, the plugin finally appears in https://galaxy.odahub.fr Not sure, what was the problem... @volodymyrss @andriineronov you can now test it as galaxy plugin and report issues

Are we nearly happy with the implementation? Is further work needed on @francoismg side give what we already discussed?

on my side it's now possible to have only one error bar and it works with bokeh and d3 like denys requested, I have also added the possibility to choose product type and link arf and rmf files directly in the file 'settings' but there's still a few bugs, also have been working on arithmetic columns creation and multiple plot by graph

For me it looked quite reasonable and almost ready to propose to galaxy upstream. Maybe we could do that after @andriineronov provides some additional review, if @dsavchenko can arrange with @andriineronov (while I am still away)

[dsavchenko]

I don't see recent updates, neither here nor in galaxy branch. @francoismg did you push them?

Also, it seems that in galaxy, the plugin code contains just a copy of the library files. Because we wanted it to potentially be also a standalone visualisation lib, it makes sense to keep it here and use as npm package when building galaxy visualisation plugin. Also, note that the latest galaxy release is now 24.1.1, so let's probably use it as a base branch for preview in our staging server. When ready, we may rebase it to dev (and then also backport to release(s))

[francoismg]

I don't see recent updates, neither here nor in galaxy branch. @francoismg did you push them?

Also, it seems that in galaxy, the plugin code contains just a copy of the library files. Because we wanted it to potentially be also a standalone visualisation lib, it makes sense to keep it here and use as npm package when building galaxy visualisation plugin. Also, note that the latest galaxy release is now 24.1, so let's probably use it as a base branch for preview in our staging server. When ready, we may rebase it to dev (and then also backport to release(s))

I will push what's working just need to cherrypick what to commit so it doesn't break the rest

Regarding the lib I will make an npm package like for aladin lite and update the build script. Maybe it makes sense to have a github action for that, I will check it too

[volodymyrss]

I don't see recent updates, neither here nor in galaxy branch. @francoismg did you push them? Also, it seems that in galaxy, the plugin code contains just a copy of the library files. Because we wanted it to potentially be also a standalone visualisation lib, it makes sense to keep it here and use as npm package when building galaxy visualisation plugin. Also, note that the latest galaxy release is now 24.1, so let's probably use it as a base branch for preview in our staging server. When ready, we may rebase it to dev (and then also backport to release(s))

I will push what's working just need to cherrypick what to commit so it doesn't break the rest

Please do.

Regarding the lib I will make an npm package like for aladin lite and update the build script. Maybe it makes sense to have a github action for that, I will check it too

[volodymyrss]

@francoismg could you please cross-check that the TODO points in the description are still open?

[francoismg]

@francoismg could you please cross-check that the TODO points in the description are still open?

edit by @volodymyrss : I will remove the checkboxes from here to avoid confusion.

[volodymyrss]

@francoismg could you please cross-check that the TODO points in the description are still open?

* [x]  Add error bars processing for D3

* [x]  More robust error bar processing for Bokeh

* [ ]  Code from data processor and wrappers needs to be improved (some enhancements but still work to do)

* [ ]  Some inconsistencies to fix and inheritance to lessen code duplication (some enhancements but still word to do)

* [x]  Some clean up

* [ ]  make an npm package (refactoring in progress as discussed)

I click the checkboxes right in the issue description. It does not work for you like that?

[francoismg]

@francoismg could you please cross-check that the TODO points in the description are still open?

* [x]  Add error bars processing for D3

* [x]  More robust error bar processing for Bokeh

* [ ]  Code from data processor and wrappers needs to be improved (some enhancements but still work to do)

* [ ]  Some inconsistencies to fix and inheritance to lessen code duplication (some enhancements but still word to do)

* [x]  Some clean up

* [ ]  make an npm package (refactoring in progress as discussed)

I click the checkboxes right in the issue description. It does not work for you like that?

Yes I can do it too, will update that And no problem will re request when it's complete (still one issue with component instantiation and a bug with column creation since the refactoring for npm packaging)

[volodymyrss]

We should have a look if this is useful, @dsavchenko @francoismg , suggested by @bgruening