swap out mine jbrowses for embedded jbrowse2s

adf-ncgr commented 4 months ago

per discussion among group in 04-18-2024 LIS/PB mtg; partially because the post-processing needed (populate-child-features) seems to be unsustainably time-consuming for incremental updates to large mines, and partially because JBrowse2 is where we are heading more generally (and no one seems to feel like sourcing the embedded browser content from the mine itself is an important requirement)

adf-ncgr commented 3 months ago

@svengato we can use this issue to discuss further progress

svengato commented 3 months ago

I managed to replace the embedded JBrowse [1] with a JBrowse 2 instance, using TrifoliumMine as a dæmo. Some notes follow below (while still fresh in my mind).

svengato commented 3 months ago

The JBrowse 2 URL goes in webapp/src/main/web.properties, and must end with a slash. For example,

jbrowse.install.url = https://all-genera.lis.ncgr.org/tools/jbrowse2/

This properties file also includes genomicRegionSearch.jbrowse.url and genomicRegionSearch.jbrowse.install.url, but I am not sure what they do, they do not appear to affect the embedded JBrowse.

svengato commented 3 months ago

The other relevant file is not (yet) part of our LIS mines,

webapp/src/main/webapp/model/jbrowseDisplayer.jsp

I copied it from the InterMine repository, and then modified a few things to override the default behavior when you redeploy the mine. These include

    <c:set var="assembly" value="${fn:substringBefore(reportObject.object.primaryIdentifier, '.ann')}"/>

which returns the correct assembly name (like tripr.MilvusB.gnm2) with which to launch JBrowse 2, at least for Gene and similar entities, and

    <c:set var="jbLink" value="${jbrowseURL}?session=spec-%7B%22views%22%3A%5B%7B%22assembly%22%3A%22${assembly}%22%2C%22loc%22%3A%22${chr}%3A${start}-${end}%22%2C%22type%22%3A%22LinearGenomeView%22%7D%5D%7D"/>

which creates the JBrowse 2 link. There is a <c:url ...> tag which is supposed to URL-encode the special characters in the URL, but I could not get it to work, hence the explicit URL encoding for now.

There is no obvious way to determine from the InterMine data alone which track(s) the user wants, so I omitted the tracks field. I think we did it this way in the JBrowse that talks to GCV.

Finally, this sets the "more about JBrowse" link to the JBrowse 2 site,

    <p><a href="javascript:;" onclick="jQuery('iframe').css({height: '600px'});">Expand viewer</a>&nbsp;(more about <a href="https://jbrowse.org/jb2/">JBrowse</a>)</p>

To do: add jbrowseDisplayer.jsp to the mine-webapp repository.

adf-ncgr commented 3 months ago

Thanks, this seems like a good start. Figuring out the names of tracks we'd like to have automatically open is indeed a PITA given the way that things currently work (the unpredictable KEY4 in the file names is the main issue, I think). I can imagine various ways of tackling the issue, the main two contenders in my mind would be:

have our JBrowse config generator set trackIds that are not just the file names, but would use info like the full yuck prefix that could be easily obtained from the gene id in the report page context;
make tracklists properties of individual mine setups (ie just make it a mine configuration issue)

not sure which of the two would be preferable. The first is somewhat attractive in that it doesn't just solve the issue for Intermine but potentially for other contexts (e.g. the linkouts, which currently do this by having access to the datastore-metadata repo, but it's definitely non-ideal). The second has the advantage that it's very explicit and gives us full control over what we do, regardless of inevitable special cases. Maybe a good topic of discussion for tomorrow's meeting. In any case, this is good proof of concept for now.

svengato commented 3 months ago

I added a brief dæmo to the LIS meeting agenda.

svengato commented 3 months ago

I figured out the c:url tag, which makes the JSP code easier to read.

svengato commented 3 months ago

Regarding using the linkouts (that is, those containing 'jbrowse2') to determine the assembly and tracks for the embedded JBrowse 2, note that a different JSP file (lisLinkDisplayer.jsp) handles the linkouts, so we may have to duplicate some of that code in jbrowseDisplayer.jsp.

adf-ncgr commented 3 months ago

that's fine, I don't think there would be much of significance that's duplicated (basically just the service call, I guess)

svengato commented 3 months ago

Yes, and then it would handle the results a little differently.

svengato commented 3 months ago

It is straightforward to use JavaScript to obtain the lists of gene and genomic region linkouts (as in the linkout displayer JSP), find the first one that contains 'jbrowse2', and stuff that into the iframe src (and the "Centre at" link) in the JBrowse displayer JSP. I am now trying to find out whether JSP allows you to get data from a URL without bothering with JavaScript, which seems like a cleaner approach.

In our case, the linkout comes from https://other-genera.lis.ncgr.org/tools/jbrowse2. Is it important to replace this with the JBrowse2 instance we specified in a configuration file, https://all-genera.lis.ncgr.org/tools/jbrowse2?

adf-ncgr commented 3 months ago

no, you should just use whatever jbrowse instance is offered by the linkout. The "all-genera" was merely intended to simplify life for you...

svengato commented 3 months ago

Good, then there is also no need for a lot of the variables previously defined in jbrowseDisplayer.jsp (jbrowseURL, chr, offset, start, end, genus, species, taxon, assembly, tracks). Some of them are passed to a function called bookmarkCallback() which probably still uses JBrowse 1 syntax.

On closer examination, this function is within HTML comments, so we could eliminate it -- I do not know what it might have been used for.

svengato commented 3 months ago

I committed the change (new jbrowseDisplayer.jsp file) to the 5.1.0.3 branch of mine-webapp, and got this error message on git push, but it looks like the push succeeded anyway. Locally it thinks I still need to do git push. Have you seen this before? Maybe it needs a submodule update?

legumista@lupini-mines:~/trifoliummine/webapp/src/main/webapp$ git push
Username for 'https://github.com': svengato
Password for 'https://svengato@github.com': 
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 96 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 1.38 KiB | 1.38 MiB/s, done.
Total 4 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To https://github.com/legumeinfo/mine-webapp
   6e42f08..f150e3a  5.1.0.3 -> 5.1.0.3
error: update_ref failed for ref 'refs/remotes/origin/5.1.0.3': cannot update the ref 'refs/remotes/origin/5.1.0.3': unable to append to '/home/legumista/trifoliummine/.git/modules/webapp/src/main/webapp/logs/refs/remotes/origin/5.1.0.3': Permission denied

adf-ncgr commented 3 months ago

well, not sure why permissions are this way but: -rw-r--r--. 1 root root 146 May 14 10:28 /home/legumista/trifoliummine/.git/modules/webapp/src/main/webapp/logs/refs/remotes/origin/5.1.0.3 so that's why legumista can't update it

svengato commented 3 months ago

I fixed it with chown, now git push says "Everything up to date". Not sure what caused that, but I will look out for it when I update the mine-webapp submodule in ViciaMine.

svengato commented 3 months ago

Added embedded JBrowse 2 to ViciaMine with no problems.

svengato commented 3 months ago

To do: Update production mines.

Question: mines.legumeinfo.org redirects to jicama, not mines-vm.lis.ncgr.org (which looks like where the production mines live). Is this correct?

adf-ncgr commented 3 months ago

Yes, that's correct although when you say it redirects to jicama I think you just mean that the DNS resolves to jicama's IP not that an http redirect is being used, right? In any case only jicama and lupini have IP addresses accessible to the internet so requests get proxied through them.

svengato commented 3 months ago

Correct. But to update the production mines, I should be on mines-vm.

adf-ncgr commented 3 months ago

yes

svengato commented 3 months ago

Production mines: updated. (except its-probably-fine-to-delete-this-legumemine-nee-legumemine-5103)

Development mines: mostly updated, see issue #162 for the remaining holdouts.

svengato commented 3 months ago

Committed 5.1.0.3 branch, up to date in both development and production:

[x] aeschynomenemine
[x] arachismine
[x] cajanusmine
[x] cicermine
[x] glycinemine
[x] legumemine
[x] lensmine
[x] lupinusmine
[x] medicagomine
[x] minimine (not up to date on dev, which is at 5.1.0.4)
[ ] phapanmine (only exists on dev server)
[x] phaseolusmine
[x] trifoliummine (only exists on dev server)
[x] viciamine (only exists on dev server)
[x] vignamine

adf-ncgr commented 2 months ago

@svengato should we close this? we need not worry about phapanmine, that's just an experiment we did for the lima bean project but we need not maintain it.

legumeinfo / mine-issues

swap out mine jbrowses for embedded jbrowse2s #158