cli search tool behaves differently in 1.5.1 vs. 1.4.4

[bbimber]

Hello,

We're experimenting with upgrading to 1.5.1, including the cli tool. When we run what I think is a fairly vanilla command using either version 1.4.4 or 1.5.1:

cli-win.exe text-index --force --quiet --attributes Name,ID,gene_id,gene_name --file C:\Projects\discvr-lk\build\deploy\files\JBrowseVerifyProject\@files\.jbrowse\resources\4.codingRegions\4.codingRegionscodingRegions.gff.gz

When running using 1.5.1, it makes what seems like a valid .ix file, but does not make an iix or metadata json file. Should it?

Here is my input:

4.codingRegionscodingRegions.gff.gz

[cmdcolin]

if you are seeing it not make an ixx or meta.json that is a problem, and if you are not e.g. getting a "Finished!" message then maybe it is silently crashing(?) or if it is printing Finished and still not outputting the ixx and meta.json then there may be something else happening, would have to review

can you ensure that you are using 1.5.1? if I run a similar command locally with 1.5.1 i get this output

% jbrowse --version
@jbrowse/cli/1.5.1 linux-x64 node-v15.14.0

% jbrowse text-index --force --quiet --attributes Name,ID,gene_id,gene_name --file 4.codingRegionscodingRegions.gff.gz

Successfully created index for these files. See https://jbrowse.org/storybook/lgv/main/?path=/story/text-searching--page for info about usage
Finished!
% ls -1 trix
4.codingRegionscodingRegions.gff.gz.ix
4.codingRegionscodingRegions.gff.gz.ixx
4.codingRegionscodingRegions.gff.gz_meta.json

if there is still an issue, i might need to try to test on windows or know how you created cli-win.exe. i am not on windows but you could also try using npm install -g @jbrowse/cli and running the command like i have above jbrowse text-index --force --quiet --attributes Name,ID,gene_id,gene_name --file 4.codingRegionscodingRegions.gff.gz

[bbimber]

Thanks for the quick reply. You can see our package.json here:

https://github.com/BimberLab/DiscvrLabKeyModules/blob/21.7_fb_sampleFilter2/jbrowse/package.json

and the way it creates the package executable is with 'pkg'. See the jb-pkg command above, or:

pkg ./node_modules/@jbrowse/cli --out-path ./resources/external/jb-cli

This is built when our app is generated, and is deployed on the server. When a user creates jbrowse sessions, this executable gets executed.

I will do some more testing, but I suspect you are right about some internal crash. it has non-zero exit, fwiw.

[bbimber]

If it's helpful, here is the full output:

bimber@RNB08380 MINGW64 ~
$ export DEBUG=*

bimber@RNB08380 MINGW64 ~
$ /c/Projects/discvr-lk/server/modules/DiscvrLabKeyModules/jbrowse/resources/external/jb-cli/cli-win.exe text-index --force --quiet --attributes Name,ID,gene_id,gene_name --file /c/Projects/discvr-lk/build/deploy/files/JBrowseVerifyProject/@files/.jbrowse/resources/9.codingRegions/9.codingRegionscodingRegions.gff.gz
2021-12-09T05:18:40.012Z @oclif/config reading core plugin C:\snapshot\node_modules\@jbrowse\cli
2021-12-09T05:18:40.012Z @oclif/config loadJSON C:\snapshot\node_modules\@jbrowse\cli\package.json
2021-12-09T05:18:40.014Z @oclif/config loadJSON C:\snapshot\node_modules\@jbrowse\cli\oclif.manifest.json
2021-12-09T05:18:40.014Z @oclif/config:@jbrowse/cli using manifest from C:\snapshot\node_modules\@jbrowse\cli\oclif.manifest.json
2021-12-09T05:18:40.015Z @oclif/config reading user plugins pjson C:\Users\bimber\AppData\Local\@jbrowse\cli\package.json
2021-12-09T05:18:40.015Z @oclif/config loadJSON C:\Users\bimber\AppData\Local\@jbrowse\cli\package.json
2021-12-09T05:18:40.015Z @oclif/config loading plugins [ '@oclif/plugin-help' ]
2021-12-09T05:18:40.017Z @oclif/config reading core plugin C:\snapshot\node_modules\@oclif\plugin-help
2021-12-09T05:18:40.017Z @oclif/config loadJSON C:\snapshot\node_modules\@oclif\plugin-help\package.json
2021-12-09T05:18:40.017Z @oclif/config loadJSON C:\snapshot\node_modules\@oclif\plugin-help\oclif.manifest.json
2021-12-09T05:18:40.017Z @oclif/config:@oclif/plugin-help using manifest from C:\snapshot\node_modules\@oclif\plugin-help\oclif.manifest.json
2021-12-09T05:18:40.017Z @oclif/config config done
2021-12-09T05:18:40.018Z @oclif/config start init hook
2021-12-09T05:18:40.018Z @oclif/config init hook done
2021-12-09T05:18:40.018Z jbrowse init version: @oclif/command@1.8.9 argv: [ 'text-index', '--force', '--quiet', '--attributes', 'Name,ID,gene_id,gene_name', '--file', 'C:/Projects/discvr-lk/build/deploy/files/JBrowseVerifyProject/@files/.jbrowse/resources/9.codingRegions/9.codingRegionscodingRegions.gff.gz' ]
2021-12-09T05:18:40.019Z @oclif/config runCommand text-index [ '--force', '--quiet', '--attributes', 'Name,ID,gene_id,gene_name', '--file', 'C:/Projects/discvr-lk/build/deploy/files/JBrowseVerifyProject/@files/.jbrowse/resources/9.codingRegions/9.codingRegionscodingRegions.gff.gz' ]
2021-12-09T05:18:40.106Z @oclif/config:@jbrowse/cli require C:\snapshot\node_modules\@jbrowse\cli\lib\commands\text-index.js
2021-12-09T05:18:40.146Z @oclif/config start prerun hook
2021-12-09T05:18:40.146Z @oclif/config prerun hook done

and then it ends w/ exit code 0, but without creating the ixx or metadata files.

[bbimber]

@cmdcolin I looked at dependencies, and I forced it to install with ixiix 1.0.15, rather than the latest 1.0.18. This fixed my issue.

[cmdcolin]

hm, that is weird. there are bugfixes in 1.0.18 compared to 1.0.15 that address issues that you have raised, so staying on 1.0.15 may not be ideal https://github.com/GMOD/ixixx-js/blob/master/CHANGELOG.md

are you sure you are using v1.5.1? the code here suggests it is using v1.4.4 https://github.com/BimberLab/DiscvrLabKeyModules/blob/21.7_fb_sampleFilter2/jbrowse/package.json#L17

note: I tried testing using @jbrowse/cli@v1.5.1 with plain npm install -g @jbrowse/cli on a windows vm with node 12 and it worked for me (outputted ix,ixx,meta.json)

[bbimber]

OK, here is what I can see:

Using @jbrowse/cli": "~1.4.4", anything above "ixixx": "1.0.16" silently dies w/o creating the ixx/meta files.

Using "@jbrowse/cli": "~1.5.1", it forces ixiix > 1.0.17, so 1.0.18 is installed, and it fails as described above

Using "@jbrowse/cli": "~1.5.0", it forces ixiix > 1.0.15, so i can force it to use 1.0.16, and it works. If I update ixiix to 1.017, back to failure

[cmdcolin]

if i am to understand, the diff between 1.0.16 works and 1.0.17 fails then

what node version are you using?

[cmdcolin]

this is a diff just for reference https://github.com/GMOD/ixixx-js/compare/v1.0.16..v1.0.17

[cmdcolin]

I made a new update to ixixx in 1.0.19 that could possibly help with this issue

It includes a change from the author of a package called external-sorting that fixes a possible error that was causing some rare crashes seen in our CI test suite

If you are interested, can try reinstalling the jbrowse cli tools (may automatically get ixixx v1.0.19 due to semver) and try out

[bbimber]

Thanks - I just tried this. Unfortunately the local behavior is the same with 1.5.1 and ixixx 1.0.19.

[cmdcolin]

Darn. Have you tried testing it without pkg by chance e.g. just installing @jbrowse/cli with npm?

[bbimber]

apologies - your reply made me realize my quick test (which ran my build w/o pkg) wasnt complete. this might actually be working.

[cmdcolin]

awesome :) thanks again for reporting. if anything else comes up let us know

[cmdcolin]

maybe can optimistically close for now:)