GMOD / Apollo

Genome annotation editor with a Java Server backend and a Javascript client that runs in a web browser as a JBrowse plugin.
http://genomearchitect.readthedocs.io/
Other
128 stars 85 forks source link

Installing a JBrowse plugin via npm / git in 2.0.7 #1752

Closed luke-c-sargent closed 7 years ago

luke-c-sargent commented 7 years ago

I am attempting to './apollo deploy' with a particular JBrowse plugin, but get this error:

94 error code EISDIR
95 error errno -21
96 error syscall read
97 error eisdir EISDIR: illegal operation on a directory, read

Having given it the ol' Google/StackOverflow treatment, I suspect the change from bower dependency resolution has brought this on; I had no issues compiling a WAR file with the plugin in 2.0.6, but npm requires a 'package.json' in the repository for 'install' to work -- which this repo lacks.

Also, the previous version did not seem to need SSH configuration to be able to clone repos, presumably using https; this seems to be a consequence of the syntax chosen, as

git clone https://github.com/Yating-L/JBrowse_plugins.git

works without SSH configuration and

git clone git@github.com:Yating-L/JBrowse_plugins.git

does not.

It seems as though there are a few options:

speaking to the 'url' parameter options, i made some minor alterations to the source to test that out (right now it just tests to see if the URL parameter is true then copies whatever was linked to a file named after the plugin name), allowing you to set a 'zip' flag like in other areas of the build script. The issue there is that github archives contain the repo contents in a folder named after the repo, whereas other zip files linked to in the 'url' field have no such guarantee, so we are left unsure as to wether or not to move files from the base directory in the archive. One option is to check if the link supplied was a github link; these archives would always have their first level directory stripped, whereas default behavior would to leave the archive structure as is.

So, in summary: JBrowse plugin installation is giving me guff; I can see a few ways to resolve the issue and am happy to make whatever changes and issue a pull request, but wanted to see what the prevailing opinion was.

If there's any more information I can provide, do let me know.

Thanks, -Luke

relevant apollo-config.groovy snippet:

    plugins {
        "G-OnRamp_plugin"{
        git= "https://github.com/Yating-L/JBrowse_plugins"
        branch = "master"
      }
deepakunni3 commented 7 years ago

Hi Luke,

Thanks for reporting this issue. Yes, we realized that there are issues with building Apollo. If you have a solution for this issue then feel free to create a PR.

We can take a look, make suggestions (if any) and then merge it.

Cheers,

Deepak

On Wed, Sep 13, 2017 at 4:15 PM, luke notifications@github.com wrote:

I am attempting to './apollo deploy' with a particular JBrowse plugin https://github.com/Yating-L/JBrowse_plugins, but get this error:

94 error code EISDIR 95 error errno -21 96 error syscall read 97 error eisdir EISDIR: illegal operation on a directory, read

Having given it the ol' Google/StackOverflow treatment, I suspect the change from bower dependency resolution has brought this on; I had no issues compiling a WAR file with the plugin in 2.0.6, but npm requires a 'package.json' in the repository for 'install' to work https://docs.npmjs.com/cli/install#description -- which this repo lacks.

Also, the previous version did not seem to need SSH configuration to be able to clone repos, presumably using https; this seems to be a consequence of the syntax chosen, as

git clone https://github.com/Yating-L/JBrowse_plugins.git

works without SSH configuration and

git clone git@github.com:Yating-L/JBrowse_plugins.git

does not.

It seems as though there are a few options:

  • add an (empty?) package.json to the repo so that npm passes its superficial check
  • revise the github installation logic to avoid using npm for plugins
  • use the 'url' parameter to pass in files directly

speaking to the 'url' parameter options, i made some minor alterations to the source to test that out (right now it just tests to see if the URL parameter is true then copies whatever was linked to a file named after the plugin name https://github.com/GMOD/Apollo/blob/eb4764788909907013d02664742b6a99489480a7/build.gradle#L204), allowing you to set a 'zip' flag like in other areas of the build script. The issue there is that github archives contain the repo contents in a folder named after the repo, whereas other zip files linked to in the 'url' field have no such guarantee, so we are left unsure as to wether or not to move files from the base directory in the archive. One option is to check if the link supplied was a github link; these archives would always have their first level directory stripped, whereas default behavior would to leave the archive structure as is.

So, in summary: JBrowse plugin installation is giving me guff; I can see a few ways to resolve the issue and am happy to make whatever changes and issue a pull request, but wanted to see what the prevailing opinion was.

If there's any more information I can provide, do let me know.

Thanks, -Luke

relevant apollo-config.groovy snippet:

plugins {
    "G-OnRamp_plugin"{
    git= "https://github.com/Yating-L/JBrowse_plugins"
    branch = "master"
  }

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/GMOD/Apollo/issues/1752, or mute the thread https://github.com/notifications/unsubscribe-auth/AEUgYAlEBoOc0Pm9j9qf2WP_GmRQQqMTks5siEX2gaJpZM4PWvO6 .

-- Research Analyst S104A Animal Science Research Center, University of Missouri, Columbia

nathandunn commented 7 years ago

@luke-c-sargent Some questions:

luke-c-sargent commented 7 years ago

Thanks for the reply; to answer your questions: node: v6.11.3 npm v3.10.10 Gradle not found using gradlew

i have been experimenting on a fork branch to see if I can make some progress (by modifying build.gradle): https://github.com/luke-c-sargent/Apollo/blob/jbrowse_plugin_git_methods/build.gradle

roughly, this sets the git default method to "shell" (with the option to request "npm" installation) which uses the commented-out function cloneRepoNoBower (renamed cloneRepoShell, with a small addendum of tasks.installJBrowseScript.execute() at the end). in this configuration, a WAR is created that seems to behave as desired (i was able to use the custom plugins and visualize some data / tracks), though I've since had some issues in getting unit tests to run (that output is pasted after the stacktrace -- any input on those issues would be welcome).

stacktrace:

Grails not found using grailsw
Gradle not found using gradlew
Node Version: 6
Npm Version: 3
javac 1.8.0_131 found
javac installed
JDK 1.8 found: javac 1.8.0_131
final JBrowse settings [git:[url:https://github.com/GMOD/jbrowse, branch:master, alwaysPull:true, alwaysRecheck:true], plugins:[G-OnRamp_plugin:[git:https://github.com/Yating-L/JBrowse_plugins, branch:master]]]
final plugins [WebApollo:[included:true], RegexSequenceSearch:[included:true], HideTrackLabels:[included:true], G-OnRamp_plugin:[git:https://github.com/Yating-L/JBrowse_plugins, branch:master]]
Copying apollo plugin
handling jbrowse release [git:[url:https://github.com/GMOD/jbrowse, branch:master, alwaysPull:true, alwaysRecheck:true], plugins:[G-OnRamp_plugin:[git:https://github.com/Yating-L/JBrowse_plugins, branch:master]]]
handling jbrowse [git:[url:https://github.com/GMOD/jbrowse, branch:master, alwaysPull:true, alwaysRecheck:true], plugins:[G-OnRamp_plugin:[git:https://github.com/Yating-L/JBrowse_plugins, branch:master]]]
:evaluateJBrowseConfigs UP-TO-DATE
:installJBrowse
installing jbrowse [git:[url:https://github.com/GMOD/jbrowse, branch:master, alwaysPull:true, alwaysRecheck:true], plugins:[G-OnRamp_plugin:[git:https://github.com/Yating-L/JBrowse_plugins, branch:master]]]
jbrowse@1.12.3 (git://github.com/gmod/jbrowse.git#9d765aecaee02a41844fed11a241fdb4c35fc9f8) /opt/Apollo-2.0.7/jbrowse-download
├── jDataView@1.1.0  (git+https://github.com/rbuels/jdataview.git#db0893d468ba034fadbbb5700d4ed79ab0dc236e)
├── json-schema@0.2.3  (git+https://github.com/kriszyp/json-schema.git#24c4ed1b2359ab457a00e90606a777c2962ecd3b)
├── jszlib@0.0.1  (git+https://github.com/cmdcolin/jszlib.git#c966ceac3b283d2248764c09e07a4b268edd5032)
└── lazyload@1.0.0  (git+https://github.com/cmdcolin/lazyload.git#042bdb8e5f9d84932c192ee1c295579b5e0c8100)

npm WARN jbrowse@1.12.3 No license field.
:copyApolloPlugin UP-TO-DATE
:installJBrowsePlugins
installing jbrowse plugins  [WebApollo:[included:true], RegexSequenceSearch:[included:true], HideTrackLabels:[included:true], G-OnRamp_plugin:[git:https://github.com/Yating-L/JBrowse_plugins, branch:master]]
Evaluating plugin WebApollo=[included:true]
Plugin jbrowse-download/plugins/WebApollo exists and appears valid.
Evaluating plugin RegexSequenceSearch=[included:true]
Plugin jbrowse-download/plugins/RegexSequenceSearch exists and appears valid.
Evaluating plugin HideTrackLabels=[included:true]
Plugin jbrowse-download/plugins/HideTrackLabels exists and appears valid.
Evaluating plugin G-OnRamp_plugin=[git:https://github.com/Yating-L/JBrowse_plugins, branch:master]
Plugin is supplied by git
Cloning 'https://github.com/Yating-L/JBrowse_plugins' into 'jbrowse-download/plugins/G-OnRamp_plugin'
Cloning repo with https://github.com/Yating-L/JBrowse_plugins,jbrowse-download/plugins/G-OnRamp_plugin,master
final github url: Yating-L/JBrowse_plugins#master
using npm to install: Yating-L/JBrowse_plugins#master
npm ERR! addLocal Could not install /tmp/npm-14186-2d204e48/git-cache-83179de2/346ffd7265beea56ec6f72519bbb9d8ad104b6d0
npm ERR! addLocal Could not install /tmp/npm-14186-2d204e48/git-cache-339967b8/346ffd7265beea56ec6f72519bbb9d8ad104b6d0
npm ERR! addLocal Could not install /tmp/npm-14186-2d204e48/git-cache-47d52f4d/346ffd7265beea56ec6f72519bbb9d8ad104b6d0
npm ERR! Linux 4.4.0-62-generic
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "install" "Yating-L/JBrowse_plugins#master" "--ignore-scripts"
npm ERR! node v6.11.3
npm ERR! npm  v3.10.10
npm ERR! code EISDIR
npm ERR! errno -21
npm ERR! syscall read

npm ERR! eisdir EISDIR: illegal operation on a directory, read
npm ERR! eisdir This is most likely not a problem with npm itself
npm ERR! eisdir and is related to npm not being able to find a package.json in
npm ERR! eisdir a package you are trying to install.

npm ERR! Please include the following file with any support request:
npm ERR!     /opt/Apollo-2.0.7/npm-debug.log
:installJBrowsePlugins FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/opt/Apollo-2.0.7/build.gradle' line: 309

* What went wrong:
Execution failed for task ':installJBrowsePlugins'.
> Process 'command 'npm'' finished with non-zero exit value 235

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 6.41 secs
***************************************
NOTE: Please set the memory for your servlet container (tomcat, jetty, etc.) or Apollo may not start correctly: http://genomearchitect.readthedocs.io/en/latest/Troubleshooting.html#suggested-tomcat-memory-settings
***************************************

relevant ./apollo test output:

(...)
2017-09-18 14:00:35,971 [main] WARN  apollo.FeatureService  - No proper type for the CV is set {"cv":{"name":"feature_property"}}
| Running 15 integration tests... 97 of 97
| Failure:  a round-trip of GFF3 export and import(org.bbop.apollo.RequestHandlingServiceIntegrationSpec)
|  org.codehaus.groovy.grails.web.converters.exceptions.ConverterException: Error parsing JSON
    at grails.converters.JSON.parse(JSON.java:278)
    at org.bbop.apollo.RequestHandlingServiceIntegrationSpec.a round-trip of GFF3 export and import(RequestHandlingServiceIntegrationSpec.groovy:3413)
Caused by: org.codehaus.groovy.grails.web.json.JSONException: Missing value. at character 0 of 
    at grails.converters.JSON.parse(JSON.java:269)
    ... 1 more
| Running 15 integration tests... 98 of 98
| Failure:  a round-trip with all feature types(org.bbop.apollo.RequestHandlingServiceIntegrationSpec)
|  org.codehaus.groovy.grails.web.converters.exceptions.ConverterException: Error parsing JSON
    at grails.converters.JSON.parse(JSON.java:278)
    at org.bbop.apollo.RequestHandlingServiceIntegrationSpec.a round-trip with all feature types(RequestHandlingServiceIntegrationSpec.groovy:3521)
Caused by: org.codehaus.groovy.grails.web.json.JSONException: Missing value. at character 0 of 
    at grails.converters.JSON.parse(JSON.java:269)
    ... 1 more
| Completed 101 integration tests, 2 failed in 0m 31s
| Tests FAILED  - view reports in /opt/Apollo-2.0.7/target/test-reports
nathandunn commented 7 years ago

jbrowse@1.12.3 (git://github.com/gmod/jbrowse.git#9d765aecaee02a41844fed11a241fdb4c35fc9f8) /opt/Apollo-2.0.7/jbrowse-download

So, you should comment out the 1.12.3 part as that version of JBrowse didn’t use NPM.

I would use:

tag = "9d765aecaee02a41844fed11a241fdb4c35fc9f8"

instead or

branch = “master"

I’m not sure it will fix everything, but I know it won’t work the other way.

Also, if you have your full apollo-config.groovy minus passwords, that might help as well.

luke-c-sargent commented 7 years ago

my apollo-config.groovy is a slightly modified version of the sample-postgres-apollo-config.groovy; the only difference (besides user/pass) is in the jbrowse section below:

jbrowse {
    git {
        url= "https://github.com/GMOD/jbrowse"
//         tag = "9d765aecaee02a41844fed11a241fdb4c35fc9f8"
        branch = "master"
        alwaysPull = true
        alwaysRecheck = true

// Warning: We are still testing the performance of NeatFeatures plugins in combination with Apollo.
// We advise caution if enabling these plugins with Apollo until this process is finalized.
    }
    plugins {
        "G-OnRamp_plugin"{
        git= "https://github.com/Yating-L/JBrowse_plugins"
        branch = "master"
        }
//        NeatHTMLFeatures{
//            included = true
//        }
//        NeatCanvasFeatures{
//            included = true
//        }
//        RegexSequenceSearch{
//            included = true
//        }
//        HideTrackLabels{
//            included = true
//        }
//        MyVariantInfo {
//            git = 'https://github.com/GMOD/myvariantviewer'
//            branch = 'master'
//            alwaysRecheck = "true"
//            alwaysPull = "true"
//        }
//        SashimiPlot {
//            git = 'https://github.com/cmdcolin/sashimiplot'
//            branch = 'master'
//            alwaysPull = "true"
//        }
    }
}

i will switch to preferring the tag; though from what you wrote it should work regardless, this will be useful to lock in version should the master change significantly.

thanks again for the assistance!

luke-c-sargent commented 7 years ago

In further news, I have resolved the test errors*. the RequestHandlingServiceIntegrationSpec test used a process that it executes to create a JSONArray, but it was only outputting to stderr (so, no text -> Error parsing JSON). that stderr indicated that the two gff3 perl scripts could not find the requisite libraries; as it turns out, JBrowse installs perl dependencies to the jbrowse-download dir, so adding that to those scripts resolved that issue.

TL;DR: changed this: use lib "$RealBin/../../src/perl5"; to this: use lib "$RealBin/../../jbrowse-download/src/perl5";

*: as it turns out, the first ./apollo test displays this error:

Fatal error running tests: org/jsondoc/core/pojo/ApiObjectDoc

but subsequent test runs do not. I will look in to that further.

nathandunn commented 7 years ago

Weird. Thanks for reporting.

I don’t see what the difference between the changes are.

On Sep 18, 2017, at 4:15 PM, luke notifications@github.com wrote:

In further news, I have resolved the test errors*. the RequestHandlingServiceIntegrationSpec test used a process that it executes to create a JSONArray, but it was only outputting to stderr (so, no text -> Error parsing JSON). that stderr indicated that the two gff3 perl scripts could not find the requisite libraries; as it turns out, JBrowse installs perl dependencies to the jbrowse-download dir, so adding that to those scripts resolved that issue.

TL;DR: changed this https://github.com/GMOD/Apollo/blob/7e766517f4fbf471f51253ede6fb28a59c0c14a0/tools/data/add_transcripts_from_gff3_to_annotations.pl#L7: use lib "$RealBin/../../src/perl5"; to this: use lib "$RealBin/../../src/perl5";

*: as it turns out, the first ./apollo test displays this error:

Fatal error running tests: org/jsondoc/core/pojo/ApiObjectDoc but subsequent test runs do not. I will look in to that further.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/GMOD/Apollo/issues/1752#issuecomment-330382525, or mute the thread https://github.com/notifications/unsubscribe-auth/AAt2qs-XMjHhgd0m1SYktnm9hAzyUUjLks5sjvl6gaJpZM4PWvO6.

luke-c-sargent commented 7 years ago

there isnt one, because i forgot to make the change before submitting... terribly sorry. use lib "$RealBin/../../src/perl5"; to this: use lib "$RealBin/../../jbrowse-download/src/perl5";

nathandunn commented 7 years ago

Ahh . . K. this explains some of what we have been seeing.

On Sep 19, 2017, at 9:06 AM, luke notifications@github.com wrote:

there isnt one, because i forgot to make the change before submitting... terribly sorry. use lib "$RealBin/../../src/perl5"; to this: use lib "$RealBin/../../jbrowse-download/src/perl5";

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/GMOD/Apollo/issues/1752#issuecomment-330589039, or mute the thread https://github.com/notifications/unsubscribe-auth/AAt2qtNnWuTlkyFwEb7akSKrGFj3cHG4ks5sj-aJgaJpZM4PWvO6.

luke-c-sargent commented 7 years ago

I have submitted a pull request concerning these issues: #1758

nathandunn commented 7 years ago

This was fixed by #1758. Retesting seemed to indicate that it was still working: :+1: