Closed luke-c-sargent closed 7 years ago
Hi Luke,
Thanks for reporting this issue. Yes, we realized that there are issues with building Apollo. If you have a solution for this issue then feel free to create a PR.
We can take a look, make suggestions (if any) and then merge it.
Cheers,
Deepak
On Wed, Sep 13, 2017 at 4:15 PM, luke notifications@github.com wrote:
I am attempting to './apollo deploy' with a particular JBrowse plugin https://github.com/Yating-L/JBrowse_plugins, but get this error:
94 error code EISDIR 95 error errno -21 96 error syscall read 97 error eisdir EISDIR: illegal operation on a directory, read
Having given it the ol' Google/StackOverflow treatment, I suspect the change from bower dependency resolution has brought this on; I had no issues compiling a WAR file with the plugin in 2.0.6, but npm requires a 'package.json' in the repository for 'install' to work https://docs.npmjs.com/cli/install#description -- which this repo lacks.
Also, the previous version did not seem to need SSH configuration to be able to clone repos, presumably using https; this seems to be a consequence of the syntax chosen, as
git clone https://github.com/Yating-L/JBrowse_plugins.git
works without SSH configuration and
git clone git@github.com:Yating-L/JBrowse_plugins.git
does not.
It seems as though there are a few options:
- add an (empty?) package.json to the repo so that npm passes its superficial check
- revise the github installation logic to avoid using npm for plugins
- use the 'url' parameter to pass in files directly
speaking to the 'url' parameter options, i made some minor alterations to the source to test that out (right now it just tests to see if the URL parameter is true then copies whatever was linked to a file named after the plugin name https://github.com/GMOD/Apollo/blob/eb4764788909907013d02664742b6a99489480a7/build.gradle#L204), allowing you to set a 'zip' flag like in other areas of the build script. The issue there is that github archives contain the repo contents in a folder named after the repo, whereas other zip files linked to in the 'url' field have no such guarantee, so we are left unsure as to wether or not to move files from the base directory in the archive. One option is to check if the link supplied was a github link; these archives would always have their first level directory stripped, whereas default behavior would to leave the archive structure as is.
So, in summary: JBrowse plugin installation is giving me guff; I can see a few ways to resolve the issue and am happy to make whatever changes and issue a pull request, but wanted to see what the prevailing opinion was.
If there's any more information I can provide, do let me know.
Thanks, -Luke
relevant apollo-config.groovy snippet:
plugins { "G-OnRamp_plugin"{ git= "https://github.com/Yating-L/JBrowse_plugins" branch = "master" }
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/GMOD/Apollo/issues/1752, or mute the thread https://github.com/notifications/unsubscribe-auth/AEUgYAlEBoOc0Pm9j9qf2WP_GmRQQqMTks5siEX2gaJpZM4PWvO6 .
-- Research Analyst S104A Animal Science Research Center, University of Missouri, Columbia
@luke-c-sargent Some questions:
Thanks for the reply; to answer your questions: node: v6.11.3 npm v3.10.10 Gradle not found using gradlew
i have been experimenting on a fork branch to see if I can make some progress (by modifying build.gradle): https://github.com/luke-c-sargent/Apollo/blob/jbrowse_plugin_git_methods/build.gradle
roughly, this sets the git default method to "shell" (with the option to request "npm" installation) which uses the commented-out function cloneRepoNoBower (renamed cloneRepoShell, with a small addendum of tasks.installJBrowseScript.execute()
at the end).
in this configuration, a WAR is created that seems to behave as desired (i was able to use the custom plugins and visualize some data / tracks), though I've since had some issues in getting unit tests to run (that output is pasted after the stacktrace -- any input on those issues would be welcome).
stacktrace:
Grails not found using grailsw
Gradle not found using gradlew
Node Version: 6
Npm Version: 3
javac 1.8.0_131 found
javac installed
JDK 1.8 found: javac 1.8.0_131
final JBrowse settings [git:[url:https://github.com/GMOD/jbrowse, branch:master, alwaysPull:true, alwaysRecheck:true], plugins:[G-OnRamp_plugin:[git:https://github.com/Yating-L/JBrowse_plugins, branch:master]]]
final plugins [WebApollo:[included:true], RegexSequenceSearch:[included:true], HideTrackLabels:[included:true], G-OnRamp_plugin:[git:https://github.com/Yating-L/JBrowse_plugins, branch:master]]
Copying apollo plugin
handling jbrowse release [git:[url:https://github.com/GMOD/jbrowse, branch:master, alwaysPull:true, alwaysRecheck:true], plugins:[G-OnRamp_plugin:[git:https://github.com/Yating-L/JBrowse_plugins, branch:master]]]
handling jbrowse [git:[url:https://github.com/GMOD/jbrowse, branch:master, alwaysPull:true, alwaysRecheck:true], plugins:[G-OnRamp_plugin:[git:https://github.com/Yating-L/JBrowse_plugins, branch:master]]]
:evaluateJBrowseConfigs UP-TO-DATE
:installJBrowse
installing jbrowse [git:[url:https://github.com/GMOD/jbrowse, branch:master, alwaysPull:true, alwaysRecheck:true], plugins:[G-OnRamp_plugin:[git:https://github.com/Yating-L/JBrowse_plugins, branch:master]]]
jbrowse@1.12.3 (git://github.com/gmod/jbrowse.git#9d765aecaee02a41844fed11a241fdb4c35fc9f8) /opt/Apollo-2.0.7/jbrowse-download
├── jDataView@1.1.0 (git+https://github.com/rbuels/jdataview.git#db0893d468ba034fadbbb5700d4ed79ab0dc236e)
├── json-schema@0.2.3 (git+https://github.com/kriszyp/json-schema.git#24c4ed1b2359ab457a00e90606a777c2962ecd3b)
├── jszlib@0.0.1 (git+https://github.com/cmdcolin/jszlib.git#c966ceac3b283d2248764c09e07a4b268edd5032)
└── lazyload@1.0.0 (git+https://github.com/cmdcolin/lazyload.git#042bdb8e5f9d84932c192ee1c295579b5e0c8100)
npm WARN jbrowse@1.12.3 No license field.
:copyApolloPlugin UP-TO-DATE
:installJBrowsePlugins
installing jbrowse plugins [WebApollo:[included:true], RegexSequenceSearch:[included:true], HideTrackLabels:[included:true], G-OnRamp_plugin:[git:https://github.com/Yating-L/JBrowse_plugins, branch:master]]
Evaluating plugin WebApollo=[included:true]
Plugin jbrowse-download/plugins/WebApollo exists and appears valid.
Evaluating plugin RegexSequenceSearch=[included:true]
Plugin jbrowse-download/plugins/RegexSequenceSearch exists and appears valid.
Evaluating plugin HideTrackLabels=[included:true]
Plugin jbrowse-download/plugins/HideTrackLabels exists and appears valid.
Evaluating plugin G-OnRamp_plugin=[git:https://github.com/Yating-L/JBrowse_plugins, branch:master]
Plugin is supplied by git
Cloning 'https://github.com/Yating-L/JBrowse_plugins' into 'jbrowse-download/plugins/G-OnRamp_plugin'
Cloning repo with https://github.com/Yating-L/JBrowse_plugins,jbrowse-download/plugins/G-OnRamp_plugin,master
final github url: Yating-L/JBrowse_plugins#master
using npm to install: Yating-L/JBrowse_plugins#master
npm ERR! addLocal Could not install /tmp/npm-14186-2d204e48/git-cache-83179de2/346ffd7265beea56ec6f72519bbb9d8ad104b6d0
npm ERR! addLocal Could not install /tmp/npm-14186-2d204e48/git-cache-339967b8/346ffd7265beea56ec6f72519bbb9d8ad104b6d0
npm ERR! addLocal Could not install /tmp/npm-14186-2d204e48/git-cache-47d52f4d/346ffd7265beea56ec6f72519bbb9d8ad104b6d0
npm ERR! Linux 4.4.0-62-generic
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "install" "Yating-L/JBrowse_plugins#master" "--ignore-scripts"
npm ERR! node v6.11.3
npm ERR! npm v3.10.10
npm ERR! code EISDIR
npm ERR! errno -21
npm ERR! syscall read
npm ERR! eisdir EISDIR: illegal operation on a directory, read
npm ERR! eisdir This is most likely not a problem with npm itself
npm ERR! eisdir and is related to npm not being able to find a package.json in
npm ERR! eisdir a package you are trying to install.
npm ERR! Please include the following file with any support request:
npm ERR! /opt/Apollo-2.0.7/npm-debug.log
:installJBrowsePlugins FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '/opt/Apollo-2.0.7/build.gradle' line: 309
* What went wrong:
Execution failed for task ':installJBrowsePlugins'.
> Process 'command 'npm'' finished with non-zero exit value 235
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
BUILD FAILED
Total time: 6.41 secs
***************************************
NOTE: Please set the memory for your servlet container (tomcat, jetty, etc.) or Apollo may not start correctly: http://genomearchitect.readthedocs.io/en/latest/Troubleshooting.html#suggested-tomcat-memory-settings
***************************************
relevant ./apollo test
output:
(...)
2017-09-18 14:00:35,971 [main] WARN apollo.FeatureService - No proper type for the CV is set {"cv":{"name":"feature_property"}}
| Running 15 integration tests... 97 of 97
| Failure: a round-trip of GFF3 export and import(org.bbop.apollo.RequestHandlingServiceIntegrationSpec)
| org.codehaus.groovy.grails.web.converters.exceptions.ConverterException: Error parsing JSON
at grails.converters.JSON.parse(JSON.java:278)
at org.bbop.apollo.RequestHandlingServiceIntegrationSpec.a round-trip of GFF3 export and import(RequestHandlingServiceIntegrationSpec.groovy:3413)
Caused by: org.codehaus.groovy.grails.web.json.JSONException: Missing value. at character 0 of
at grails.converters.JSON.parse(JSON.java:269)
... 1 more
| Running 15 integration tests... 98 of 98
| Failure: a round-trip with all feature types(org.bbop.apollo.RequestHandlingServiceIntegrationSpec)
| org.codehaus.groovy.grails.web.converters.exceptions.ConverterException: Error parsing JSON
at grails.converters.JSON.parse(JSON.java:278)
at org.bbop.apollo.RequestHandlingServiceIntegrationSpec.a round-trip with all feature types(RequestHandlingServiceIntegrationSpec.groovy:3521)
Caused by: org.codehaus.groovy.grails.web.json.JSONException: Missing value. at character 0 of
at grails.converters.JSON.parse(JSON.java:269)
... 1 more
| Completed 101 integration tests, 2 failed in 0m 31s
| Tests FAILED - view reports in /opt/Apollo-2.0.7/target/test-reports
jbrowse@1.12.3 (git://github.com/gmod/jbrowse.git#9d765aecaee02a41844fed11a241fdb4c35fc9f8) /opt/Apollo-2.0.7/jbrowse-download
So, you should comment out the 1.12.3 part as that version of JBrowse didn’t use NPM.
I would use:
tag = "9d765aecaee02a41844fed11a241fdb4c35fc9f8"
instead or
branch = “master"
I’m not sure it will fix everything, but I know it won’t work the other way.
Also, if you have your full apollo-config.groovy minus passwords, that might help as well.
my apollo-config.groovy is a slightly modified version of the sample-postgres-apollo-config.groovy; the only difference (besides user/pass) is in the jbrowse section below:
jbrowse {
git {
url= "https://github.com/GMOD/jbrowse"
// tag = "9d765aecaee02a41844fed11a241fdb4c35fc9f8"
branch = "master"
alwaysPull = true
alwaysRecheck = true
// Warning: We are still testing the performance of NeatFeatures plugins in combination with Apollo.
// We advise caution if enabling these plugins with Apollo until this process is finalized.
}
plugins {
"G-OnRamp_plugin"{
git= "https://github.com/Yating-L/JBrowse_plugins"
branch = "master"
}
// NeatHTMLFeatures{
// included = true
// }
// NeatCanvasFeatures{
// included = true
// }
// RegexSequenceSearch{
// included = true
// }
// HideTrackLabels{
// included = true
// }
// MyVariantInfo {
// git = 'https://github.com/GMOD/myvariantviewer'
// branch = 'master'
// alwaysRecheck = "true"
// alwaysPull = "true"
// }
// SashimiPlot {
// git = 'https://github.com/cmdcolin/sashimiplot'
// branch = 'master'
// alwaysPull = "true"
// }
}
}
i will switch to preferring the tag
; though from what you wrote it should work regardless, this will be useful to lock in version should the master change significantly.
thanks again for the assistance!
In further news, I have resolved the test errors*. the RequestHandlingServiceIntegrationSpec
test used a process that it executes to create a JSONArray, but it was only outputting to stderr (so, no text -> Error parsing JSON). that stderr indicated that the two gff3 perl scripts could not find the requisite libraries; as it turns out, JBrowse installs perl dependencies to the jbrowse-download
dir, so adding that to those scripts resolved that issue.
TL;DR:
changed this:
use lib "$RealBin/../../src/perl5";
to this:
use lib "$RealBin/../../jbrowse-download/src/perl5";
*: as it turns out, the first ./apollo test
displays this error:
Fatal error running tests: org/jsondoc/core/pojo/ApiObjectDoc
but subsequent test runs do not. I will look in to that further.
Weird. Thanks for reporting.
I don’t see what the difference between the changes are.
On Sep 18, 2017, at 4:15 PM, luke notifications@github.com wrote:
In further news, I have resolved the test errors*. the RequestHandlingServiceIntegrationSpec test used a process that it executes to create a JSONArray, but it was only outputting to stderr (so, no text -> Error parsing JSON). that stderr indicated that the two gff3 perl scripts could not find the requisite libraries; as it turns out, JBrowse installs perl dependencies to the jbrowse-download dir, so adding that to those scripts resolved that issue.
TL;DR: changed this https://github.com/GMOD/Apollo/blob/7e766517f4fbf471f51253ede6fb28a59c0c14a0/tools/data/add_transcripts_from_gff3_to_annotations.pl#L7: use lib "$RealBin/../../src/perl5"; to this: use lib "$RealBin/../../src/perl5";
*: as it turns out, the first ./apollo test displays this error:
Fatal error running tests: org/jsondoc/core/pojo/ApiObjectDoc but subsequent test runs do not. I will look in to that further.
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/GMOD/Apollo/issues/1752#issuecomment-330382525, or mute the thread https://github.com/notifications/unsubscribe-auth/AAt2qs-XMjHhgd0m1SYktnm9hAzyUUjLks5sjvl6gaJpZM4PWvO6.
there isnt one, because i forgot to make the change before submitting... terribly sorry.
use lib "$RealBin/../../src/perl5";
to this:
use lib "$RealBin/../../jbrowse-download/src/perl5";
Ahh . . K. this explains some of what we have been seeing.
On Sep 19, 2017, at 9:06 AM, luke notifications@github.com wrote:
there isnt one, because i forgot to make the change before submitting... terribly sorry. use lib "$RealBin/../../src/perl5"; to this: use lib "$RealBin/../../jbrowse-download/src/perl5";
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/GMOD/Apollo/issues/1752#issuecomment-330589039, or mute the thread https://github.com/notifications/unsubscribe-auth/AAt2qtNnWuTlkyFwEb7akSKrGFj3cHG4ks5sj-aJgaJpZM4PWvO6.
I have submitted a pull request concerning these issues: #1758
This was fixed by #1758. Retesting seemed to indicate that it was still working: :+1:
I am attempting to './apollo deploy' with a particular JBrowse plugin, but get this error:
Having given it the ol' Google/StackOverflow treatment, I suspect the change from bower dependency resolution has brought this on; I had no issues compiling a WAR file with the plugin in 2.0.6, but npm requires a 'package.json' in the repository for 'install' to work -- which this repo lacks.
Also, the previous version did not seem to need SSH configuration to be able to clone repos, presumably using https; this seems to be a consequence of the syntax chosen, as
works without SSH configuration and
does not.
It seems as though there are a few options:
speaking to the 'url' parameter options, i made some minor alterations to the source to test that out (right now it just tests to see if the URL parameter is true then copies whatever was linked to a file named after the plugin name), allowing you to set a 'zip' flag like in other areas of the build script. The issue there is that github archives contain the repo contents in a folder named after the repo, whereas other zip files linked to in the 'url' field have no such guarantee, so we are left unsure as to wether or not to move files from the base directory in the archive. One option is to check if the link supplied was a github link; these archives would always have their first level directory stripped, whereas default behavior would to leave the archive structure as is.
So, in summary: JBrowse plugin installation is giving me guff; I can see a few ways to resolve the issue and am happy to make whatever changes and issue a pull request, but wanted to see what the prevailing opinion was.
If there's any more information I can provide, do let me know.
Thanks, -Luke
relevant apollo-config.groovy snippet: