search

GMOD / Apollo

Genome annotation editor with a Java Server backend and a Javascript client that runs in a web browser as a JBrowse plugin.
http://genomearchitect.readthedocs.io/
Other
127 stars 85 forks source link

Setting up BLAT #1528

Closed mastiles closed 7 years ago

mastiles commented 7 years ago

Sorry, me again.

I've been playing around with WebApollo today, and realised the BLAT search doesn't work - I'm not sure what to input for the BLAT database option, or whether I will need to configure something else/download something else (I had simply inputted the directory my sequence is in, i.e. /opt/apollo/data/*). The only information I can find in the guide is "When you setup your organism in the web interface, you can then enter the location of the sequence search database for BLAT" - but I'm stumped as to whether this is meant to be a link to an online database, or if I should be downloading datasets from NCBI to create something (in which case how do I configure this?)

Sorry if this is an obvious question

Morgaine

nathandunn commented 7 years ago

You need to point to a 2bit file and point to it in the Organism Tab:

/myorganisms/myblatdirectory/amel.2bit

More info here:

https://genome.ucsc.edu/goldenpath/help/twoBit.html https://genome.ucsc.edu/goldenpath/help/twoBit.html

Nathan

On Mar 15, 2017, at 7:17 AM, mastiles notifications@github.com wrote:

Sorry, me again.

I've been playing around with WebApollo today, and realised the BLAT search doesn't work - I'm not sure what to input for the BLAT database option, or whether I will need to configure something else/download something else (I had simply inputted the directory my sequence is in, i.e. /opt/apollo/data/*). The only information I can find in the guide is "When you setup your organism in the web interface, you can then enter the location of the sequence search database for BLAT" - but I'm stumped as to whether this is meant to be a link to an online database, or if I should be downloading datasets from NCBI to create something (in which case how do I configure this?)

Sorry if this is an obvious question

Morgaine

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/GMOD/Apollo/issues/1528, or mute the thread https://github.com/notifications/unsubscribe-auth/AAt2qre82NgNM31Xr-Drgt2FD3MHGulgks5rl_L6gaJpZM4Md__w.

nathandunn commented 7 years ago

Please re-open if you have any trouble.

mastiles commented 7 years ago

Sorry, this is definitely something that I'm sure will be really simple - I downloaded the utilities faToTwoBit twoBitInfo and twoBitToFa, but these then when I do faToTwoBit ~/Apollo-2.0.6/data/h_pylori.fasta /usr/local/bin/blat/h_pylori.2bit, I get an error message: faToTwoBit: command not found

I'm doing this in the Downloads directory, where it is, so I'm pretty sure it's not to do with location...

nathandunn commented 7 years ago

It has to be in your path. Probably the easiest is to prepend with: ./

./faToTwoBit ~/Apollo-2.0.6/data/h_pylori.fasta /usr/local/bin/blat/h_pylori.2bit

Conversely, you can add the binaries to a directory supported by the $PATH variable and it will always be available.

mastiles commented 7 years ago

Okay that worked (thank you!), but now I have another issue - I keep getting this error when trying to use the search feature: Error running search: Cannot run program "/usr/local/bin/blat": error=13, Permission denied

I set permissions chmod ugo+rwx on blat and all of the files within it...

nathandunn commented 7 years ago 
ls -la /usr/local/bin/blat

?

mastiles commented 7 years ago

drwxrwxrwx 2 root root 4096 Mar 15 17:53 . drwxr-xr-x 3 root root 4096 Mar 15 16:36 .. -rwxrwxrwx 1 beebylab beebylab 410419 Mar 15 17:48 c_jejuni.2bit -rwxrwxrwx 1 beebylab beebylab 417335 Mar 15 17:47 h_pylori.2bit -rwxrwxrwx 1 beebylab beebylab 1202308 Mar 15 17:49 s_enterica.2bit -rwxrwxrwx 1 beebylab beebylab 4298188 Mar 15 17:18 twoBitInfo -rwxrwxrwx 1 beebylab beebylab 4626021 Mar 15 17:18 twoBitToFa

Could the issue be that the user for the MySQL database linked to Web Apollo is called apollo rather than beebylab?

nathandunn commented 7 years ago

Its because you don't have blat installed there. You need to install it.

nathandunn commented 7 years ago

It might be available as a package on your system (it is for the mac), or you can just download it directly:

https://genome.ucsc.edu/FAQ/FAQblat.html

mastiles commented 7 years ago

Oh that makes sense, thank you - silly of me to think I could get away with just downloading the individual utilities!

mastiles commented 7 years ago

Hi again - sorry to be such a nuisance.

I downloaded BLAT and went through the same process, changed permissions again and am still getting the permission denied message. The directory /usr/local/bin/blat ls -la looks like:

drwxrwxrwx 2 root root 4096 Mar 16 16:54 . drwxr-xr-x 3 root root 4096 Mar 15 16:36 .. -rwxrwxrwx 1 root root 410419 Mar 16 16:54 c_jejuni.2bit -rwxrwxrwx 1 root root 417335 Mar 16 16:54 h_pylori.2bit -rwxrwxrwx 1 root root 1202308 Mar 16 16:51 s_enterica.2bit

Wondering what I'm doing wrong/have overlooked now...

deepakunni3 commented 7 years ago

Hi Morgaine,

It looks like you have a folder called blat in /usr/local/bin when in reality, /usr/local/bin should contain only binary files. i.e. programs that are executed.

The blat tool that you need is located at UCSC FTP: http://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64/blat/

Download the blat binary, and then move it to /usr/local/bin such that when you now call blat from the command line, you get an output that looks like this:

blat - Standalone BLAT v. 36x1 fast sequence search command line tool
usage:
   blat database query [-ooc=11.ooc] output.psl
where:
   database and query are each either a .fa, .nib or .2bit file,
      or a list of these files with one file name per line.
   -ooc=11.ooc tells the program to load over-occurring 11-mers from
      an external file.  This will increase the speed
      by a factor of 40 in many cases, but is not required.
   output.psl is the name of the output file.
   Subranges of .nib and .2bit files may be specified using the syntax:
      /path/file.nib:seqid:start-end
   or
...

PS.: It would be better to move out your *.2bit files out of /usr/local/bin to somewhere more proper - perhaps your home directory or /opt/apollo.

mastiles commented 7 years ago

That worked, thank you!

Only issue now is, when I search with a part of sequence I know to be in my genome, I'm seeing "No Matches Found" - I assume this is an issue with the configuration - my blat and faToTwoBit binaries are in /usr/local/bin and my 2bit files are in ~/Apollo-2.0.6/data/twoBit

I've inputted "blat directory" in the database as, ie ~/Apollo-2.0.6/data/twoBit/c_jejuni.2bit

Thank you for all your help and advice

Morgaine

deepakunni3 commented 7 years ago

Glad to hear it worked out.

What is the length of your input sequence? It has to be 40 bases or more for nucleotide sequences.

mastiles commented 7 years ago

Yeah it was definitely over 40 - I tried lengths ranging fro 70 to 280 bases.

This is what I did:

Download BLAT binaries http://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64/blat/ and move to /usr/local/bin then chmod ugo+rwx blat

Download faToTwoBit binaries: http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/faToTwoBit/ and move to /usr/local/bin then chmod ugo+rwx faToTwoBit

(realised I could have just wgetted them straight into the folder, but I'm still learning the tricks of the trade!)

In Apollo-2.0.6/data

faToTwoBit s_enterica.fasta twoBit/s_enterica.2bit

Then in the 'BLAT Database' section of the Add Organism, put ~/Apollo-2.0.6/data/twoBit/c_jejuni.2bit

Thank you for taking time to help me!

nathandunn commented 7 years ago

Did you put in "~/Apollo-2.0.6/data/twoBit/c_jejuni.2bit” or did you put in your actual home directory like “/home/mastiles/Apollo-2.0.6/data/twoBit/c_jejuni.2bit”? I don’t think it will work in the former.

Nathan

On Mar 17, 2017, at 2:51 AM, mastiles notifications@github.com wrote:

Yeah it was definitely over 40 - I tried lengths ranging fro 70 to 280 bases.

This is what I did:

Download BLAT binaries http://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64/blat/ http://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64/blat/ and move to /usr/local/bin then chmod ugo+rwx blat

Download faToTwoBit binaries: http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/faToTwoBit/ http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/faToTwoBit/ and move to /usr/local/bin then chmod ugo+rwx faToTwoBit

(realised I could have just wgetted them straight into the folder, but I'm still learning the tricks of the trade!)

In Apollo-2.0.6/data

faToTwoBit s_enterica.fasta twoBit/s_enterica.2bit

Then in the 'BLAT Database' section of the Add Organism, put ~/Apollo-2.0.6/data/twoBit/c_jejuni.2bit

Thank you for taking time to help me!

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/GMOD/Apollo/issues/1528#issuecomment-287311680, or mute the thread https://github.com/notifications/unsubscribe-auth/AAt2qoKW2EbRLMjAqIMJvRvVygsS_eVfks5rmlemgaJpZM4Md__w.

mastiles commented 7 years ago

I hadn't but I just tried the latter and it still doesn't seem to be workin - I wonder if there are more binaries that I should install or if I need to do anything other than just faToTwoBit? I feel like I haven't done anything different to what is in the Web Apollo 2013 tutorial though...

nathandunn commented 7 years ago

What failure are you getting on the back-end in the tomcat logs (catalina.out)?

On Mar 17, 2017, at 7:56 AM, mastiles notifications@github.com wrote:

I hadn't but I just tried the latter and it still doesn't seem to be workin - I wonder if there are more binaries that I should install or if I need to do anything other than just faToTwoBit? I feel like I haven't done anything different to what is in the Web Apollo 2013 tutorial though...

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/GMOD/Apollo/issues/1528#issuecomment-287376721, or mute the thread https://github.com/notifications/unsubscribe-auth/AAt2qmlIUU_tIkQN0jFIUZar8xVNuy2Tks5rmp8OgaJpZM4Md__w.

mastiles commented 7 years ago

This is the output from yesterday and today:

2017-03-16 10:28:27,742 [DefaultPluginScheduler_Worker-6] ERROR apollo.PreferenceService - Removing stale preferences 2017-03-16 10:28:27,747 [DefaultPluginScheduler_Worker-6] ERROR apollo.PreferenceService - Removed 2 stale preferences 2017-03-16 16:55:30,735 [http-bio-8080-exec-43] INFO apollo.UserController - updateTrackListPreference 2017-03-16 16:55:30,736 [http-bio-8080-exec-43] INFO apollo.UserController - Added userOrganismPreference false Comparing c_jejuni to names 2017-03-16 16:55:31,797 [http-bio-8080-exec-49] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/names/root.json 2017-03-16 16:55:35,249 [clientInboundChannel-3] ERROR springwebsocket.GrailsSimpAnnotationMethodMessageHandler - Unhandled exception java.lang.reflect.InvocationTargetException at org.bbop.apollo.AnnotationEditorController$do_annotationEditor_closure4$_closure5$_closure6.doCall(AnnotationEditorController.groovy:1144) at org.bbop.apollo.AnnotationEditorController$do_annotationEditor_closure4$_closure5.doCall(AnnotationEditorController.groovy:1143) at org.bbop.apollo.AnnotationEditorController$do_annotationEditor_closure4.doCall(AnnotationEditorController.groovy:1139) at groovyx.gpars.group.PGroup$3.call(PGroup.java:289) at groovyx.gpars.group.PGroup$4.run(PGroup.java:313) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: Cannot get property 'uniqueName' on null object at org.bbop.apollo.RequestHandlingService.$ttdeleteFeature(RequestHandlingService.groovy:1761) ... 8 more 2017-03-16 16:55:42,763 [clientInboundChannel-2] ERROR springwebsocket.GrailsSimpAnnotationMethodMessageHandler - Unhandled exception java.lang.reflect.InvocationTargetException at org.bbop.apollo.AnnotationEditorController$do_annotationEditor_closure4$_closure5$_closure6.doCall(AnnotationEditorController.groovy:1144) at org.bbop.apollo.AnnotationEditorController$do_annotationEditor_closure4$_closure5.doCall(AnnotationEditorController.groovy:1143) at org.bbop.apollo.AnnotationEditorController$do_annotationEditor_closure4.doCall(AnnotationEditorController.groovy:1139) at groovyx.gpars.group.PGroup$3.call(PGroup.java:289) at groovyx.gpars.group.PGroup$4.run(PGroup.java:313) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: Cannot get property 'uniqueName' on null object at org.bbop.apollo.RequestHandlingService.$ttdeleteFeature(RequestHandlingService.groovy:1761) ... 8 more 2017-03-16 16:55:46,875 [clientInboundChannel-3] ERROR springwebsocket.GrailsSimpAnnotationMethodMessageHandler - Unhandled exception java.lang.reflect.InvocationTargetException at org.bbop.apollo.AnnotationEditorController$do_annotationEditor_closure4$_closure5$_closure6.doCall(AnnotationEditorController.groovy:1144) at org.bbop.apollo.AnnotationEditorController$do_annotationEditor_closure4$_closure5.doCall(AnnotationEditorController.groovy:1143) at org.bbop.apollo.AnnotationEditorController$do_annotationEditor_closure4.doCall(AnnotationEditorController.groovy:1139) at groovyx.gpars.group.PGroup$3.call(PGroup.java:289) at groovyx.gpars.group.PGroup$4.run(PGroup.java:313) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException: Cannot get property 'uniqueName' on null object at org.bbop.apollo.RequestHandlingService.$ttdeleteFeature(RequestHandlingService.groovy:1761) ... 8 more Comparing c_jejuni to names 2017-03-16 16:56:23,778 [http-bio-8080-exec-50] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/names/root.json Comparing h_pylori to names 2017-03-16 16:56:42,689 [http-bio-8080-exec-43] WARN apollo.JbrowseController - File not found: /opt/apollo/data/h_pylori/names/root.json 2017-03-16 16:56:43,978 [http-bio-8080-exec-4] ERROR errors.GrailsExceptionResolver - AnnotationException occurred when processing request: [POST] /apollo/49619322316529713481197571651/AnnotationEditorService No sequence found for name 'NC_000915.1' and organism 'Campylobacter jejuni'. Stacktrace follows: org.bbop.apollo.AnnotationException: No sequence found for name 'NC_000915.1' and organism 'Campylobacter jejuni' at org.bbop.apollo.PermissionService.$tt__checkPermissions(PermissionService.groovy:327) at org.bbop.apollo.AnnotationEditorController._do_getSequenceAlterations(AnnotationEditorController.groovy:475) at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198) at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63) at org.grails.plugins.web.rest.api.ControllersRestApi.forward(ControllersRestApi.groovy:53) at org.bbop.apollo.AnnotationEditorController.handleOperation(AnnotationEditorController.groovy:63) at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198) at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63) at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Comparing c_jejuni to names 2017-03-16 16:56:45,272 [http-bio-8080-exec-43] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/names/root.json 2017-03-16 16:57:09,943 [http-bio-8080-exec-51] INFO apollo.UserController - updateTrackListPreference 2017-03-16 16:57:09,944 [http-bio-8080-exec-51] INFO apollo.UserController - Added userOrganismPreference false Comparing h_pylori to names 2017-03-16 16:57:10,929 [http-bio-8080-exec-4] WARN apollo.JbrowseController - File not found: /opt/apollo/data/h_pylori/names/root.json 2017-03-16 16:57:47,843 [http-bio-8080-exec-43] INFO apollo.UserController - updateTrackListPreference 2017-03-16 16:57:47,844 [http-bio-8080-exec-43] INFO apollo.UserController - Added userOrganismPreference false 2017-03-16 16:57:47,847 [http-bio-8080-exec-51] ERROR errors.GrailsExceptionResolver - StaleObjectStateException occurred when processing request: [GET] /apollo/annotator/getAppState Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect) : [org.bbop.apollo.UserOrganismPreference#15]. Stacktrace follows: org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect) : [org.bbop.apollo.UserOrganismPreference#15] at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198) at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63) at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Comparing c_jejuni to names 2017-03-16 16:57:48,697 [http-bio-8080-exec-54] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/names/root.json 2017-03-16 17:01:52,131 [http-bio-8080-exec-53] INFO apollo.UserController - updateTrackListPreference 2017-03-16 17:01:52,132 [http-bio-8080-exec-53] INFO apollo.UserController - Added userOrganismPreference false 2017-03-16 17:01:52,136 [http-bio-8080-exec-50] ERROR errors.GrailsExceptionResolver - StaleObjectStateException occurred when processing request: [GET] /apollo/annotator/getAppState Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect) : [org.bbop.apollo.UserOrganismPreference#16]. Stacktrace follows: org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect) : [org.bbop.apollo.UserOrganismPreference#16] at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198) at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63) at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Comparing c_jejuni to names 2017-03-16 17:01:53,061 [http-bio-8080-exec-50] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/names/root.json 2017-03-16 17:02:18,703 [http-bio-8080-exec-49] WARN apollo.PreferenceService - Multiple preferences found: 2 2017-03-16 18:13:55,961 [http-bio-8080-exec-38] WARN apollo.PreferenceService - Multiple preferences found: 2 2017-03-16 18:16:01,082 [http-bio-8080-exec-46] WARN apollo.PreferenceService - Multiple preferences found: 2 2017-03-16 18:16:06,729 [http-bio-8080-exec-46] WARN apollo.PreferenceService - Multiple preferences found: 2 2017-03-16 18:16:07,843 [http-bio-8080-exec-46] WARN apollo.PreferenceService - Multiple preferences found: 2 2017-03-16 18:16:16,065 [http-bio-8080-exec-38] WARN apollo.PreferenceService - Multiple preferences found: 2 2017-03-16 18:16:21,638 [http-bio-8080-exec-38] WARN apollo.PreferenceService - Multiple preferences found: 2 2017-03-16 18:16:23,277 [http-bio-8080-exec-38] WARN apollo.PreferenceService - Multiple preferences found: 2 2017-03-16 18:18:15,638 [http-bio-8080-exec-53] WARN apollo.PreferenceService - Multiple preferences found: 2 2017-03-16 18:18:20,981 [http-bio-8080-exec-53] WARN apollo.PreferenceService - Multiple preferences found: 2 Comparing c_jejuni to names 2017-03-16 18:18:26,231 [http-bio-8080-exec-53] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/names/root.json Comparing h_pylori to names 2017-03-16 18:18:30,790 [http-bio-8080-exec-47] WARN apollo.JbrowseController - File not found: /opt/apollo/data/h_pylori/names/root.json 2017-03-16 18:18:38,283 [http-bio-8080-exec-47] INFO apollo.UserController - updateTrackListPreference 2017-03-16 18:18:38,284 [http-bio-8080-exec-47] INFO apollo.UserController - Added userOrganismPreference false Comparing h_pylori to names 2017-03-16 18:18:39,257 [http-bio-8080-exec-47] WARN apollo.JbrowseController - File not found: /opt/apollo/data/h_pylori/names/root.json 2017-03-17 09:56:10,318 [http-bio-8080-exec-46] ERROR authenticator.UsernamePasswordAuthenticatorService - Problem authenticating: java.lang.IllegalArgumentException: Method argumet (authentication token) cannot be null. 2017-03-17 09:56:10,318 [http-bio-8080-exec-46] WARN apollo.PermissionService - Failed to authenticate user 2017-03-17 09:56:10,318 [http-bio-8080-exec-46] ERROR apollo.UserController - Failed to authenticate 2017-03-17 10:28:27,743 [DefaultPluginScheduler_Worker-8] ERROR apollo.PreferenceService - Removing stale preferences 2017-03-17 10:28:27,753 [DefaultPluginScheduler_Worker-8] ERROR apollo.PreferenceService - Removed 1 stale preferences 2017-03-17 13:39:31,175 [http-bio-8080-exec-47] ERROR apollo.SequenceController - Sequence name is invalid NC_000915.1 2017-03-17 13:39:32,468 [http-bio-8080-exec-47] ERROR apollo.SequenceController - Sequence name is invalid NC_000915.1 2017-03-17 13:39:37,093 [http-bio-8080-exec-49] ERROR apollo.SequenceController - Sequence name is invalid NC_000915.1 2017-03-17 13:39:38,404 [http-bio-8080-exec-49] ERROR apollo.SequenceController - Sequence name is invalid NC_000915.1 2017-03-17 13:39:39,729 [http-bio-8080-exec-38] ERROR apollo.SequenceController - Sequence name is invalid NC_000915.1 2017-03-17 13:39:41,264 [http-bio-8080-exec-46] ERROR apollo.SequenceController - Sequence name is invalid NC_000915.1 Comparing c_jejuni to seq 2017-03-17 13:39:42,563 [http-bio-8080-exec-49] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt Comparing c_jejuni to seq 2017-03-17 13:39:42,563 [http-bio-8080-exec-43] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt Comparing c_jejuni to seq 2017-03-17 13:39:42,564 [http-bio-8080-exec-38] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt Comparing c_jejuni to seq 2017-03-17 13:39:42,565 [http-bio-8080-exec-46] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt Comparing c_jejuni to seq 2017-03-17 13:39:42,565 [http-bio-8080-exec-49] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt Comparing c_jejuni to seq 2017-03-17 13:39:42,567 [http-bio-8080-exec-43] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt Comparing c_jejuni to seq 2017-03-17 13:39:42,568 [http-bio-8080-exec-38] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt Comparing c_jejuni to seq 2017-03-17 13:39:42,569 [http-bio-8080-exec-54] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt Comparing c_jejuni to seq 2017-03-17 13:39:42,570 [http-bio-8080-exec-49] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt Comparing c_jejuni to seq 2017-03-17 13:39:42,572 [http-bio-8080-exec-43] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt 2017-03-17 13:39:42,602 [http-bio-8080-exec-38] ERROR apollo.SequenceController - Sequence name is invalid NC_000915.1 2017-03-17 13:39:45,980 [http-bio-8080-exec-54] ERROR apollo.SequenceController - Sequence name is invalid NC_000915.1 Comparing c_jejuni to seq 2017-03-17 13:39:45,987 [http-bio-8080-exec-49] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt 2017-03-17 13:39:47,787 [http-bio-8080-exec-43] ERROR apollo.SequenceController - Sequence name is invalid NC_000915.1 Comparing c_jejuni to seq Comparing c_jejuni to seq 2017-03-17 13:39:47,799 [http-bio-8080-exec-38] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt 2017-03-17 13:39:47,799 [http-bio-8080-exec-54] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt Comparing c_jejuni to seq 2017-03-17 13:39:47,799 [http-bio-8080-exec-49] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt 2017-03-17 13:39:48,240 [http-bio-8080-exec-43] ERROR apollo.SequenceController - Sequence name is invalid NC_000915.1 2017-03-17 13:39:48,588 [http-bio-8080-exec-38] ERROR apollo.SequenceController - Sequence name is invalid NC_000915.1 2017-03-17 13:39:58,668 [http-bio-8080-exec-49] ERROR apollo.SequenceController - Sequence name is invalid NC_000915.1 Comparing c_jejuni to seq 2017-03-17 13:39:58,679 [http-bio-8080-exec-54] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt Comparing c_jejuni to seq 2017-03-17 13:39:58,680 [http-bio-8080-exec-43] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt 2017-03-17 13:39:59,132 [http-bio-8080-exec-38] ERROR apollo.SequenceController - Sequence name is invalid NC_000915.1 Comparing c_jejuni to seq 2017-03-17 13:39:59,143 [http-bio-8080-exec-49] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt Comparing c_jejuni to seq 2017-03-17 13:39:59,143 [http-bio-8080-exec-54] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt Comparing c_jejuni to seq 2017-03-17 13:39:59,145 [http-bio-8080-exec-43] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt Comparing c_jejuni to seq 2017-03-17 13:39:59,146 [http-bio-8080-exec-38] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt 2017-03-17 13:39:59,589 [http-bio-8080-exec-46] ERROR apollo.SequenceController - Sequence name is invalid NC_000915.1 Comparing c_jejuni to seq 2017-03-17 13:39:59,600 [http-bio-8080-exec-4] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt Comparing c_jejuni to seq 2017-03-17 13:39:59,601 [http-bio-8080-exec-38] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt Comparing c_jejuni to seq 2017-03-17 13:39:59,601 [http-bio-8080-exec-43] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt 2017-03-17 13:40:00,076 [http-bio-8080-exec-46] ERROR apollo.SequenceController - Sequence name is invalid NC_000915.1 Comparing c_jejuni to seq 2017-03-17 13:40:00,089 [http-bio-8080-exec-4] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt Comparing c_jejuni to seq 2017-03-17 13:40:00,090 [http-bio-8080-exec-38] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt Comparing c_jejuni to seq 2017-03-17 13:40:00,092 [http-bio-8080-exec-43] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/seq/82d/42b/dd/NC_000915.1-6.txt Comparing c_jejuni to names 2017-03-17 13:40:08,779 [http-bio-8080-exec-43] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/names/root.json Comparing s_enterica to names 2017-03-17 13:43:21,757 [http-bio-8080-exec-54] WARN apollo.JbrowseController - File not found: /opt/apollo/data/s_enterica/names/root.json 2017-03-17 13:43:45,600 [http-bio-8080-exec-56] INFO apollo.UserController - updateTrackListPreference 2017-03-17 13:43:45,601 [http-bio-8080-exec-56] INFO apollo.UserController - Added userOrganismPreference false Comparing c_jejuni to names 2017-03-17 13:43:46,541 [http-bio-8080-exec-56] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/names/root.json Comparing c_jejuni to tracks 2017-03-17 13:44:24,543 [http-bio-8080-exec-57] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/tracks/Genes/NC_002163.1/trackData.json Comparing c_jejuni to tracks 2017-03-17 13:44:25,196 [http-bio-8080-exec-57] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/tracks/Transcript/NC_002163.1/trackData.json Comparing c_jejuni to tracks 2017-03-17 13:44:26,769 [http-bio-8080-exec-57] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/tracks/CDS/NC_002163.1/trackData.json Comparing c_jejuni to tracks 2017-03-17 13:44:27,479 [http-bio-8080-exec-57] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/tracks/rRNA/NC_002163.1/trackData.json

Also attaching the whole readout catalinaout.txt

mastiles commented 7 years ago

Looks like the issue is here?

2017-03-16 18:18:26,231 [http-bio-8080-exec-53] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/names/root.json

nathandunn commented 7 years ago

@mastiles Can you do this:

1 - open up your javascript / developer console 2 - on the server to tail -f pathto/catalina.out. This will just show a reaction on the server to your search command. 3 - perform a search

Provide the following information:

mastiles commented 7 years ago

I assume the developer console is the terminal? If so, I accidentally closed the terminal I was working on, so can't give you the readout from that

beebylab@vroomfondel /var/log/tomcat7 $ tail -f catalina.out Comparing c_jejuni to tracks 2017-03-17 13:44:24,543 [http-bio-8080-exec-57] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/tracks/Genes/NC_002163.1/trackData.json Comparing c_jejuni to tracks 2017-03-17 13:44:25,196 [http-bio-8080-exec-57] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/tracks/Transcript/NC_002163.1/trackData.json Comparing c_jejuni to tracks 2017-03-17 13:44:26,769 [http-bio-8080-exec-57] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/tracks/CDS/NC_002163.1/trackData.json Comparing c_jejuni to tracks 2017-03-17 13:44:27,479 [http-bio-8080-exec-57] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/tracks/rRNA/NC_002163.1/trackData.json 2017-03-17 15:20:37,874 [http-bio-8080-exec-56] ERROR apollo.FeatureEventController - [controller:featureEvent, format:null, action:report] after date Sat Mar 22 15:20:37 GMT 1997

Nothing seemed to be written after I performed the search...

Comparing c_jejuni to tracks 2017-03-17 13:44:24,543 [http-bio-8080-exec-57] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/tracks/Genes/NC_002163.1/trackData.json Comparing c_jejuni to tracks 2017-03-17 13:44:25,196 [http-bio-8080-exec-57] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/tracks/Transcript/NC_002163.1/trackData.json Comparing c_jejuni to tracks 2017-03-17 13:44:26,769 [http-bio-8080-exec-57] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/tracks/CDS/NC_002163.1/trackData.json Comparing c_jejuni to tracks 2017-03-17 13:44:27,479 [http-bio-8080-exec-57] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/tracks/rRNA/NC_002163.1/trackData.json 2017-03-17 15:20:37,874 [http-bio-8080-exec-56] ERROR apollo.FeatureEventController - [controller:featureEvent, format:null, action:report] after date Sat Mar 22 15:20:37 GMT 1997

screenshot at 2017-03-17 15 47 09

Thank you for your continued patience!

mastiles commented 7 years ago

An update: moving to a new organism and searching shows:

beebylab@vroomfondel /var/log/tomcat7 $ tail -f catalina.out Comparing c_jejuni to tracks 2017-03-17 13:44:24,543 [http-bio-8080-exec-57] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/tracks/Genes/NC_002163.1/trackData.json Comparing c_jejuni to tracks 2017-03-17 13:44:25,196 [http-bio-8080-exec-57] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/tracks/Transcript/NC_002163.1/trackData.json Comparing c_jejuni to tracks 2017-03-17 13:44:26,769 [http-bio-8080-exec-57] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/tracks/CDS/NC_002163.1/trackData.json Comparing c_jejuni to tracks 2017-03-17 13:44:27,479 [http-bio-8080-exec-57] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/tracks/rRNA/NC_002163.1/trackData.json 2017-03-17 15:20:37,874 [http-bio-8080-exec-56] ERROR apollo.FeatureEventController - [controller:featureEvent, format:null, action:report] after date Sat Mar 22 15:20:37 GMT 1997 2017-03-17 15:57:01,724 [http-bio-8080-exec-59] INFO apollo.UserController - updateTrackListPreference 2017-03-17 15:57:01,725 [http-bio-8080-exec-59] INFO apollo.UserController - Added userOrganismPreference false Comparing h_pylori to names 2017-03-17 15:57:02,721 [http-bio-8080-exec-55] WARN apollo.JbrowseController - File not found: /opt/apollo/data/h_pylori/names/root.json 2017-03-17 15:57:19,082 [http-bio-8080-exec-54] INFO apollo.UserController - updateTrackListPreference 2017-03-17 15:57:19,083 [http-bio-8080-exec-54] INFO apollo.UserController - Added userOrganismPreference false Comparing s_enterica to names 2017-03-17 15:57:19,952 [http-bio-8080-exec-49] WARN apollo.JbrowseController - File not found: /opt/apollo/data/s_enterica/names/root.json

nathandunn commented 7 years ago

In you apollo-config.groovy can you add:

log4j.main = { debug "grails.app" } And redeploy. You should see stuff like this when you do the search:

2017-03-17 08:56:18,914 [http-bio-8080-exec-18] DEBUG apollo.SequenceService - input at getSequenceForFeature: [features:[[uniquename:cbf81b68-6acc-482e-81a8-748dbfe9626b]], clientToken:14804977910930635361849398913, track:Group1.1, type:genomic, operation:get_sequence, username:ndunn@me.com] 2017-03-17 08:56:24,781 [http-bio-8080-exec-18] DEBUG apollo.AnnotationEditorController - handleOperation annotationEditor get_sequence_search_tools -> getSequenceSearchTools 2017-03-17 08:56:24,782 [http-bio-8080-exec-18] DEBUG apollo.AnnotationEditorController - getSequenceSearchTools {"track":"Group1.1","operation":"get_sequence_search_tools"} 2017-03-17 08:56:26,856 [http-bio-8080-exec-3] DEBUG apollo.AnnotationEditorController - handleOperation annotationEditor search_sequence -> searchSequence 2017-03-17 08:56:26,858 [http-bio-8080-exec-3] DEBUG apollo.AnnotationEditorController - sequenceSearch {"search":{"database_id":"Group1.1","residues":"GAAAGAGCTTCACCCCTCGAGTTTCCCCAACACGTTGAGGAAGAGTCCGCTTGGCTCGTCCATGGGCGCTATCAATCACACGAGGAAAGATTTTTACGCGCCCACGTTGACCGGTTCGCCGCTGCGACGAGATGCCATTGGTAGGTCGATGGGGTGCCTGAAGAAGGACACCGCGTCGAGCTACAATTCTCCGCTGAGGCGCGATTACGTGGCGAAATCGATGACCAGCCTGCAGAGGACGATGCCTCCGCGGCGAGACGCTAATAATTTGTCCAGCCTGTTCCCGACGTCGCACAACGATTCCTCGAGACCCGTCGGCCTCGTCTCCTCCTCGCCCTCGAACAACGATCCGAAGACGAGAGGCCCGGCGAACTTGAAAAGCTGTCTGG","key":"blat_nuc"},"track":"Group1.1","operation":"search_sequence"}

Also, you can just re-open the console before you perform another search. If you forget to, just open it and run the search again.

Nathan

On Mar 17, 2017, at 8:51 AM, mastiles notifications@github.com wrote:

I assume the developer console is the terminal? If so, I accidentally closed the terminal I was working on, so can't give you the readout from that

beebylab@vroomfondel /var/log/tomcat7 $ tail -f catalina.out Comparing c_jejuni to tracks 2017-03-17 13:44:24,543 [http-bio-8080-exec-57] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/tracks/Genes/NC_002163.1/trackData.json Comparing c_jejuni to tracks 2017-03-17 13:44:25,196 [http-bio-8080-exec-57] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/tracks/Transcript/NC_002163.1/trackData.json Comparing c_jejuni to tracks 2017-03-17 13:44:26,769 [http-bio-8080-exec-57] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/tracks/CDS/NC_002163.1/trackData.json Comparing c_jejuni to tracks 2017-03-17 13:44:27,479 [http-bio-8080-exec-57] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/tracks/rRNA/NC_002163.1/trackData.json 2017-03-17 15:20:37,874 [http-bio-8080-exec-56] ERROR apollo.FeatureEventController - [controller:featureEvent, format:null, action:report] after date Sat Mar 22 15:20:37 GMT 1997

Nothing seemed to be written after I performed the search...

Comparing c_jejuni to tracks 2017-03-17 13:44:24,543 [http-bio-8080-exec-57] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/tracks/Genes/NC_002163.1/trackData.json Comparing c_jejuni to tracks 2017-03-17 13:44:25,196 [http-bio-8080-exec-57] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/tracks/Transcript/NC_002163.1/trackData.json Comparing c_jejuni to tracks 2017-03-17 13:44:26,769 [http-bio-8080-exec-57] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/tracks/CDS/NC_002163.1/trackData.json Comparing c_jejuni to tracks 2017-03-17 13:44:27,479 [http-bio-8080-exec-57] WARN apollo.JbrowseController - File not found: /opt/apollo/data/c_jejuni/tracks/rRNA/NC_002163.1/trackData.json 2017-03-17 15:20:37,874 [http-bio-8080-exec-56] ERROR apollo.FeatureEventController - [controller:featureEvent, format:null, action:report] after date Sat Mar 22 15:20:37 GMT 1997

https://cloud.githubusercontent.com/assets/26278942/24051146/154cfd06-0b29-11e7-8f20-846b8b217c38.png Thank you for your continued patience!

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/GMOD/Apollo/issues/1528#issuecomment-287392656, or mute the thread https://github.com/notifications/unsubscribe-auth/AAt2qtLgsWQtl6-X3DX3SW1bL2sCaX6pks5rmqwMgaJpZM4Md__w.

mastiles commented 7 years ago

That makes sense, nothing was coming up when I actually did the search.

I think I already have that in my apollo-config.groovy though...

environments {
    development {
        // sample config to turn on debug logging in development e.g. for apollo run-local
        log4j.main = {
            debug "grails.app"
        }
        // sample config to edit apollo specific configs in development mode
        apollo {
            gff3.source = "testing"
        }
        dataSource{
            dbCreate = "update" // one of 'create', 'create-drop', 'update', 'validate', ''
            username = "apollo"
            password = "password"
            driverClassName = "com.mysql.jdbc.Driver"
            dialect = org.hibernate.dialect.MySQL5InnoDBDialect
            url = "jdbc:mysql://localhost/apollo"
        }
    }
    test {
        dataSource{
            dbCreate = "create-drop" // one of 'create', 'create-drop', 'update', 'validate', ''
            username = "apollo"
            password = "password"
            driverClassName = "com.mysql.jdbc.Driver"
            dialect = org.hibernate.dialect.MySQL5InnoDBDialect
            url = "jdbc:mysql://localhost/apollo-test"
        }
    }
    production {
        dataSource{
            dbCreate = "update" // one of 'create', 'create-drop', 'update', 'validate', ''
            username = "apollo"
            password = "password"
            driverClassName = "com.mysql.jdbc.Driver"
            dialect = org.hibernate.dialect.MySQL5InnoDBDialect
            url = "jdbc:mysql://localhost/apollo-production"
            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = true
                initialSize = 5
                maxActive = 50
                minIdle = 5
                maxIdle = 25
                maxWait = 10000
                maxAge = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis = 60000
                validationQuery = "SELECT 1"
                validationQueryTimeout = 3
                validationInterval = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}

// Uncomment to make changes
//
//jbrowse {
//    git {
//        url= "https://github.com/GMOD/jbrowse"
//        tag = "1.12.2-apollo"
////        branch = "master"
//        alwaysPull = true
//        alwaysRecheck = true
//    }
//    plugins {
//        WebApollo{
//            included = true
//        }
//        NeatHTMLFeatures{
//            included = true
//        }
//        NeatCanvasFeatures{
//            included = true
//        }
//        RegexSequenceSearch{
//            included = true
//        }
//        HideTrackLabels{
//            included = true
//        }
////        MyVariantInfo {
////            git = 'https://github.com/GMOD/myvariantviewer'
////            branch = 'master'
////            alwaysRecheck = "true"
////            alwaysPull = "true"
////        }
////        SashimiPlot {
////            git = 'https://github.com/cmdcolin/sashimiplot'
////            branch = 'master'
////            alwaysPull = "true"
////        }
//    }
//}
nathandunn commented 7 years ago

Are you running this in dev mode (i.e., ./apollo run-local)?

If you’re using tomcat, you’ll need to add the

log4j.main = { debug "grails.app" }

to the production block.

Also, do you want me to remove your password (this is a public server)?

On Mar 17, 2017, at 9:06 AM, mastiles notifications@github.com wrote:

That makes sense, nothing was coming up when I actually did the search.

I think I already have that in my apollo-config.groovy though...

environments { development { // sample config to turn on debug logging in development e.g. for apollo run-local log4j.main = { debug "grails.app" } // sample config to edit apollo specific configs in development mode apollo { gff3.source = "testing" } dataSource{ dbCreate = "update" // one of 'create', 'create-drop', 'update', 'validate', '' username = "apollo" password = "tomography" driverClassName = "com.mysql.jdbc.Driver" dialect = org.hibernate.dialect.MySQL5InnoDBDialect url = "jdbc:mysql://localhost/apollo" } } test { dataSource{ dbCreate = "create-drop" // one of 'create', 'create-drop', 'update', 'validate', '' username = "apollo" password = "tomography" driverClassName = "com.mysql.jdbc.Driver" dialect = org.hibernate.dialect.MySQL5InnoDBDialect url = "jdbc:mysql://localhost/apollo-test" } } production { dataSource{ dbCreate = "update" // one of 'create', 'create-drop', 'update', 'validate', '' username = "apollo" password = "tomography" driverClassName = "com.mysql.jdbc.Driver" dialect = org.hibernate.dialect.MySQL5InnoDBDialect url = "jdbc:mysql://localhost/apollo-production" properties { // See http://grails.org/doc/latest/guide/conf.html#dataSource http://grails.org/doc/latest/guide/conf.html#dataSource for documentation jmxEnabled = true initialSize = 5 maxActive = 50 minIdle = 5 maxIdle = 25 maxWait = 10000 maxAge = 10 * 60000 timeBetweenEvictionRunsMillis = 5000 minEvictableIdleTimeMillis = 60000 validationQuery = "SELECT 1" validationQueryTimeout = 3 validationInterval = 15000 testOnBorrow = true testWhileIdle = true testOnReturn = false jdbcInterceptors = "ConnectionState" defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED } } } }

// Uncomment to make changes // //jbrowse { // git { // url= "https://github.com/GMOD/jbrowse https://github.com/GMOD/jbrowse" // tag = "1.12.2-apollo" //// branch = "master" // alwaysPull = true // alwaysRecheck = true // } // plugins { // WebApollo{ // included = true // } // NeatHTMLFeatures{ // included = true // } // NeatCanvasFeatures{ // included = true // } // RegexSequenceSearch{ // included = true // } // HideTrackLabels{ // included = true // } //// MyVariantInfo { //// git = 'https://github.com/GMOD/myvariantviewer https://github.com/GMOD/myvariantviewer' //// branch = 'master' //// alwaysRecheck = "true" //// alwaysPull = "true" //// } //// SashimiPlot { //// git = 'https://github.com/cmdcolin/sashimiplot https://github.com/cmdcolin/sashimiplot' //// branch = 'master' //// alwaysPull = "true" //// } // } //}

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/GMOD/Apollo/issues/1528#issuecomment-287396956, or mute the thread https://github.com/notifications/unsubscribe-auth/AAt2qoRw91pl3Wy_j61kq2I88Va1HqHHks5rmq9pgaJpZM4Md__w.

mastiles commented 7 years ago

I'm using tomcat, so running it from the war file. What is the production block?

Yes please, that was really stupid of me - just edited it out, but if you can do something more stable that would be much appreciated.

nathandunn commented 7 years ago

If you look below, you can see "development", "test", and "production". I added the log4j.main to production below to demonstrate. 

environments {
    development {
        // sample config to turn on debug logging in development e.g. for apollo run-local
        log4j.main = {
            debug "grails.app"
        }
        // sample config to edit apollo specific configs in development mode
        apollo {
            gff3.source = "testing"
        }
        dataSource{
            dbCreate = "update" // one of 'create', 'create-drop', 'update', 'validate', ''
            username = "apollo"
            password = "password"
            driverClassName = "com.mysql.jdbc.Driver"
            dialect = org.hibernate.dialect.MySQL5InnoDBDialect
            url = "jdbc:mysql://localhost/apollo"
        }
    }
    test {
        dataSource{
            dbCreate = "create-drop" // one of 'create', 'create-drop', 'update', 'validate', ''
            username = "apollo"
            password = "password"
            driverClassName = "com.mysql.jdbc.Driver"
            dialect = org.hibernate.dialect.MySQL5InnoDBDialect
            url = "jdbc:mysql://localhost/apollo-test"
        }
    }
    production {
        log4j.main = {
            debug "grails.app"
        }
        dataSource{
            dbCreate = "update" // one of 'create', 'create-drop', 'update', 'validate', ''
            username = "apollo"
            password = "password"
            driverClassName = "com.mysql.jdbc.Driver"
            dialect = org.hibernate.dialect.MySQL5InnoDBDialect
            url = "jdbc:mysql://localhost/apollo-production"
            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = true
                initialSize = 5
                maxActive = 50
                minIdle = 5
                maxIdle = 25
                maxWait = 10000
                maxAge = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis = 60000
                validationQuery = "SELECT 1"
                validationQueryTimeout = 3
                validationInterval = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}
mastiles commented 7 years ago

I did it and the catalina. log is looking better (thank you!)

tail -f catalina.out 2017-03-17 16:36:14,621 [http-bio-8080-exec-60] DEBUG apollo.JbrowseController - got organism org.bbop.apollo.Organism : 8 for client token 49619322316529713481197571651 2017-03-17 16:36:14,621 [http-bio-8080-exec-60] DEBUG apollo.JbrowseController - data directory: /opt/apollo/data/c_jejuni 2017-03-17 16:36:14,622 [http-bio-8080-exec-62] DEBUG apollo.JbrowseController - got organism org.bbop.apollo.Organism : 8 for client token 49619322316529713481197571651 2017-03-17 16:36:14,622 [http-bio-8080-exec-62] DEBUG apollo.JbrowseController - data directory: /opt/apollo/data/c_jejuni 2017-03-17 16:36:14,715 [http-bio-8080-exec-63] DEBUG apollo.AnnotationEditorController - handleOperation annotationEditor get_translation_table -> getTranslationTable 2017-03-17 16:36:14,716 [http-bio-8080-exec-63] DEBUG apollo.AnnotationEditorController - getTranslationTable 2017-03-17 16:36:14,716 [http-bio-8080-exec-63] DEBUG apollo.PermissionService - dataObject does not contain organism (may not be needed) 2017-03-17 16:36:14,967 [http-bio-8080-exec-65] DEBUG apollo.AnnotationEditorController - handleOperation annotationEditor get_sequence_alterations -> getSequenceAlterations 2017-03-17 16:36:14,968 [http-bio-8080-exec-65] DEBUG apollo.PermissionService - dataObject does not contain organism (may not be needed) 2017-03-17 16:37:12,829 [http-bio-8080-exec-62] DEBUG apollo.AnnotatorController - permissions checked and alive 2017-03-17 16:37:59,685 [http-bio-8080-exec-63] DEBUG apollo.AnnotationEditorController - handleOperation annotationEditor get_sequence_search_tools -> getSequenceSearchTools 2017-03-17 16:37:59,687 [http-bio-8080-exec-63] DEBUG apollo.AnnotationEditorController - getSequenceSearchTools {"track":"NC_002163.1","operation":"get_sequence_search_tools"} 2017-03-17 16:38:11,641 [http-bio-8080-exec-66] DEBUG apollo.AnnotationEditorController - handleOperation annotationEditor search_sequence -> searchSequence 2017-03-17 16:38:11,643 [http-bio-8080-exec-66] DEBUG apollo.AnnotationEditorController - sequenceSearch {"search":{"database_id":"NC_002163.1","residues":"ATGAATCCAAGCCAAATACTTGAAAATTTAAAAAAAGAATTAAGTGAAAACGAATACGAAAACTATTTATCAAATTTAAAATTCAACGAAAAACAAAGCAAAGCAGATCTTTTAGTTTTTAATGCTCCAAATGAACTCAT","key":"blat_nuc"},"track":"NC_002163.1","operation":"search_sequence"} 2017-03-17 16:38:11,643 [http-bio-8080-exec-66] DEBUG apollo.PermissionService - dataObject does not contain organism (may not be needed) 2017-03-17 16:38:11,656 [http-bio-8080-exec-66] DEBUG apollo.PreferenceService - PS: getCurrentOrganismForCurrentUser 49619322316529713481197571651 2017-03-17 16:38:11,660 [http-bio-8080-exec-66] DEBUG apollo.AnnotationEditorController - Organism to string: {"id":8,"commonName":"Campylobacter jejuni","genus":"Campylobacter","species":"jejuni","directory":"/opt/apollo/data/c_jejuni"} 2017-03-17 16:38:12,944 [http-bio-8080-exec-67] DEBUG apollo.AnnotatorController - permissions checked and alive 2017-03-17 16:38:26,923 [http-bio-8080-exec-61] DEBUG apollo.AnnotationEditorController - handleOperation annotationEditor get_sequence_search_tools -> getSequenceSearchTools 2017-03-17 16:38:26,924 [http-bio-8080-exec-61] DEBUG apollo.AnnotationEditorController - getSequenceSearchTools {"track":"NC_002163.1","operation":"get_sequence_search_tools"} 2017-03-17 16:38:29,887 [http-bio-8080-exec-61] DEBUG apollo.AnnotationEditorController - handleOperation annotationEditor search_sequence -> searchSequence 2017-03-17 16:38:29,887 [http-bio-8080-exec-61] DEBUG apollo.AnnotationEditorController - sequenceSearch {"search":{"database_id":"NC_002163.1","residues":"ATGAATCCAAGCCAAATACTTGAAAATTTAAAAAAAGAATTAAGTGAAAACGAATACGAAAACTATTTATCAAATTTAAAATTCAACGAAAAACAAAGCAAAGCAGATCTTTTAGTTTTTAATGCTCCAAATGAACTCAT","key":"blat_nuc"},"track":"NC_002163.1","operation":"search_sequence"} 2017-03-17 16:38:29,888 [http-bio-8080-exec-61] DEBUG apollo.PermissionService - dataObject does not contain organism (may not be needed) 2017-03-17 16:38:29,897 [http-bio-8080-exec-61] DEBUG apollo.PreferenceService - PS: getCurrentOrganismForCurrentUser 49619322316529713481197571651 2017-03-17 16:38:29,900 [http-bio-8080-exec-61] DEBUG apollo.AnnotationEditorController - Organism to string: {"id":8,"commonName":"Campylobacter jejuni","genus":"Campylobacter","species":"jejuni","directory":"/opt/apollo/data/c_jejuni"} 2017-03-17 16:39:12,960 [http-bio-8080-exec-65] DEBUG apollo.AnnotatorController - permissions checked and alive

But it still isn't finding a match - I'm using a sequence straight from the reference Campylobacter sequence, so it's not a matter of identity being too low...

nathandunn commented 7 years ago

Hmm, yeah, no error. If you drag up an annotation and then right-click on it to do "getSequence" (and get the nucleotide sequence) and enter it into the search, what do you see?

Also, you might try the blat function directly on the command-line to see if maybe you indexed the fasta incorrectly.

mastiles commented 7 years ago

I did that, and still no recognition.

What do you mean by indexed the fasta incorrectly?

I made a new fasta file with a section of the genome in it, and when I blat search on the command line:

beebylab@vroomfondel ~/Apollo-2.0.6/data $ blat -stepSize=5 -repMatch=2253 -minScore=0 -minIdentity=60 twoBit/c_jejuni.2bit c_jejuni_section.fasta c_jejuni.psl Loaded 1641481 letters in 1 sequences Searched 210 bases in 1 sequences

nathandunn commented 7 years ago

So, your output .psl file has the expected results?

mastiles commented 7 years ago

Yeah, the formatting is a bit odd but

psLayout version 3

match mis- rep. N's Q gap Q gap T gap T gap strand Q Q Q Q T T T T block blockSizes qStarts tStarts match match count bases count bases name size start end name size start end count

210 0 0 0 0 0 0 0 + NC_002163.1 210 0 210 NC_002163.1 1641481 0 210 1 210, 0, 0, 23 0 0 0 1 2 1 1 + NC_002163.1 210 138 163 NC_002163.1 1641481 913629 913653 2 18,5, 138,158, 913629,913648, 22 0 0 0 0 0 0 0 - NC_002163.1 210 17 39 NC_002163.1 1641481 996613 996635 1 22, 171, 996613,

It has worked.

What does this mean for my database?

nathandunn commented 7 years ago

You'll want to index the entire sequence so that it correlates properly, internally (i.e., it points back to the correct version.

BLAT-specific questions would probably be better go to biostar or UCSC.

mastiles commented 7 years ago

BLAT itself seems to be working though?

Does this mean that there is an issue with the way Web Apollo is communicating to BLAT in this instance?

I’m not sure what to do with regards to indexing the sequence then - is that not done when I used the command sudo bin/prepare-refseqs.pl --fasta c_jejuni.fasta --out /opt/apollo/data/c_jejuni To prepare the refseq?

And what do you mean by point back to the correct version?

Sorry for multiple questions, and thank you so much for your continued help!

nathandunn commented 7 years ago

Blat seems to be working. You can open the "network" developer tab to see what it is sending and returning.

You had said that you were only using a portion of your file to create the 2bit file. I think this is very likely your problem.

What I send / return looks like this:

screen shot 2017-03-17 at 12 44 33 pm screen shot 2017-03-17 at 12 44 22 pm

mastiles commented 7 years ago

It certainly doesn't look like it's connecting to BLAT. I didn't use just part of the FASTA to create the 2bit in the actual database (just in the test of BLAT locally) so I don't think that is the issue. I've attached the screenshots of the network developer tab.

screenshot at 2017-03-20 11 02 38 screenshot at 2017-03-20 11 02 55

If it makes a difference I also don't seem to have a blat_config.xml file - tried using the find command with various additions and to no avail. Having said that, it seems that in the config_groovy file everything is pointing to the right place

nathandunn commented 7 years ago

I did some tests and was getting the same errors, then I realized that I actually wasn't pointing to the proper 2bit file in the organism tab. The command that gets executed by the system is this (for searching all volvox):

/usr/local/bin/blat /opt/apollo/volvox/raw/volvox/volvox.2bit /var/folders/yc/y5cfqt8j72165cyr68_bpw9r0000gn/T/blat_tmp4604678760948733505/query.fa /var/folders/yc/y5cfqt8j72165cyr68_bpw9r0000gn/T/blat_tmp4604678760948733505/results.tab -out=blast8

query.fa is the query of the form (with that header):

>query
attgtagatac

So, you should be able to run:

/usr/local/bin/blat /opt/apollo/volvox/raw/volvox/volvox.2bit query.fa results.tab -out=blast8

and verify that you see results are in results.tab. If there is an error, you will see that. Whatever you are using for your 2bit file needs to be an absolute path and readable by whatever is running tomcat. Worst-case just do chmod og+x volvox.2bit and it should be able to be run by anyone.

mastiles commented 7 years ago

I changed the permissions as you suggested and it's working! Thank you so much for your help! :)

nathandunn commented 7 years ago

Excellent. Whew!

mictadlo commented 5 years ago

Hi, I am not able to get the BLAT search running because I get the below error: image

Those are the steps how I set BLAT up:

wget -c http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/blat/blat
wget -c http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/faToTwoBit
sudo mv blat /usr/local/bin/
sudo mv faToTwoBit /usr/local/bin/
chmod +x /usr/local/bin/blat 
chmod +x /usr/local/bin/faToTwoBit 

/data/apollo$ faToTwoBit NbV1ChF.fasta NbV1ChF.2bit
/data/apollo$ chmod og+x NbV1ChF.2bit
/data/apollo$ ls -ahl
total 3.4G
drwxr-xr-x  4 ubuntu ubuntu 4.0K Feb 21 00:12 .
drwxr-xr-x  5 root   root   4.0K Feb 20 03:00 ..
-rw-rw-r--  1 ubuntu ubuntu  541 Feb 20 23:03 .htaccess
-rwxrwx---  1 ubuntu ubuntu  77M Feb 20 04:04 Gmp_NbV1_Final.gff3
-rw-rwxr-x  1 ubuntu ubuntu 655M Feb 21 00:12 NbV1ChF.2bit
-rwxrwx---  1 ubuntu ubuntu 2.6G Feb 20 03:59 NbV1ChF.fasta
-rwxrwx---  1 ubuntu ubuntu  670 Feb 20 03:59 NbV1ChF.fasta.fai
drwxrwxr-x 21 ubuntu ubuntu 4.0K Feb 20 22:59 seq
-rw-rw-r--  1 ubuntu ubuntu  863 Feb 20 23:03 trackList.json
drwxrwxr-x  3 ubuntu ubuntu 4.0K Feb 20 23:03 tracks
-rw-rw-r--  1 ubuntu ubuntu    0 Feb 20 22:58 tracks.conf

What did I miss?

Thank you in advance,

Michal

nathandunn commented 5 years ago

You have to add the path on the organism tab:

image

nathandunn commented 5 years ago

Maybe you delete the comment (that’s fine), I think you may have figured out that you need the full name to the file?

But everything else looked good. If it doesn’t work, let me know.

Nathan

On Feb 20, 2019, at 4:41 PM, Michał T. Lorenc notifications@github.com wrote:

P.S. I updated to the Search Database field /data/apollo but I got No match found. Next, I ran blat on command line I got hits:

blat NbV1ChF.2bit test.fasta test.results -out=blast8 Loaded 2742960125 letters in 19 sequences Searched 2598 bases in 1 sequences NbV1Ch01 NbV1Ch01 100.00 2598 0 0 1 2598 93537241 93539838 0.0e+00 5040.0 NbV1Ch01 NbV1Ch02 95.53 917 40 1 141 1057 70849273 70850188 0.0e+00 1639.0 NbV1Ch01 NbV1Ch02 94.87 409 21 0 2190 2598 70851376 70851784 1.2e-208 722.0 NbV1Ch01 NbV1Ch02 95.58 294 13 0 1342 1635 70850504 70850797 7.4e-152 534.0 NbV1Ch01 NbV1Ch02 94.79 307 15 1 1650 1956 70850825 70851130 7.6e-152 533.0 NbV1Ch01 NbV1Ch02 95.57 203 9 0 1139 1341 70850271 70850473 3.8e-101 365.0 NbV1Ch01 NbV1Ch02 96.05 177 7 0 2006 2182 70851198 70851374 6.3e-87 318.0 NbV1Ch01 NbV1Ch02 94.78 134 6 1 1 134 70849133 70849265 1.7e-60 230.0 NbV1Ch01 NbV1Ch02 100.00 31 0 0 1072 1102 70850202 70850232 2.9e-09 60.0 NbV1Ch01 NbV1Ch02 100.00 14 0 0 1124 1137 70850255 70850268 1.7e+01 27.0 NbV1Ch01 NbV1Ch02 100.00 14 0 0 1978 1991 70851137 70851150 2.2e+01 27.0 NbV1Ch01 NbV1Ch14 84.71 641 98 0 88 728 55561088 55561728 5.6e-266 913.0 NbV1Ch01 NbV1Ch14 81.59 277 50 1 2242 2517 55563231 55563507 8.7e-102 367.0 NbV1Ch01 NbV1Ch14 86.75 249 32 1 1693 1940 55562649 55562897 8.6e-98 354.0 NbV1Ch01 NbV1Ch14 86.70 203 27 0 1388 1590 55562302 55562504 9.8e-86 314.0 NbV1Ch01 NbV1Ch14 87.80 123 15 0 2019 2141 55562999 55563121 1.6e-49 194.0 NbV1Ch01 NbV1Ch14 93.62 47 3 0 943 989 55561913 55561959 6.7e-15 79.0 NbV1Ch01 NbV1Ch14 89.66 29 3 0 2541 2569 55563528 55563556 2.0e-05 47.0 NbV1Ch01 NbV1Ch14 95.24 21 1 0 1284 1304 55562159 55562179 1.1e-02 38.0 NbV1Ch01 NbV1Ch14 83.33 24 4 0 748 771 55561748 55561771 1.1e-01 35.0 NbV1Ch01 NbV1Ch14 100.00 11 0 0 1662 1672 55562592 55562602 1.5e+03 21.0 NbV1Ch01 NbV1Ch14 100.00 10 0 0 931 940 55561798 55561807 3.8e+03 20.0 NbV1Ch01 NbV1Ch14 100.00 7 0 0 1677 1683 55562614 55562620 2.7e+05 13.0 NbV1Ch01 NbV1Ch13 92.31 39 3 0 2070 2108 22520151 22520113 1.4e-10 64.0 What did I miss?

Thank you in advance,

Michal

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/GMOD/Apollo/issues/1528#issuecomment-465814570, or mute the thread https://github.com/notifications/unsubscribe-auth/AAt2qp8NMCdbHE-FAtIvvAaAFYU5edxIks5vPeshgaJpZM4Md__w.

mictadlo commented 5 years ago

Thank you it works like a charm after I added /data/apollo/NbV1ChF.2bit