Open mictadlo opened 4 years ago
nathandunn
commented
4 years ago @mictadlo Is there something that the elasticsearch plugin is doing that the regular name search is not?
Apologize if you've already answered this.
This is kind of an advanced plugin to setup if you don't need it.
mictadlo
commented
4 years ago Hi,
I presented Apollo to our group and the biggest request was to improve the search. Most of our annotation starts with Protein i.e. we have first type Protein follow by the next part of the gene description.
NbV1Ch01 NbGenome mRNA 31788319 31794554 . + . ID=Nb3PK41369.mrna1;Name=Nb3PK41369;Note=Protein CASEIN KINASE I-LIKE 3;Parent=Nb3PK41369.path1;coverage=100.0;identity=100.0;indels=0;matches=1272;mismatches=0;unknowns=0However, the biologist would like to to be able to type e.g. KINASE and find all possibilities in a similar way to https://www.youtube.com/watch?v=W2FLc-sRzLA .
If I understood this plugin correctly then it will provide this kind of improved search.
Michal
nathandunn
commented
4 years ago @mictadlo gotcha.
Is this elasticsearch @cmdcolin ? It looks different from https://github.com/elsiklab/jbrowse_elasticsearch , but maybe its just newer.
@abretadeau is this a project you are familiar with (I see the INRA label).
nathandunn
commented
4 years ago Also, as much as I like docker, it might be easier to do without it to get it started.
cmdcolin
commented
4 years ago This is the error message you need to resolve
ERROR: LWP will support https URLs if the LWP::Protocol::https module is installed.
If you install this module manually maybe with
cpanm LWP::Protocol::httpsThen you can then re-run the setup.sh
mictadlo
commented
4 years ago Hi, Thank you. To solve the Error few additianal packages had to be intalled:
# cd /usr/local/tomcat/webapps/ROOT/jbrowse/plugins/ElasticSearch
# apt install cpanminus libssl-dev
# cpanm LWP::Protocol::https
# bash setup.sh
...
Skipping /usr/local/bin/bp_extract_feature_seq.pl (unchanged)
Skipping /usr/local/bin/bp_parse_hmmsearch.pl (unchanged)
Writing /usr/local/lib/x86_64-linux-gnu/perl/5.24.1/auto/Bio/.packlist
OK
Successfully installed BioPerl-1.007002
Installing /usr/local/share/perl/5.24.1/x86_64-linux-gnu-thread-multi/.meta/BioPerl-1.007002/install.json
1 distribution installed
--> Working on .
Configuring /usr/local/tomcat/webapps/ROOT/jbrowse/plugins/ElasticSearch ... OK
Building jbrowse_elasticsearch-1.0 ... OK
Successfully installed jbrowse_elasticsearch-1.0
1 distribution installed
--> Working on https://github.com/GMOD/jbrowse/archive/master.tar.gz
Fetching https://github.com/GMOD/jbrowse/archive/master.tar.gz ... OK
Configuring jbrowse-master ... OK
Building JBrowse- ... OK
Successfully installed JBrowse-
1 distribution installed
Installing NodeJS pre-requisites
audited 148 packages in 1.526s
found 16 vulnerabilities (7 low, 2 moderate, 6 high, 1 critical)
run `npm audit fix` to fix them, or `npm audit` for details
╭────────────────────────────────────────────────────────────────╮
│ │
│ New minor version of npm available! 6.4.1 → 6.10.3 │
│ Changelog: https://github.com/npm/cli/releases/tag/v6.10.3 │
│ Run npm install -g npm to update! │
│ │
╰────────────────────────────────────────────────────────────────╯
Setting up test dataset
Done
# npm start
> jbrowse_elasticsearch@1.0.5 start /usr/local/tomcat/webapps/ROOT/jbrowse/plugins/ElasticSearch
> node ./bin/wwwHowever, when I search I get Error: RequestError: Unable to load undefined status: 404 as show below.
What did I miss?
@nathandunn The source code from MyGenomeBrowser can be found here. It appears that there are not using any EleasticSearch. Thank you in advance.
Michal
cmdcolin
commented
4 years ago You have to run the bin/generate-elastic-search.pl script probably, it is looking for a config variable that this script outputs
mictadlo
commented
4 years ago /data/apollo_data/benth-NbV1Ch03# perl /usr/local/tomcat/webapps/ROOT/jbrowse/plugins/ElasticSearch/bin/generate-elastic-search.pl --genome /data/apollo_data/benth-NbV1Ch03/NbV1Ch03.fa --out /data/apollo_data/benth-NbV1Ch03
[NoNodes] ** No nodes are available: [http://localhost:9200], called from sub Search::Elasticsearch::Client::6_0::Role::Bulk::index at /usr/local/tomcat/webapps/ROOT/jbrowse/plugins/ElasticSearch/bin/../src/perl5/Bio/JBrowse/Cmd/ElasticSearch.pm line 104.root@3870b7ed7057What did I miss?
Thank you in advance,
Michal
cmdcolin
commented
4 years ago This means your elasticsearch daemon is not running. You need to do something like sudo service elasticsearch start. The default elasticsearch daemon runs on localhost:9200 which is why it is looking for nodes there
mictadlo
commented
4 years ago I add the below missing dependencies to Apollo/Dockerfile
...
RUN apt-get -qq update --fix-missing && \
apt-get --no-install-recommends -y install cpanminus libssl-dev vim less && \
apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \
cpanm LWP::Protocol::https
ADD docker-files/createenv.sh /createenv.sh
CMD "/createenv.sh"
ADD docker-files/launch.sh /launch.sh
CMD "/launch.sh"When I log into docker-compose exec apollo /bin/bash and I execute
service elasticsearch start
elasticsearch: unrecognized serviceand
/usr/local/tomcat# perl /usr/local/tomcat/webapps/ROOT/jbrowse/plugins/ElasticSearch/bin/generate-elastic-search.pl --genome /data/apollo_data/benth-NbV1Ch03/NbV1Ch03.fa --out /data/apollo_data/benth-NbV1Ch03
Can't locate Search/Elasticsearch.pm in @INC (you may need to install the Search::Elasticsearch module) (@INC contains: /usr/local/tomcat/webapps/ROOT/jbrowse/plugins/ElasticSearch/bin/../src/perl5 /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at /usr/local/tomcat/webapps/ROOT/jbrowse/plugins/ElasticSearch/bin/../src/perl5/Bio/JBrowse/Cmd/ElasticSearch.pm line 14.
BEGIN failed--compilation aborted at /usr/local/tomcat/webapps/ROOT/jbrowse/plugins/ElasticSearch/bin/../src/perl5/Bio/JBrowse/Cmd/ElasticSearch.pm line 14.
Compilation failed in require at /usr/local/tomcat/webapps/ROOT/jbrowse/plugins/ElasticSearch/bin/generate-elastic-search.pl line 7.P. S. I just discovered that
elasticsearch | [2019-08-19T04:57:50,322][INFO ][o.e.n.Node ] [tOzB2e6] closed
elasticsearch | [2019-08-19T04:57:50,324][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started
elasticsearch exited with code 78What did I miss?
Thank you in advance,
Michal
cmdcolin
commented
4 years ago I think elasticsearch is just not installed on your machine? Follow instructions e.g. https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html
mictadlo
commented
4 years ago The problem with:
elasticsearch | [2019-08-19T04:57:50,322][INFO ][o.e.n.Node ] [tOzB2e6] closed
elasticsearch | [2019-08-19T04:57:50,324][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started
elasticsearch exited with code 78I solved it with:
> sudo sysctl -w vm.max_map_count=262144
> docker-compose up --buildHowever, I still have the below problem. Your right ElasticSearch is not install in the same Docker as Apollo but in a seperated one similar to.
> cd /usr/local/tomcat/webapps/ROOT/jbrowse/plugins/ElasticSearch
> bash setup.sh
found 16 vulnerabilities (7 low, 2 moderate, 6 high, 1 critical)
run `npm audit fix` to fix them, or `npm audit` for details
╭────────────────────────────────────────────────────────────────╮
│ │
│ New minor version of npm available! 6.4.1 → 6.10.3 │
│ Changelog: https://github.com/npm/cli/releases/tag/v6.10.3 │
│ Run npm install -g npm to update! │
│ │
╰────────────────────────────────────────────────────────────────╯
Setting up test dataset
Done
# perl /usr/local/tomcat/webapps/ROOT/jbrowse/plugins/ElasticSearch/bin/generate-elastic-search.pl --genome /data/apollo_data/benth-NbV1Ch03/NbV1Ch03.fa --out /data/apollo_data/benth-NbV1Ch03
[NoNodes] ** No nodes are available: [http://localhost:9200], called from sub Search::Elasticsearch::Client::6_0::Role::Bulk::index at /usr/local/tomcat/webapps/ROOT/jbrowse/plugins/ElasticSearch/bin/../src/perl5/Bio/JBrowse/Cmd/ElasticSearch.pm line 104.root@c37f95c35332What did I miss?
Thank you in advance
Michal
cmdcolin
commented
4 years ago If it is a different machine you can pass the --elasticurl parameter in the bin/generate-elastic-search.pl
mictadlo
commented
4 years ago A new error came up after
perl /usr/local/tomcat/webapps/ROOT/jbrowse/plugins/ElasticSearch/bin/generate-elastic-search.pl --genome /data/apollo_data/benth-NbV1Ch03/NbV1Ch03.fa --out /data/apollo_data/benth-NbV1Ch03 --elasticurl elasticsearch:9200
...
Bulk error [index]: {"_type":"loc","_index":"gene/data/apollo_data/benth-NbV1Ch03/NbV1Ch03.fa","error":{"index":"gene/data/apollo_data/benth-NbV1Ch03/NbV1Ch03.fa","reason":"Invalid index name [gene/data/apollo_data/benth-NbV1Ch03/NbV1Ch03.fa], must not contain the following characters [ , \", *, \\, <, |, ,, >, /, ?]","index_uuid":"_na_","type":"invalid_index_name_exception"},"_id":null,"status":400} at /usr/local/share/perl/5.24.1/Search/Elasticsearch/Client/6_0/Role/Bulk.pm line 47, <$fh> line 8153.What did I miss?
Thank you in advance,
Michal
cmdcolin
commented
4 years ago Just use a simple "genome" name e.g. --genome FruitFly
This will output a config parameter in your trackList.json called elasticIndexName, and the error message you are seeing meant that the index names are not allowed to contain complicated characters, so just use a simple genome name
mictadlo
commented
4 years ago I changed it to the below one
# perl /usr/local/tomcat/webapps/ROOT/jbrowse/plugins/ElasticSearch/bin/generate-elastic-search.pl --genome nbv1ch03 --out /data/apollo_data/benth-NbV1Ch03 --elasticurl elasticsearch:9200and the previous error disappeared but in Apollo I got now:
Error: RequestError: Unable to load http://localhost:3000/?contains=d&index=nbv1ch03&exact=false status: 0What did I miss?
Thank you in advance.
Michal
cmdcolin
commented
4 years ago You have to configure where the plugin looks for the node.js middleware app e.g. the "node bin/www" the --url argument to bin/generate-elastic-search.pl
By default it runs on Port 3000 and for testing it makes since to be localhost but in a production instance it may be http://yourhost.com:3000
mictadlo
commented
4 years ago The error remained after the below command
perl /usr/local/tomcat/webapps/ROOT/jbrowse/plugins/ElasticSearch/bin/generate-elastic-search.pl --genome nbv1ch03 --out /data/apollo_data/benth-NbV1Ch03 --elasticurl elasticsearch:9200 --url http://apollo:3000What did I miss?
Thank you in advance,
Michal
mictadlo
commented
4 years ago I changed the docker-compose.yml file to support port 3000:
version: "3.2"
services:
apollo:
build: Apollo
volumes:
- ./data/:/data
- postgres-data:/var/lib/postgresql
ports:
- 8888:8080
- 3000:3000
restart: always
depends_on:
- elasticsearch
elasticsearch:
image: elasticsearch:6.4.2
container_name: elasticsearch
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata1:/usr/share/elasticsearch/data
ports:
- 9200:9200
- 9300:9300
volumes:
postgres-data:
esdata1:
driver: localAdditionally, I changed also Apollo's Dockerfile:
EXPOSE 3000
RUN apt-get -qq update --fix-missing && \
apt-get --no-install-recommends -y install cpanminus libssl-dev vim less && \
apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \
cpanm LWP::Protocol::https
ADD docker-files/createenv.sh /createenv.sh
CMD "/createenv.sh"
ADD docker-files/launch.sh /launch.sh
CMD "/launch.sh"Then I did
docker-compose up --build
docker-compose exec apollo /bin/bashInside the Apollo container I tried the below comands but I got Connection refused
root@2c797efffc92:/usr/local/tomcat/webapps/ROOT/jbrowse/plugins/ElasticSearch# curl http://apollo:3000
curl: (7) Failed to connect to apollo port 3000: Connection refusedroot@2c797efffc92:/usr/local/tomcat/webapps/ROOT/jbrowse/plugins/ElasticSearch# curl http://localhost:3000
curl: (7) Failed to connect to localhost port 3000: Connection refusedWhat possible could I miss?
Thank you in advance,
Michal
cmdcolin
commented
4 years ago Does the docker container launch the nodejs middleware e.g. the node ./bin/www
You can configure nodejs middlewares to be launched by system services e.g. pm2 but otherwise just need to launch this middleware
mictadlo
commented
4 years ago Still the same problem with:
root@2c797efffc92:/usr/local/tomcat/webapps/ROOT/jbrowse/plugins/ElasticSearch# node ./bin/wwwI'd recommend looking up how simple express.js nodejs webservices work in your environment. See if you can get a hello world example from express.js to work perhaps https://expressjs.com/en/starter/hello-world.html and then you can see if my service works if you can get that to work
Hi @cmdcolin and @nathandunn, I followed this topic and did change a little of its
docker-compose.ymlfile. Please find the updated file below:To start it:
However, I ran into the following errors:
Next, I did:
I also tried:
What did I miss?
Thank you in advance
Michal