Closed GMaciag closed 4 years ago
dweemx
commented
4 years ago Hi @GMaciag,
We are happy to hear that SCope useful for your research. SCope is currently under active development. So, it's possible that we broke something to deploy SCope on AWS. I'm going to look into this and will keep you in touch as soon as possible
GMaciag
commented
4 years ago Hi @dweemx
Thank you for a prompt reply. It would be really great to have SCope deployed on AWS, so I appreciate you looking into this. Please let me know if I can assist you in figuring out the issue.
dweemx
commented
4 years ago Hi @GMaciag,
Good news, I could reproduce the problem you are experiencing. The tutorial is indeed outdated and some fixes are required to make SCope work on AWS.
I've made a branch fixes_for_aws_deployment with fixes (code fixes and tutorial update) that solve the issue
Could you give it a try and let me know if it works for you ?
GMaciag
commented
4 years ago Hi again
Unfortunately, it still doesn't work for me :(
I made an entirely new aws instance (from scratch, following the tutorial). I followed the new tutorial https://github.com/aertslab/SCope/blob/fixes_for_aws_deployment/docs/aws-deployment-source/README.md , while paying attention to download the fixed version of the package with git clone https://github.com/aertslab/SCope.git --branch fixes_for_aws_deployment. I confirmed that bin/launch.js file has been changed (as I saw from the commits that's the only code fix that has been applied). Now when running npm run scope-dev-aws (or npm run scope-aws, doesn't matter), I get the following output:
(base) [ec2-user@ip-172-31-13-26 SCope]$ npm run scope-dev-aws
> SCope@1.8.2 scope-dev-aws /home/ec2-user/SCope
> node bin/launch.js --dev --aws
/$$$$$$ /$$$$$$
/$$__ $$ /$$__ $$
| $$ \__/| $$ \__/ /$$$$$$ /$$$$$$ /$$$$$$
| $$$$$$ | $$ /$$__ $$ /$$__ $$ /$$__ $$
\____ $$| $$ | $$ \ $$| $$ \ $$| $$$$$$$$
/$$ \ $$| $$ $$| $$ | $$| $$ | $$| $$_____/
| $$$$$$/| $$$$$$/| $$$$$$/| $$$$$$$/| $$$$$$$
\______/ \______/ \______/ | $$____/ \_______/
| $$
| $$
|__/
-------- running settings --------
Running on Linux
Running in AWS: true
Running as daemon: true
Running in development mode: true
*** please run in production mode except if it's for testing purposes.
----------------------------------
--------- sanity checks ----------
poetry command found.
- Checking SCope Server...
scope-server cannot be found.
- Checking SCope poetry environment...
- Checking tmux...
tmux command found.
- Checking directories (assets)...
- Checking Apache configuration...
Making SCope HTTP Apache web server config file...
Done!
Copying SCope HTTP Apache web server config file apache/scope.conf to /etc/httpd/conf.d
Done!
- Checking public host address...
----------------------------------
--------- running scope ----------
Starting SCope...
Starting SCope Server...
SCope Server started as daemon!
It can be accessed using 'tmux a -t scope-server' command.
Running SCope Client...
SCope Client started as daemon!
It can be accessed using 'tmux a -t scope-client' command.
A SCope instance is running at http://ec2-13-48-97-71.eu-north-1.compute.amazonaws.com
----------------------------------Unlike previously it now throws:
- Checking SCope Server...
scope-server cannot be found.Despite that, the command 'tmux a -t scope-server' gives (same as before):
2020-07-07 11:23:58,557 - scopeserver - INFO -
/$$$$$$ /$$$$$$ /$$$$$$
/$$__ $$ /$$__ $$ /$$__ $$
| $$ \__/| $$ \__/ /$$$$$$ /$$$$$$ /$$$$$$ | $$ \__/ /$$$$$$ /$$$$$$ /$$ /$$ /$$$$$$ /$$$$$$
| $$$$$$ | $$ /$$__ $$ /$$__ $$ /$$__ $$ | $$$$$$ /$$__ $$ /$$__ $$| $$ /$$//$$__ $$ /$$__ $$
\____ $$| $$ | $$ \ $$| $$ \ $$| $$$$$$$$ \____ $$| $$$$$$$$| $$ \__/ \ $$/$$/| $$$$$$$$| $$ \__/
/$$ \ $$| $$ $$| $$ | $$| $$ | $$| $$_____/ /$$ \ $$| $$_____/| $$ \ $$$/ | $$_____/| $$
| $$$$$$/| $$$$$$/| $$$$$$/| $$$$$$$/| $$$$$$$ | $$$$$$/| $$$$$$$| $$ \ $/ | $$$$$$$| $$
\______/ \______/ \______/ | $$____/ \_______/ \______/ \_______/|__/ \_/ \_______/|__/
| $$
| $$
|__/
2020-07-07 11:23:58,557 - scopeserver - DEBUG - Starting data server on port 55853. app_mode: False
2020-07-07 11:23:58,558 - scopeserver - DEBUG - Starting upload server on port 55851
2020-07-07 11:23:58,570 - scopeserver - DEBUG - Starting bind server on port 55852
Running on 127.0.0.1:8000 over http (CTRL + C to quit)
{"msg": {"origin": "PServer", "value": "SIGSTART"}}
2020-07-07 11:23:59,105 - scopeserver.dataserver.utils.data_file_handler - DEBUG - Building UUID "database"
2020-07-07 11:23:59,110 - scopeserver.dataserver.utils.data_file_handler - DEBUG - Existing User UUIDs:
2020-07-07 11:23:59,111 - scopeserver.dataserver.utils.data_file_handler - DEBUG - Existing Permanent Sessions:"
2020-07-07 11:23:59,111 - scopeserver.dataserver.utils.data_file_handler - DEBUG - UUID SCopeApp__2240e768-5063-4940-a048-8951c639eb7c, mode rw. Valid until 2021-07-07 11:23:59
2020-07-07 11:23:59,111 - scopeserver.dataserver.utils.data_file_handler - DEBUG - Building UUID "database"
{"msg": {"origin": "GServer", "value": "SIGSTART"}}The command 'tmux a -t scope-client' again gives session not found: scope-client.
Trying to connect to the webpage gives me 'service unavailable' error.
dweemx
commented
4 years ago Hi @GMaciag ,
Unfortunately, I cannot reproduce the error you get. In my case the client and server are starting and I can access SCope on the AWS server.
Maybe some stuff to take into consideration:
SCope seems to require ~1.3Gb, therefore I would recommend to us a t2.small AWS instance insteadSCope in production mode. So I would recommend to use npm run scope-dev-aws command instead of npm run scope-aws. You mentioned, there was no difference in running these 2 different commands. You were right but now it has been fixed in the latest updates of the branch fixes_for_aws_deployment.Here the complete commands I run until having SCope running and being accessible in the browser.
sudo yum update -y
sudo yum install -y httpd
sudo usermod -a -G apache ec2-user
exit
ssh aws-scope
sudo chown -R ec2-user:apache /var/www
sudo chown -R ec2-user:apache /etc/httpd/conf.d
sudo chmod 2775 /var/www && find /var/www -type d -exec sudo chmod 2775 {} \;
sudo chmod 2775 /etc/httpd/conf.d
find /var/www -type f -exec sudo chmod 0664 {} \;
IS_ENABLED=$(sudo systemctl is-enabled httpd)
if [ "${IS_ENABLED}" = "enabled" ]; then
echo "Apache Web Server is enabled"
else
# If not enabled start
echo "Apache Web Server is disabled. Starting Apache Web Server..."
sudo systemctl start httpd && sudo systemctl enable httpd
fi
sudo systemctl is-enabled httpd
sudo systemctl enable httpd
sudo yum install -y git
curl --silent --location https://rpm.nodesource.com/setup_10.x | sudo bash -
sudo yum install -y nodejs
sudo yum groupinstall -y "Development tools"
sudo yum install -y tmux
wget --content-disposition http://bit.ly/miniconda3
bash Miniconda3-latest-Linux-x86_64.sh
bash
pip install poetry
git clone https://github.com/aertslab/SCope.git
cd SCope
git checkout fixes_for_aws_deployment
npm install
# Here you need to update the publicHostAddress enapache/config.json file by updating the value of publicHostAddress
npm run scope-dev-aws
sudo systemctl restart httpdThe tutorial is updated with all the considerations aforementioned.
GMaciag
commented
4 years ago When it comes to the type of instance, I'm using micro since it's included in the free account on aws. I don't want to pay for a small instance, especially I don't know it will work in the end. Also I'm using t3 instances since that's what is offered by aws now (and not t2). I'm not sure if that makes any difference.
I followed the commands you posted step by step. This time I didn't get the
- Checking SCope Server...
scope-server cannot be found.but still at the end the client doesn't start the webaddress returns the "Service Unavailable" page.
I apologise for not being able to provide more details on my issue, I haven't worked with aws before, I don't know what could be wrong.
dweemx
commented
4 years ago Ok, good to know you're working t3.micro. I'll give it a try and hope being able to reproduce the problem
dweemx
commented
4 years ago @GMaciag ,
Are you sure you pulled (git pull) the latest changes from the branch ? because the following message:
- Checking SCope Server...
scope-server cannot be found.should not appear
I tried on t3.micro, here is my console after running npm run scope-dev-aws (:
> SCope@1.8.2 scope-dev-aws /home/ec2-user/SCope
> node bin/launch.js --dev --aws
/$$$$$$ /$$$$$$
/$$__ $$ /$$__ $$
| $$ \__/| $$ \__/ /$$$$$$ /$$$$$$ /$$$$$$
| $$$$$$ | $$ /$$__ $$ /$$__ $$ /$$__ $$
\____ $$| $$ | $$ \ $$| $$ \ $$| $$$$$$$$
/$$ \ $$| $$ $$| $$ | $$| $$ | $$| $$_____/
| $$$$$$/| $$$$$$/| $$$$$$/| $$$$$$$/| $$$$$$$
\______/ \______/ \______/ | $$____/ \_______/
| $$
| $$
|__/
-------- running settings --------
Running on Linux
Running in AWS: true
Running as daemon: true
Running in development mode: true
*** please run in production mode except if it's for testing purposes.
----------------------------------
--------- sanity checks ----------
poetry command found.
- Checking SCope poetry environment...
- Checking tmux...
tmux command found.
- Checking directories (assets)...
- Checking Apache configuration...
Making SCope HTTP Apache web server config file...
Done!
Copying SCope HTTP Apache web server config file apache/scope.conf to /etc/httpd/conf.d
Done!
- Checking public host address...
----------------------------------
--------- running scope ----------
Starting SCope...
Running SCope Client...
SCope Client started as daemon!
It can be accessed using 'tmux a -t scope-client' command.
Starting SCope Server...
SCope Server started as daemon!
It can be accessed using 'tmux a -t scope-server' command.
A SCope instance is running at http://ec2-XXX-XXX-XXX-XXX.eu-west-3.compute.amazonaws.comThe log is not entirely correct:
SCope server is running fine (tmux a -tscope-server)SCope client is being killed (tmux a -tscope-client doesn't exist) by the server because it's trying to get more memory than there is (limit 1Gb for t3.micro). You need at least 1.3
Gb.One thing you could try if you really want to test on a micro instance is build the JavaScript file on your local computer and then rsync it to the AWS server:
npm run build-awsrsync the index.html file and the assets folder to the AWS instanceSCope repo: cp -rf . /var/www/html/sudo systemctl restart httpdI tried it my self but having problems (don't know exactly why yet):
GMaciag
commented
4 years ago Hi @dweemx
Yes, you are correct. After I have pulled the correct branch (following your instructions), I do not get the:
- Checking SCope Server...
scope-server cannot be found.message anymore.
It's good to know where the problem lies. I think we can close the issue, since the requirement for launching a t2.small instance is now mentioned right in the tutorial.
I'll look into the possibility of running a small instance in the future, for now I'll focus on alternative solutions. Thank you very much for your help with solving the issue.
dweemx
commented
4 years ago Thanks for your bug report. We will try in the future to smoothen the deployment
Hi!
SCope is a great package that I have used on my own computer, however I wanted to set it up to be easily accessed by the rest of my team. I have followed the aws tutorial https://github.com/aertslab/SCope/tree/master/tutorials/aws-deployment-source, but after successfully completing all the steps, I still cannot deploy SCope instance over public internet.
I've installed all the necessary packages in a conda enviroment (output of
conda listbelow) and after runningnpm run scope-awsI get the following output:Next I run the command
sudo systemctl restart httpd, but the address http://ec2-13-48-97-71.eu-north-1.compute.amazonaws.com shows only the test page for apache server (attached below) and not the SCope interface. So it seems my instance and the apache server are properly configured, but there are some issues on the SCope client side? This is also suggested by the fact that running the commandtmux a -t scope-clientgives me the outputsession not found: scope-client(tmux a -t scope-serverallows me access the server as usual).I'm not very experienced in networking and so don't really know what to do (or what could be wrong). Could you please help me.