openstf / stf

Control and manage Android devices from your browser.
https://openstf.io
Other
13.17k stars 2.71k forks source link

Is there any video tutorial to deploy stf on real servers? #622

Open suvansh-dev opened 6 years ago

suvansh-dev commented 6 years ago

Hi, I am currently using stf on localhost but i want to use stf on external network too, from the issues i have found that stf can be deployed to real servers by following the tutorial from Deployment.md but first thing is i don't have any actual server right now so can i deploy it again on my linux machine behaving like a server and the second thing is a bit straight forward video tutorial of deploying stf on real servers would be best.

thinkhy commented 6 years ago

I've worked out a Shell script to automate the procedure of deployment. Hope this would be helpful.

#!/usr/bin/env bash
#######################################################################################
# file:    deploy_stf.sh
# brief:   deploy components of stf with docker
# creator: thinkhy
# date:    2017-04-20
# usage:   ./deploy_stf.sh  Note: run it as ROOT user
# changes: 
#    1. 2017-04-20 init  @thinkhy
#    2. 2017-04-26 add systemd unit  @thinkhy
#    3. 2017-05-04 install adb on host machine @thinkhy
# 
#######################################################################################

check_return_code() {
  if [ $? -ne 0 ]; then
    echo "Failed to run last step!"     
    return 1
  fi

  return 0
}

assert_run_ok() {
  if [ $? -ne 0 ]; then
    echo "Failed to run last step!"     
    exit 1
  fi

  return 0
}

prepare() {
  echo "setup environment ..."

  # Given advantages of performance and stability, we run adb server and rethinkdb 
  # on host(physical) machine rather than on docker containers, so need to
  #  install package of android-tools-adb first [ thinkhy 2017-05-04 ]

  # install adb
  apt-get install -y android-tools-adb

  apt-get install -y docker.io
  assert_run_ok

  docker pull openstf/stf 
  assert_run_ok

  docker pull sorccu/adb 
  assert_run_ok

  docker pull rethinkdb 
  assert_run_ok

  docker pull openstf/ambassador 
  assert_run_ok

  docker pull nginx
  assert_run_ok

  cp -rf systemd.unit/adbd.service /etc/systemd/system 
  assert_run_ok

  echo 1>"env.ok"

}

if [ ! -e env.ok ]; then
  prepare
fi

# start local adb server
echo "start adb server"
adb start-server

# start rethinkdb
echo "start docker container: rethinkdb"
docker run -d --name rethinkdb -v /srv/rethinkdb:/data --net host rethinkdb rethinkdb --bind all --cache-size 8192 --http-port 8090
check_return_code

# start nginx, note: generate nginx.conf first
echo "start docker container: nginx"
docker run -d -v $(pwd)/nginx.conf:/etc/nginx/nginx.conf:ro --name nginx --net host nginx nginx
check_return_code

# create tables 
echo "start docker container: stf-migrate"
docker run -d --name stf-migrate --net host openstf/stf stf migrate
check_return_code

# create storage components 
echo "start docker container: storage-plugin-apk-3300"
docker run -d --name storage-plugin-apk-3300 -p 3300:3000 --dns __DNS_ADDRESS__  openstf/stf stf storage-plugin-apk  --port 3000 --storage-url http://__IP_ADDRESS__/
check_return_code

echo "start docker container: storage-plugin-image-3400"
docker run -d --name storage-plugin-image-3400 -p 3400:3000 --dns __DNS_ADDRESS__  openstf/stf stf storage-plugin-image  --port 3000 --storage-url http://__IP_ADDRESS__/
check_return_code  

echo "start docker container: storage-temp-3500"
docker run -d --name storage-temp-3500 -v /mnt/storage:/data -p 3500:3000 --dns __DNS_ADDRESS__  openstf/stf stf storage-temp  --port 3000 --save-dir /data
check_return_code

# tri-proxy
echo "start docker container: triproxy-app"
docker run -d --name triproxy-app --net host openstf/stf stf triproxy app --bind-pub "tcp://*:7150" --bind-dealer "tcp://*:7160" --bind-pull "tcp://*:7170"
check_return_code

echo "start docker container: triproxy-dev"
docker run -d --name triproxy-dev --net host openstf/stf stf triproxy dev --bind-pub "tcp://*:7250" --bind-dealer "tcp://*:7260" --bind-pull "tcp://*:7270"
check_return_code

# auth
echo "start docker container: stf-auth-3200"
 docker run -d --name stf-auth-3200 -e "SECRET=YOUR_SESSION_SECRET_HERE" -p 3200:3000 --dns __DNS_ADDRESS__ openstf/stf stf auth-mock --port 3000 --app-url http://__IP_ADDRESS__/
check_return_code

# api 
echo "start docker container: stf-api"
docker run -d --name stf-api --net host -e "SECRET=YOUR_SESSION_SECRET_HERE"  openstf/stf stf api --port 3700 --connect-sub tcp://__IP_ADDRESS__:7150  --connect-push tcp://__IP_ADDRESS__:7170
check_return_code

# stf APP
echo "start docker container: stf-app-3100"
docker run -d --name stf-app-3100 --net host -e "SECRET=YOUR_SESSION_SECRET_HERE" -p 3100:3000 openstf/stf stf app --port 3100 --auth-url http://__IP_ADDRESS__/auth/mock/ --websocket-url http://__IP_ADDRESS__/
check_return_code

# processor
echo "start docker container: stf-processor"
docker run -d --name stf-processor --net host openstf/stf stf processor stf-processor.service --connect-app-dealer tcp://__IP_ADDRESS__:7160 --connect-dev-dealer tcp://__IP_ADDRESS__:7260
check_return_code

# websocket
echo "start docker container: websocket"
docker run -d --name websocket -e "SECRET=YOUR_SESSION_SECRET_HERE" --net host openstf/stf stf websocket --port 3600 --storage-url http://__IP_ADDRESS__/ --connect-sub tcp://__IP_ADDRESS__:7150 --connect-push tcp://__IP_ADDRESS__:7170
check_return_code

# reaper
echo "start docker container: reaper"
docker run -d --name reaper --net host openstf/stf stf reaper dev --connect-push tcp://__IP_ADDRESS__:7270 --connect-sub tcp://__IP_ADDRESS__:7150 --heartbeat-timeout 30000
check_return_code

# provider
echo "start docker container: provider-__HOSTNAME__"
docker run -d --name provider1 --net host openstf/stf stf provider --name "provider-__HOSTNAME__" --connect-sub tcp://__IP_ADDRESS__:7250 --connect-push tcp://__IP_ADDRESS__:7270 --storage-url http://__IP_ADDRESS__ --public-ip __IP_ADDRESS__ --min-port=15000 --max-port=25000 --heartbeat-interval 20000 --screen-ws-url-pattern "ws://__IP_ADDRESS__/d/floor4/<%= serial %>/<%= publicPort %>/"
check_return_code
suvansh-dev commented 6 years ago

Hi, Thanks for the response but while running the following command:

cp -rf systemd.unit/adbd.service /etc/systemd/system

i have met with the following error:

cp: cannot stat 'systemd.unit/adbd.service': No such file or directory

I have tried looking in the following directory system but there was no systemd.unit file.

rahul0698 commented 6 years ago

me too having the same error while running the shell script.

cannot stat 'systemd.unit/adbd.service': No such file or directory

thinkhy commented 6 years ago

OK. I missed the file of adbd.service below ......

[Unit] Description=Android Debug Bridge daemon

[Service]
#TimeoutStartSec=1min
Restart=always
RestartSec=2s
Type=forking
User=root
ExecStartPre=/usr/bin/adb kill-server
ExecStart=/usr/bin/adb start-server
ExecStop=/usr/bin/adb kill-server
sorccu commented 6 years ago

It's the output of the pwd command.

On Tue, Jul 11, 2017 at 19:39 RAHUL RAJ CHAKRAVORTY < notifications@github.com> wrote:

Hi, What is $(pwd) in the docker command that you are running for nginx ?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/openstf/stf/issues/622#issuecomment-314417236, or mute the thread https://github.com/notifications/unsubscribe-auth/AAB-_Q4CCzxHIhr735lmHQHpXINHtrUTks5sM17xgaJpZM4N6wHu .

rahul0698 commented 6 years ago

hi, I am running the shell script some of the containers are running but some containers are missing when i am running sudo docker ps but they are running in background.

start adb server
start docker container: rethinkdb
db09335f3e4f0a4f793c93bcfbf8ac77a54854a4a94f7a03f839760004871e66
start docker container: nginx
/home/tft/Desktop/stf
ef85734fc6d12437ef663bd0f59d964b926a0b359cda263d86e93c165f74cb63
docker: Error response from daemon: containerd: container not started.
Failed to run last step!
start docker container: stf-migrate
43320a6dfa9c8b7cc7891583692dd7542bd58863d2217c6e9ee6a073a188c368
start docker container: storage-plugin-apk-3300
873fdb099ac1610c6f704891f937cb5b73433ee9c98fb6199c9e177a98e10b29
start docker container: storage-plugin-image-3400
607a7e7f6bc09b206a8a6535e939e9516a65798f19aac233baa1a1bd7e4abd76
start docker container: storage-temp-3500
584a37cf09f8c20831f4fe9304f9d96ae8b2536025133673c2cae7d9cdf6f667
start docker container: triproxy-app
ca5fc57ea1729e0fe8c19c4af90b181b0c57bffd203b1636d1cd9e5dec94bb72
start docker container: triproxy-dev
169a79a0de9c2e1c4dc25fe3f0213f47497a6358a1c7413573cac371843d4444
start docker container: stf-auth-3200
53c4e6b46dd0cd259c9a753b11a5f430f9b0ef0ed000095c059e18abdf714e08
start docker container: stf-api
b65750b4be188939914c32dcfd2bad1c459c274bc4d5dd23766aa87cd30a8ef5
start docker container: stf-app-3100
6415b056adef0342a94a82818b9346456197419fde584545195aeb85b0620738
start docker container: stf-processor
2e09f8daee3db117781c1f3e3399111327ed969608d1043377fb664f11b2cb7a
start docker container: websocket
4a3f822bdd714830158ea56443fc9b42c797b1bda5bcb71a09124887a0790e43
start docker container: reaper
bd700cca1d8fad4a69adb648c1ae451ee6c480d2b8a20b0faabae51b768b1441
start docker container: provider-__HOSTNAME__
5023e4749cb696050bdedbd088fd4c423f8286c5958a5950604d87efc8987f3c

I think that nginx.conf file is having some issue. Can you please help. Thanks in advance

rahul0698 commented 6 years ago
cp -rf systemd.unit/adbd.service /etc/systemd/system 
  assert_run_ok

this was showing an error so I have to remove it as it is simply copying the adbd.service file so I made my adbd.service file in etc/systemd/system. after that when I ran my .sh file it is showing error in creating a container for nginx. I had replaced the nginx.conf file in etc/nginx/nginx.conf as per the DEPLOYMENT.md. I can connect to rethinkdb WebUI on port 8080 and when I ran 192.168.104.183:3200 it is redirecting to auth/mock but it is not able to load entry.js and commonentry.js when I look in the chrome debug tool. Not able to figure out the issue. Please guide. after running the .sh file, the console was mentioned in the previous comment Thanks in advance.

thinkhy commented 6 years ago

@rahul0698 and @suvansh-dev I've created a new repo for automation deployment of stf with docker. Please try and feedback.

https://github.com/thinkhy/deploy-stf-docker

rahul0698 commented 6 years ago

hello, Those who are having the problem in deploying stf to a real server, can now deploy with one click by the shell script written by @thinkhy. You can clone the repo https://github.com/thinkhy/deploy-stf-docker and then you just need to run the script.

After running the script if you encounter problem like not able to connect to 127.0.0.1:28015, it may be due to initially installed rethinkdb locally on your machine. So deploy it on a freshly installed Linux machine and just pull the repo and run the script without installing other dependencies and you are good to go. Thanks!

suvansh-dev commented 6 years ago

@thinkhy Thanks a lot!

thinkhy commented 6 years ago

It's my pleasure. @rahul0698 Thanks for sharing your experience and solution.

xell66 commented 6 years ago

With the script I get a blank page at localhost:3200/auth/mock. Do I need to change something? I tried to look into the stf logfiles but i dont know how I can find it inside the docker container. And isnt the rethinkdb-proxy-28015.service required? Thank You for the nice skript anyways!

thinkhy commented 6 years ago

@xell66 you can check log of docker container using the command "docker logs CONTAINER_ID"

adrianxpro commented 6 years ago

Hi i have the same error that @xell66 and I check log of docket container (stf-auth-3200) and i see this:

INF/auth-mock 1 [*] Listening on port 3000

please help

thinkhy commented 6 years ago

This log shows your stf is running well ...

terminalone commented 6 years ago

@thinkhy Need help with deploying stf on two servers?can you add the functionality to deploy to multiple machines

thinkhy commented 6 years ago

@terminalone OK. I will enhance my scripts.

xwSurfer commented 5 years ago

@thinkhy Thank you very much for your script, I run STF successfully in Virtualbox , but there is a problem that I can't access app through the browser on host, I have forwarded rethindb port (8090) and app port (3100) on vitrualbox, and I can get the DB page through 127.0.0.1:8090 on host browser. but I can't access app (auth/mock) page through 127.0.0.1:3100. I've tried many manyways, such as ngrok and deploy via docker, but none of them can work.

My problem is very similar to this: https://github.com/openstf/stf/issues/700

Thank you very very much !

terminalone commented 5 years ago

@thinkhy Any Update on the script for deploying on 2 servers.Thanks in advance.

michaelseno commented 5 years ago

Hi I am new to open STF. I am just wondering if this script will setup in ubuntu and the ip can be accessed outside the network as well? just trying to clarify. Thanks

jhonsuicide commented 4 years ago

Hi @thinkhy I've already deploy stf on virtual box, the access localhotst:3200/auth/moc. But the result still "Cannot GET /auth/moc" on my browser. Need some advices