On AWS after stopping and starting an instance, IP has changed; BBB no longer connects audio

taikedz commented 4 years ago

Hello

Summary

I am trying to set up BBB so that I can stop it and leave it dormant (cost saving on AWS, c5.xlarge is expensive for a personal use case!), and start it again later.

The problem is that when the IP changes, there seems to be no way to tell BBB and its services to use the new IP such that the server audio can work again - either ICE 1007 or WebRTC 1002 errors occur, and I am unable to find a way of fixing them.

Problem description

I am wanting to run a BBB server for personal use, and for meetup groups I host. Since the server spec is quite hefty, I want to be able to stop it when not in use, and bring it up just when I need it (cost saving).

I am deploying on AWS EC2, which means that when the server is stopped, and then started again, the IP address changes. When trying to join audio ICE 1007 error occurs in client.

Using bbb-conf --setip $FQDN ; bbb-conf --restart does not help; WebRTC reports an error (cannot connect on port 1935, in logs)

I used grep to locate where else the old IP is used, and manually changed those to the new IP, and restarted. At this point, along with the wevrtc error, I also get an error where FreeSwitch cannot be contacted on 5060

Originally I had found that running bbb-install.sh again would "fix" the situation (server became fully operational again), but after a couple more stop/starts even that no longer resolved the issue, and errors 1007 and 1002 cropped up once again with no resolution.

Environment

Install method: bbb-install.sh script with greenlight and -v xenial-22
Server: AWS EC2 c5.xlarge with 32GB EBS drive, Ubuntu 16.04 LTS
BBB version: 2.2.11, as reported in the logs
Client: Firefox 75, Linux Mint 19.4 x64

Steps to reproduce

The following are some general "human" notes, followed by a script workflow demonstrating exactly the steps taken, by-commands.

Steps

Create new EC2 instance c5.xlarge with 32GB EBS storage, allow ports 22, 80, 443, 16384 - 32768 (UDP) via security groups
Set a fully qualified domain name to use (I used DuckDNS for this)
Git clone and run bbb-install.sh -s $FQDN -e $EMAIL -g -v xenial-22
cd ~/greenlight; sudo docker exec greenlight-v2 bundle exec rake admin:create

At this point I can log in, and start a meeting, audio joins fine. Then:

stop EC2 instance to status "stopped"
start instance again : it gets a new public IP
Update IP in DuckDNS
run sudo bbb-conf --setip $FQDN, and sudo bbb-conf --restart

At this point,

the check reports that webrtc is unreachable on port 1935
I get ICE 1007 errors when trying to connect to meeting audio.

I checked on the server for any configs with the old IP, find them, and replace them with the new IP. I restart BBB as above.

At this point,

the check reports that webrtc is unreachable on port 1935
the check reports that FreeSwtich cannot be found on port 5060
I get WebRTC error 1002 in the browser

Script

The following is a step-by-step script demonstrating the exact steps I take. I use awscli and jq to further document it here, you should be able to reproduce this, editing the sg IDs and instance IDs as appropriate.

The outputs mentioned in the script are bundled in attachment. outputs.tar.gz

# Create an instance
aws ec2 run-instances --block-device-mappings 'DeviceName=/dev/sda1,Ebs={DeleteOnTermination=true,VolumeSize=32}' --image-id ami-0b1912235a9e70540 --key-name mykeys --instance-type c5.xlarge --security-group-ids sg-07e279c646c6083fe sg-c317afaa sg-07a46e6e --count 1 --associate-public-ip-address --tag-specifications 'ResourceType=instance,Tags=[{Key=app,Value=bigbluebutton}]'

# Get its public IP
aws ec2 describe-instances --instance-ids 'i-0cfbbe3e05de2e5f6'|jq .Reservations[0].Instances[0].PublicIpAddress

# ====================================

    # Connect via ssh and do the following, on the instance:

    # Set a FQDN - here, it will become "my-b3-server.duckdns.org"
    # In my test run, at this point, the public IP was 18.130.193.41
    SUBDOMAIN="my-b3-server"
    DUCKDNS_TOKEN="MY-TOKEN-DATA-XXXXXXXXXXX"
    wget "https://www.duckdns.org/update?domains=${SUBDOMAIN}&token=${DUCKDNS_TOKEN}" -q -O -; echo

    # Get the installer script
    git clone https://github.com/bigbluebutton/bbb-install
    cd bbb-install
    sudo bash bbb-install.sh -s "my-b3-server.duckdns.org" -e "myemail@example.com" -g -v xenial-22

    # Create admin user
    cd ~/greenlight
    sudo docker exec greenlight-v2 bundle exec rake admin:create

    # +++
    # At this point we can connect and see that the server works fine, echo test succeeds
    #
    # Installation and greenlight outputs in 01_install_output.txt

# ====================================

# Now we stop the server, and then start it (NOT a quick reboot). AWS will assign a new IP address
aws ec2 stop-instances --instance-ids i-0cfbbe3e05de2e5f6

# Wait until fully stopped
sleep 5
while [[ "$(aws ec2 describe-instances --instance-ids i-0cfbbe3e05de2e5f6|jq .Reservations[0].Instances[0].State.Name -r)" != stopped ]]; do
    sleep 5
done

# Start it afaub
aws ec2 start-instances --instance-ids i-0cfbbe3e05de2e5f6

# +++
# At this point, we can reconnect via browser to the server. Greenlight is up, but when we start/join a session, we get ICE 1007 error

# ====================================

    # On the server, we run the following

    # We update the IP of the domain
    # In my test run, this was now 35.176.97.16
    SUBDOMAIN="my-b3-server"
    DUCKDNS_TOKEN="MY-TOKEN-DATA-XXXXXXXXXXX"
    wget "https://www.duckdns.org/update?domains=${SUBDOMAIN}&token=${DUCKDNS_TOKEN}" -q -O -; echo

    # We check the state of the server
    # Outputs in 02_post_stopstart_checks.txt

    sudo bbb-conf --status
    sudo bbb-conf --restart
    sudo bbb-conf --check
    sudo bbb-conf --status
    sudo ss -tunlp

    # +++
    # At this point, we acknowledge the server is broken. This is to be expected, we have not reconfigured anything
    # (Greenlight is up etc, but tryo to join audio on a session causes ICE 1007 error
    #
    # As an attempt to remedy things, we do the following:
    # (outputs in 03_reconfigure.txt)
    sudo bbb-conf --setip my-b3-server.duckdns.org
    sudo bbb-conf --status
    sudo bbb-conf --check

    # This does not resolve the issue
    # ICE error 1007 still appears when trying to start audio session in browser
    # 
    # ALSO - see the result of `--check` ; some OLD IP entries remain
    # Finally, we grep as much as we can for the old IP
    # Results in 04_files_with_old_ip.txt
    sudo grep -rl "18.130.193.41" --exclude-dir '*docker*' --exclude-dir run --exclude-dir tmp --exclude-dir proc /etc /opt /var /usr 2>/dev/null | tee 04_files_with_old_ip.txt

    # We try to change these manually, excluding og files:
    sudo sed 's/18.130.193.41/35.176.97.16/g' -i $(cat 04_files_with_old_ip.txt|grep -Ev 'log$')
    # And restart (outputs in 05_post_manual_fix.txt)
    sudo bbb-conf --restart
    sudo bbb-conf --status
    sudo bbb-conf --check
    # In greenlight when starting a meeting and attempting to join audio, we get "Could not make a WebSocket connection (error 1002)"

# ====================================

# +++
# At thispoint, the server is probably totally borked.
# Shut it down to avoid extra costs
aws ec2 stop-instances --instance-ids i-0cfbbe3e05de2e5f6

oyekanwahab commented 4 years ago

@taikedz can I suggest something,

here is what I did on one other project that requires a dedicated IP, on an instance that I put off and on when I need (on-demand)

I assigned an elastic IP to the instance, for the duration the instance is on, I will not be charged but when the instance is off, I will be charged on $0.01/hr so in a day the maximum I can be charge is $0.01/hr x 24 = $0.24

in a month the maximum I can be charged for not using the IP is $0.24 X 30 = $7.2

and not all the time I am charged up to that amount, it saves me the stress of going into my application software to modify the static IP all the time

I can still trade $7.2 for some confort

ffdixon commented 4 years ago

After the IP address changes, if you update the DNS entry for my-b3-server.duckdns.org to point to the new public IP address, you should be able to run

sudo bash bbb-install.sh -s "my-b3-server.duckdns.org" -e "myemail@example.com" -g -v xenial-22

again, which will setup BigBlueButton with the new public/private IP address of your server.

taikedz commented 4 years ago

Hi @ffdixon - sorry it was not prominent, but I did mention attempting to do this. Re-running the install script did resolve the issue a couple of times when I attempted it, but after a couple of times doing this, it seems to have no effect, the problem persists despite re-running the installer.

@oyekanwahab - yes this is the route I ended up going down, Elastic IP.

bigbluebutton / bbb-install