Deploy Web Server To AWS or Azure

[philanderson888]

Hello Umesh

How are you?

Just an update with progress on this one.

I am a little stuck.

I have created a script which generates a web server.

The script is here in this commit.

https://github.com/philanderson888/scripts/commit/783d81d46ec7c791ca3ebe34ab056a3e5fecc7f9

Here is the basic code

echo remove old install
rm -rf ~/temp 

echo create new directory
mkdir ~/temp
cd ~/temp

echo scaffold app using    npx express-generator myExpressApp --view ejs
npx express-generator myExpressApp --view ejs -y

cd myExpressApp
npm install
DEBUG=myexpressapp:* npm start

to test it I run

aws/awsWindows/script-54-github-actions-test-app.zsh

echo check express app is running on port 3000
echo curl localhost:3000
curl localhost:3000

and I know its working because on the server I get the following

run server
DEBUG=myexpressapp:* npm start

> myexpressapp@0.0.0 start
> node ./bin/www

  myexpressapp:server Listening on port 3000 +0ms
GET / 200 8.854 ms - 207

so I have the GET request acknowledged in the logging so I know the server has served at least on GET request ... so its working fine

GET / 200 8.854 ms - 207

I have started taking notes on all the github deployment to aws or azure stuff but it's quite complex

devops/github-actions.md

...

so ...

I just need some help !

a) I have created a web server b) not sure what to do next

• am I deploying the web server itself ...
• or
• am I deploying the script which creates the web server

that is my first question!

thanks

[philanderson888]

OK I have made progress overnight sir ...

[philanderson888]

from this commit

https://github.com/philanderson888/scripts/commit/2c9001a273a66e04d6b8927ba212c66426fa62d7

I am getting the following output

so at least i am now building empty docker image and running it, very happy ... will continue this work and next try and put the web server inside the docker container and then run it

==============================================================
==============================================================
====                script 61                          ====
====                script elapsed time 7:08
==============================================================
==============================================================

script-61-dockerfile-01                                                       100%  104     4.6KB/s   00:00    
script 61 to create docker image

cd /home/azureuser

mkdir script-61-docker

mv script-61-dockerfile-01 script-61-docker

cd script-61-docker

mv script-61-dockerfile-01 Dockerfile

export DOCKER_BUILDKIT=0

docker build .
Sending build context to Docker daemon  2.048kB
Step 1/3 : FROM ubuntu:latest
 ---> bf3dc08bfed0
Step 2/3 : RUN apt-get update
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

 ---> Running in 55f8a9a7bae5
Get:1 http://security.ubuntu.com/ubuntu noble-security InRelease [126 kB]
Get:2 http://security.ubuntu.com/ubuntu noble-security/universe amd64 Packages [41.0 kB]
Get:3 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages [107 kB]
Get:4 http://archive.ubuntu.com/ubuntu noble InRelease [256 kB]
Get:5 http://archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
Get:6 http://archive.ubuntu.com/ubuntu noble-backports InRelease [126 kB]
Get:7 http://archive.ubuntu.com/ubuntu noble/universe amd64 Packages [19.3 MB]
Get:8 http://archive.ubuntu.com/ubuntu noble/main amd64 Packages [1808 kB]
Get:9 http://archive.ubuntu.com/ubuntu noble/restricted amd64 Packages [117 kB]
Get:10 http://archive.ubuntu.com/ubuntu noble/multiverse amd64 Packages [331 kB]
Get:11 http://archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages [52.9 kB]
Get:12 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [118 kB]
Get:13 http://archive.ubuntu.com/ubuntu noble-backports/universe amd64 Packages [6387 B]
Fetched 22.5 MB in 3s (7947 kB/s)
Reading package lists...
 ---> Removed intermediate container 55f8a9a7bae5
 ---> 2dc2c63ea91f
Step 3/3 : CMD ["echo", "this is a docker container built by phil anderson"]
 ---> Running in c06b7392e283
 ---> Removed intermediate container c06b7392e283
 ---> fc3ece782b0f
Successfully built fc3ece782b0f

docker build . -t phil:phil
Sending build context to Docker daemon  2.048kB
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Step 1/3 : FROM ubuntu:latest
 ---> bf3dc08bfed0
Step 2/3 : RUN apt-get update
 ---> Using cache
 ---> 2dc2c63ea91f
Step 3/3 : CMD ["echo", "this is a docker container built by phil anderson"]
 ---> Using cache
 ---> fc3ece782b0f
Successfully built fc3ece782b0f
Successfully tagged phil:phil

docker images
REPOSITORY    TAG       IMAGE ID       CREATED         SIZE
phil          phil      fc3ece782b0f   2 seconds ago   112MB
alpine        latest    1d34ffeaf190   9 days ago      7.79MB
debian        latest    5027089adc4c   2 weeks ago     117MB
ubuntu        latest    bf3dc08bfed0   4 weeks ago     76.2MB
hello-world   latest    d2c94e258dcb   13 months ago   13.3kB

run docker image
1dba936f8bc57525b493e5afe830c0e3d3b5492076161531511a910a26ae926f

[philanderson888]

Fresh commit

https://github.com/philanderson888/scripts/commit/9e93da71b110787d8bb06f19574b10fb8e20a41a

Step 11/12 : RUN cd /home/myExpressApp && npm install
Step 12/12 : CMD ["echo", "this is a docker container built by phil anderson"]
Successfully built 9efd5a46692b
Successfully tagged phil2:phil2

created two docker images

the phil2 one I think contains the web app ready to be deployed

docker images

REPOSITORY    TAG       IMAGE ID       CREATED         SIZE
phil2         phil2     9efd5a46692b   1 second ago    983MB
phil          phil      cc6043fbef32   4 minutes ago   983MB
alpine        latest    1d34ffeaf190   10 days ago     7.79MB
debian        latest    5027089adc4c   2 weeks ago     117MB
ubuntu        latest    bf3dc08bfed0   4 weeks ago     76.2MB
hello-world   latest    d2c94e258dcb   13 months ago   13.3kB

[Umesh-Phendarkar]

Okay i see the Dockerfile .. it looks good .. it should work i believe for Web server. now you need to work on CI CD part with Github Actions .

[philanderson888]

Hello Umesh I hope you are well ...

I have been working hard on tidying up my script and I am nearly done on that.

With the deployment of a web app to a cloud provider via docker container I have made good progress ...

I have ...

1) created web app on the MAC, just to check it works 2) created it also on the ubuntu machine that I am running my script on, in the cloud (aws) 3) managed to create a docker image running on ubuntu 4) pushed the web app from my ubuntu cloud machine into the docker image 5) validated that the docker image is valid

here is my output

docker images
REPOSITORY    TAG       IMAGE ID       CREATED         SIZE
phil2         phil2     584bf7b56054   1 second ago    961MB
phil          phil      120775b75a9f   5 minutes ago   961MB
ubuntu        latest    17c0145030df   10 days ago     76.2MB
nginx         latest    4f67c83422ec   10 days ago     188MB
alpine        latest    1d34ffeaf190   2 weeks ago     7.79MB
debian        latest    5027089adc4c   3 weeks ago     117MB
hello-world   latest    d2c94e258dcb   13 months ago   13.3kB

both the images phil and phil2 have the web server code in them

so ... I have a couple of questions

1) why are my custom docker images so large - 961MB compared to one I pulled directly from the internet which is only 76MB which I did not modify? Would that be all the installed javascript libraries taking up all that space? Surely not!

2) so ... what are my next steps? a) How do I test my web app is working fine inside my docker image first, before I deploy it to the cloud provider? b) What is first step in deploying a docker image to the cloud provider?

Thanks

[philanderson888]

Here is my latest output file - it's becoming super huge

https://github.com/philanderson888/scripts/blob/master/aws/awsWindows/20240609-script-out-put.txt

commit

https://github.com/philanderson888/scripts/commit/cc154d941b95f02d0d305fded809a9c78e05078a#diff-41f543775943ba3f664d3b3c33bcc684b2ec398f61f29f9c96fa6442395d81d9R2725

docker stuff starts at line 2700

terraform starts at line 3300

ansible at 4000

github actions at line 4500

install and run a new web app line 4611

push web app to docker line 4661

pushing all the files - maybe there is a more efficient way to do this? line 4680

repeat the whole thing but using a more efficient procedure - just copying the raw files to the docker then installing them whilst on the docker image itself, I think this is a better way - starts at line 5049. ... build docker image 'phil' containing web app and installed packages ...

... seems like the updates to the ubuntu operating system cover many thousands of lines ... and storage ... probably not needed ... lines 5284 to 9014 so that's about 4000+ lines !!! too many !!!

you can see the docker image 'phil' does run at line 9034 - just not testing the web app out ... but the image itself is running which is good ...

repeating again at line 9047 - uploading a node express app this time to docker image 'phil2' ... all the same as above ... lines 9047 to 13473

and again repeated, this time with a super simple (as simple as they come) node web app - the smallest node web app possible - creating docker image 'phil3' from lines 13485 to 13604 ... this one fails ... will fix it

[Umesh-Phendarkar]

Hi Phil, Your Custom Images may contain some layers like you said javascript , due to that it is in large size. Cloud image is Ubuntu image which Ubuntu OS images, it might be Optimized for small size. Second - You can check if your Docker Web App is Working by : -- curl http://localhost:Port # No. of Web App .. from Docker Host. Also from BRowser http://DockerHostIP : Port #No. of Web App .. Like -- http://192.168.1.12:80 . It should show Web Application Page .

[philanderson888]

OK sorry it's taken a while as my focus has been off scripting for a bit ... but ... i have kept it going in the background, did quite a bit of work last weekend which I was very happy with.

Basically summary of status so far is

1. script is much tidier now so all the output is showing exactly as I want the data to be displayed, clear and concise, with timings etc to show how the script is progressing, what has been done and how long each part of the script took.
2. i am now not running all parts of the script every time, but i only run parts of the script every x runs ... so some of the less important stuff does not get run every time ... but it still does get run once in a while ... so i have a balance between keeping these features live ... but not have them clog up every run with useless verbose information which is not required for the task at hand
3. i now have the script in a stable place, no further changes required
4. i now am building 3 servers a) ubuntu for my main server with all services running on it b) second debian server just to test if the web servers i built on the first web server are working. i use the curl command to test each web server and check i get a valid response over the given ip address and port number. this is done on the local azure network, not over the public ip address ( have not got to that stage yet ) c) third redhat server, as I want also to be familiar with building both debian and also redhat linux server flavours, in time i will progress this to centos, opensuse etc ... all the common linux server types ... i want to be able to build and update and run them all ... crazy, huh!!!

... so ... bottom line ... i have finished my crazy work with my crazy script and kind of got it in a stable, clean, working place where all my web servers are building and deploying fine and i can test they are running fine.

next stage ... probably i will want to see if i can get my web servers visible not only on the local azure network but also from the public internet ....

after that .... all those same web servers ... i want to put them in a docker container and run it .... and check if i can see the web server running from another machine on either the local azure network, or from a public ip address on the internet ...

... after that i can worry about deploying the docker container via github actions ... that is where i will need your help i think.

i will get back to you when i start deploying my web servers inside docker containers ... as i may need help there also

I was also thinking we might do a video telephone call some time ... just to talk ... and get your thoughts and opnions on what I am doing wrong, face to face.

thanks

here is latest master script file

https://github.com/philanderson888/scripts/blob/master/azure/bash/masterScript.sh

here is latest output file

https://github.com/philanderson888/scripts/blob/master/azure/bash/20240714-script-out-put.txt

thanks

[philanderson888]

OK I am getting ready to deploy my web servers to docker which is exciting

One thing however which is in my mind is that at present, I think my web servers are available only from within the local azure network and are closed to the outside world, despite me having opened the ports on the server(s).

I think this is something due to the azure cloud setup which by default will close all ports.

I will have to find a way to open the given ports to enable at least a port scan to show the port is open, and to allow a ping or curl test from one server to another to flow via an external IP address.

at present what I have set up is as follows

server 1 runs ubuntu runs apache on port 80 runs various web servers on random ports eg ports 3000 57329 58262 54892 63892 51547 51279 firewall opened for given ports

server 2 runs debian linux runs apache firewall opened for various ports ie 80 3000 57329 58262 54892 63892 51547 51279

curl from server 1 to the web servers running on the various port - test passes curl from server 2 to the web servers on server 1 using azure local network ip - test passes curl from server 2 to the web servers on server 1 using public ip - test fails

[philanderson888]

current nmap scan shows

Starting Nmap 7.40 ( https://nmap.org ) at 2024-07-20 03:50 UTC
Nmap scan report for 172.166.185.125
Host is up (0.070s latency).
PORT     STATE    SERVICE
21/tcp   filtered ftp
22/tcp   open     ssh
23/tcp   filtered telnet
80/tcp   filtered http
110/tcp  filtered pop3
143/tcp  filtered imap
443/tcp  filtered https
3389/tcp filtered ms-wbt-server

so at least we have some ports open which is a good start

[philanderson888]

If I now manually go to azure and just enter a '*' as the source/destination for a new firewall rule, effectively making the server completely open and vulnerable to the internet on all ports ... then I can now see the ports open from the internet

Discovered open port 51547/tcp on 172.166.185.125
Discovered open port 80/tcp on 172.166.185.125
Discovered open port 57329/tcp on 172.166.185.125
Discovered open port 54892/tcp on 172.166.185.125
Discovered open port 58262/tcp on 172.166.185.125
Discovered open port 63892/tcp on 172.166.185.125
Discovered open port 3000/tcp on 172.166.185.125

ok this is super crude and not the way we should be doing it but at least it shows me that i am on the right lines - all i have to do is expose the correct ports on azure for the network security group, and then we are good to go ... happy days ...

[philanderson888]

finally got my web server working - at long last !!! super happy about this !!!

[philanderson888]

this was the network security group rule

echo "====================================================================="
echo "====                 az network nsg rule create                  ===="
echo "====================================================================="
echo az network nsg rule create vm01NGSRuleAllowAll priority 1010 Open to internet
az network nsg rule create -g $resource_group_name --nsg-name vm01NSG --name vm01NGSRuleAllowAll --priority 1010 --source-address-prefixes '*' --source-port-ranges '*' --destination-address-prefixes '*' --destination-port-ranges '*' --access Allow --protocol Tcp --description "Open to internet" --output table
echo
echo
echo

this was the commit

https://github.com/philanderson888/scripts/commit/2f4f189c57ff7ff72d60cddafdbe038d7a1a8ba4

[philanderson888]

plus i am using github copilot and find it super helpful and a great time saver!!! I don't know if you have ever tried it???

[philanderson888]

so .. .super happy ... i have

express js running on port 3000

express js running on port 51547

typescript built using vite on port 54892

react app built with vite on port 58262

vue with vite js running on port 63892

bun js app built on port 57329

[philanderson888]

finally @Umesh-Phendarkar i have my web servers automatically deployed and open to the internet ... super happy ....

[philanderson888]

the script also automatically opens the web servers on my mac machine so i can see the sites running

https://github.com/philanderson888/scripts/commit/e5aaaac99d0465e6e85ccbaaf81bc25c906a0de1

[Umesh-Phendarkar]

Wow... IT is Looks Nice .. Very Good Work ..you did Phil...

[philanderson888]

OK I have started to look at deploying a web app inside a docker container and exposing it on a port

[philanderson888]

just to let you know I am closing out on how to deploy a web app in a docker container; just struggling with all the Dockerfile commands - which ones to run and in which order ... I will work it through, thanks