search

philanderson888 / scripts

MIT License
0 stars 0 forks source link

Deploy Web Server To AWS or Azure #4

Open philanderson888 opened 1 month ago

philanderson888 commented 1 month ago

Hello Umesh

How are you?

Just an update with progress on this one.

I am a little stuck.

I have created a script which generates a web server.

The script is here in this commit.

https://github.com/philanderson888/scripts/commit/783d81d46ec7c791ca3ebe34ab056a3e5fecc7f9

Here is the basic code

echo remove old install
rm -rf ~/temp 

echo create new directory
mkdir ~/temp
cd ~/temp

echo scaffold app using    npx express-generator myExpressApp --view ejs
npx express-generator myExpressApp --view ejs -y

cd myExpressApp
npm install
DEBUG=myexpressapp:* npm start

to test it I run

aws/awsWindows/script-54-github-actions-test-app.zsh

echo check express app is running on port 3000
echo curl localhost:3000
curl localhost:3000

and I know its working because on the server I get the following

run server
DEBUG=myexpressapp:* npm start

> myexpressapp@0.0.0 start
> node ./bin/www

  myexpressapp:server Listening on port 3000 +0ms
GET / 200 8.854 ms - 207

so I have the GET request acknowledged in the logging so I know the server has served at least on GET request ... so its working fine 

GET / 200 8.854 ms - 207

I have started taking notes on all the github deployment to aws or azure stuff but it's quite complex

devops/github-actions.md

...

so ...

I just need some help !

a) I have created a web server b) not sure what to do next

that is my first question!

thanks

philanderson888 commented 1 month ago

OK I have made progress overnight sir ...

philanderson888 commented 1 month ago

from this commit

https://github.com/philanderson888/scripts/commit/2c9001a273a66e04d6b8927ba212c66426fa62d7

I am getting the following output

so at least i am now building empty docker image and running it, very happy ... will continue this work and next try and put the web server inside the docker container and then run it 

==============================================================
==============================================================
====                script 61                          ====
====                script elapsed time 7:08
==============================================================
==============================================================

script-61-dockerfile-01                                                       100%  104     4.6KB/s   00:00    
script 61 to create docker image

cd /home/azureuser

mkdir script-61-docker

mv script-61-dockerfile-01 script-61-docker

cd script-61-docker

mv script-61-dockerfile-01 Dockerfile

export DOCKER_BUILDKIT=0

docker build .
Sending build context to Docker daemon  2.048kB
Step 1/3 : FROM ubuntu:latest
 ---> bf3dc08bfed0
Step 2/3 : RUN apt-get update
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

 ---> Running in 55f8a9a7bae5
Get:1 http://security.ubuntu.com/ubuntu noble-security InRelease [126 kB]
Get:2 http://security.ubuntu.com/ubuntu noble-security/universe amd64 Packages [41.0 kB]
Get:3 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages [107 kB]
Get:4 http://archive.ubuntu.com/ubuntu noble InRelease [256 kB]
Get:5 http://archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
Get:6 http://archive.ubuntu.com/ubuntu noble-backports InRelease [126 kB]
Get:7 http://archive.ubuntu.com/ubuntu noble/universe amd64 Packages [19.3 MB]
Get:8 http://archive.ubuntu.com/ubuntu noble/main amd64 Packages [1808 kB]
Get:9 http://archive.ubuntu.com/ubuntu noble/restricted amd64 Packages [117 kB]
Get:10 http://archive.ubuntu.com/ubuntu noble/multiverse amd64 Packages [331 kB]
Get:11 http://archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages [52.9 kB]
Get:12 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [118 kB]
Get:13 http://archive.ubuntu.com/ubuntu noble-backports/universe amd64 Packages [6387 B]
Fetched 22.5 MB in 3s (7947 kB/s)
Reading package lists...
 ---> Removed intermediate container 55f8a9a7bae5
 ---> 2dc2c63ea91f
Step 3/3 : CMD ["echo", "this is a docker container built by phil anderson"]
 ---> Running in c06b7392e283
 ---> Removed intermediate container c06b7392e283
 ---> fc3ece782b0f
Successfully built fc3ece782b0f

docker build . -t phil:phil
Sending build context to Docker daemon  2.048kB
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Step 1/3 : FROM ubuntu:latest
 ---> bf3dc08bfed0
Step 2/3 : RUN apt-get update
 ---> Using cache
 ---> 2dc2c63ea91f
Step 3/3 : CMD ["echo", "this is a docker container built by phil anderson"]
 ---> Using cache
 ---> fc3ece782b0f
Successfully built fc3ece782b0f
Successfully tagged phil:phil

docker images
REPOSITORY    TAG       IMAGE ID       CREATED         SIZE
phil          phil      fc3ece782b0f   2 seconds ago   112MB
alpine        latest    1d34ffeaf190   9 days ago      7.79MB
debian        latest    5027089adc4c   2 weeks ago     117MB
ubuntu        latest    bf3dc08bfed0   4 weeks ago     76.2MB
hello-world   latest    d2c94e258dcb   13 months ago   13.3kB

run docker image
1dba936f8bc57525b493e5afe830c0e3d3b5492076161531511a910a26ae926f
philanderson888 commented 1 month ago

Fresh commit

https://github.com/philanderson888/scripts/commit/9e93da71b110787d8bb06f19574b10fb8e20a41a

Step 11/12 : RUN cd /home/myExpressApp && npm install
Step 12/12 : CMD ["echo", "this is a docker container built by phil anderson"]
Successfully built 9efd5a46692b
Successfully tagged phil2:phil2

created two docker images

the phil2 one I think contains the web app ready to be deployed

docker images

REPOSITORY    TAG       IMAGE ID       CREATED         SIZE
phil2         phil2     9efd5a46692b   1 second ago    983MB
phil          phil      cc6043fbef32   4 minutes ago   983MB
alpine        latest    1d34ffeaf190   10 days ago     7.79MB
debian        latest    5027089adc4c   2 weeks ago     117MB
ubuntu        latest    bf3dc08bfed0   4 weeks ago     76.2MB
hello-world   latest    d2c94e258dcb   13 months ago   13.3kB
Umesh-Phendarkar commented 1 month ago

Okay i see the Dockerfile .. it looks good .. it should work i believe for Web server. now you need to work on CI CD part with Github Actions .

philanderson888 commented 1 month ago

Hello Umesh I hope you are well ...

I have been working hard on tidying up my script and I am nearly done on that.

With the deployment of a web app to a cloud provider via docker container I have made good progress ...

I have ...

1) created web app on the MAC, just to check it works 2) created it also on the ubuntu machine that I am running my script on, in the cloud (aws) 3) managed to create a docker image running on ubuntu 4) pushed the web app from my ubuntu cloud machine into the docker image 5) validated that the docker image is valid

here is my output

docker images
REPOSITORY    TAG       IMAGE ID       CREATED         SIZE
phil2         phil2     584bf7b56054   1 second ago    961MB
phil          phil      120775b75a9f   5 minutes ago   961MB
ubuntu        latest    17c0145030df   10 days ago     76.2MB
nginx         latest    4f67c83422ec   10 days ago     188MB
alpine        latest    1d34ffeaf190   2 weeks ago     7.79MB
debian        latest    5027089adc4c   3 weeks ago     117MB
hello-world   latest    d2c94e258dcb   13 months ago   13.3kB

both the images phil and phil2 have the web server code in them

so ... I have a couple of questions

1) why are my custom docker images so large - 961MB compared to one I pulled directly from the internet which is only 76MB which I did not modify? Would that be all the installed javascript libraries taking up all that space? Surely not!

2) so ... what are my next steps? a) How do I test my web app is working fine inside my docker image first, before I deploy it to the cloud provider? b) What is first step in deploying a docker image to the cloud provider?

Thanks

philanderson888 commented 1 month ago

Here is my latest output file - it's becoming super huge

https://github.com/philanderson888/scripts/blob/master/aws/awsWindows/20240609-script-out-put.txt

commit

https://github.com/philanderson888/scripts/commit/cc154d941b95f02d0d305fded809a9c78e05078a#diff-41f543775943ba3f664d3b3c33bcc684b2ec398f61f29f9c96fa6442395d81d9R2725

docker stuff starts at line 2700

terraform starts at line 3300

ansible at 4000

github actions at line 4500

install and run a new web app line 4611

push web app to docker line 4661

pushing all the files - maybe there is a more efficient way to do this? line 4680

repeat the whole thing but using a more efficient procedure - just copying the raw files to the docker then installing them whilst on the docker image itself, I think this is a better way - starts at line 5049. ... build docker image 'phil' containing web app and installed packages ...

... seems like the updates to the ubuntu operating system cover many thousands of lines ... and storage ... probably not needed ... lines 5284 to 9014 so that's about 4000+ lines !!! too many !!!

you can see the docker image 'phil' does run at line 9034 - just not testing the web app out ... but the image itself is running which is good ...

repeating again at line 9047 - uploading a node express app this time to docker image 'phil2' ... all the same as above ... lines 9047 to 13473

and again repeated, this time with a super simple (as simple as they come) node web app - the smallest node web app possible - creating docker image 'phil3' from lines 13485 to 13604 ... this one fails ... will fix it

Umesh-Phendarkar commented 1 month ago

Hi Phil, Your Custom Images may contain some layers like you said javascript , due to that it is in large size. Cloud image is Ubuntu image which Ubuntu OS images, it might be Optimized for small size. Second - You can check if your Docker Web App is Working by : -- curl http://localhost:Port # No. of Web App .. from Docker Host. Also from BRowser http://DockerHostIP : Port #No. of Web App .. Like -- http://192.168.1.12:80 . It should show Web Application Page .

philanderson888 commented 4 days ago

OK sorry it's taken a while as my focus has been off scripting for a bit ... but ... i have kept it going in the background, did quite a bit of work last weekend which I was very happy with.

Basically summary of status so far is

  1. script is much tidier now so all the output is showing exactly as I want the data to be displayed, clear and concise, with timings etc to show how the script is progressing, what has been done and how long each part of the script took.
  2. i am now not running all parts of the script every time, but i only run parts of the script every x runs ... so some of the less important stuff does not get run every time ... but it still does get run once in a while ... so i have a balance between keeping these features live ... but not have them clog up every run with useless verbose information which is not required for the task at hand
  3. i now have the script in a stable place, no further changes required
  4. i now am building 3 servers a) ubuntu for my main server with all services running on it b) second debian server just to test if the web servers i built on the first web server are working. i use the curl command to test each web server and check i get a valid response over the given ip address and port number. this is done on the local azure network, not over the public ip address ( have not got to that stage yet ) c) third redhat server, as I want also to be familiar with building both debian and also redhat linux server flavours, in time i will progress this to centos, opensuse etc ... all the common linux server types ... i want to be able to build and update and run them all ... crazy, huh!!!

... so ... bottom line ... i have finished my crazy work with my crazy script and kind of got it in a stable, clean, working place where all my web servers are building and deploying fine and i can test they are running fine.

next stage ... probably i will want to see if i can get my web servers visible not only on the local azure network but also from the public internet ....

after that .... all those same web servers ... i want to put them in a docker container and run it .... and check if i can see the web server running from another machine on either the local azure network, or from a public ip address on the internet ...

... after that i can worry about deploying the docker container via github actions ... that is where i will need your help i think.

i will get back to you when i start deploying my web servers inside docker containers ... as i may need help there also

I was also thinking we might do a video telephone call some time ... just to talk ... and get your thoughts and opnions on what I am doing wrong, face to face.

thanks

here is latest master script file

https://github.com/philanderson888/scripts/blob/master/azure/bash/masterScript.sh

here is latest output file

https://github.com/philanderson888/scripts/blob/master/azure/bash/20240714-script-out-put.txt

thanks