search

FIWARE / tutorials.TourGuide-App

:mortar_board: FIWARE Tour Guide Application.
https://fiwaretourguide.readthedocs.io/
MIT License
17 stars 19 forks source link

Use official FIWARE images for the different GEs #43

Closed jmcanterafonseca closed 8 years ago

albertinisg commented 8 years ago

The status of the official GEs that are (still) not being used in the TourGuide is the following:

Due to all of this, I think we are still far to migrate to all the official ones, as it will require a lot of changes. We are using 10 different containers and, if we are not sure of the proper functioning of the official images, we cannot provide a stable service in our application. In my opinion, using the tagging system in Dockerhub is a must for all the GEs.

jmcanterafonseca commented 8 years ago

thanks for the report. I've reported to the involved GEs

Wrt IdM and PEP Proxy, please could you report on the status right here in this issue so that we can analyze and take action?

thanks

frbattid commented 8 years ago

Cygnus related issue --> https://github.com/telefonicaid/fiware-cygnus/issues/536 (reopened)

urishani commented 8 years ago

CEP issue has been handled. askbot answered too.

albertinisg commented 8 years ago

@jmcanterafonseca sure. @dmabtrg, let's follow up the update process of IdM and PEP through this thread.

albertinisg commented 8 years ago

@frbattid @urishani maybe you can find useful the approach we already did for Cygnus and CEP images. Those images are built automatically, with tags support and a bit lighter than the official ones :relaxed:

urishani commented 8 years ago

Not clear top me what you want to do here, and what you did with CEP iamage and in what context. Presently, we have a simple process to generate the Docker image, or any local installation of the product. Please clarify,

From: Alberto Martín notifications@github.com To: "Fiware/tutorials.TourGuide-App" tutorials.TourGuide-App@noreply.github.com Cc: Uri Shani/Haifa/IBM@IBMIL Date: 16/03/2016 05:55 PM Subject: Re: [tutorials.TourGuide-App] Use official FIWARE images for the different GEs (#43)

@frbattid @urishani maybe you can find useful the approach we already did for Cygnus and CEP images. Those images are built automatically, with tags support and a bit lighter than the official ones ? You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

albertinisg commented 8 years ago

@urishani I'm just pointing out the image we had to create based in the FIWARE Guidelines to use in the TourGuide, as in DockerHub there was no CEP image available. I've just thought you could find it useful, as the image is also lighter (517 MB vs 189 MB).

urishani commented 8 years ago

Thanks Alberto, I looked at that link and I think we are inline with the guidelines. Yet, indeed our image is almost 900 MB. I know that we have in it an initialized tomcat which spans out the war of applications and that may increase the size. Working on reducing the size - at least based on that idea.

Any other ideas on how to make the image smaller?

In our case, the initialization creates bad applications and thus needed to be avoided anyway.

Thanks,

From: Alberto Martín notifications@github.com To: "Fiware/tutorials.TourGuide-App" tutorials.TourGuide-App@noreply.github.com Cc: Uri Shani/Haifa/IBM@IBMIL Date: 16/03/2016 06:47 PM Subject: Re: [tutorials.TourGuide-App] Use official FIWARE images for the different GEs (#43)

@urishani I'm just pointing out the image we had to create based in the FIWARE Guidelines to use in the TourGuide, as in DockerHub there was no CEP image available. I've just thought you could find it useful, as the image is also lighter (517 MB vs 189 MB). ? You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

dmabtrg commented 8 years ago

Hi @urishani

Yet, indeed our image is almost 900 MB. I know that we have in it an initialized tomcat which spans out the war of applications and that may increase the size. Working on reducing the size - at least based on that idea. Any other ideas on how to make the image smaller?

Using docker history <image> you can see which layers increase the size of the image. Keep in mind that adding something in a layer and removing later in another layer does not reduce the image size, so if you need to do that, try to do it in the same layer. Also every directive in the Dockerfile creates a new layer.

If you need to download a file on the Dockerfile, you can use the ADD directive instead of having to install wget/curl to download that, i.e. in your current Dockerfile you can change this:

RUN apt-get update && apt-get -y upgrade
RUN apt-get install -y wget bash curl
RUN wget https://raw.githubusercontent.com/ishkin/Proton/master/puppet/miscellaneous/CEP_Install_via_puppet.sh

to this:

ADD https://raw.githubusercontent.com/ishkin/Proton/master/puppet/miscellaneous/CEP_Install_via_puppet.sh /CEP_Install_via_puppet.sh

Instead of installing java and tomcat you could use the official tomcat image that already has the neccesary packages installed and automatically starts tomcat when creating a container (that's the one we used for our docker image).

Btw, I tried building the image with the current Dockerfile and, though it does not fail to build the image, puppet does not seem to install any of the required packages. I've opened an issue about this (ishkin/Proton#34).

urishani commented 8 years ago

Thanks for all these tips. We will apply them, and work out the other wrinkles of this component. Thanks,

From: dmabtrg notifications@github.com To: "Fiware/tutorials.TourGuide-App" tutorials.TourGuide-App@noreply.github.com Cc: Uri Shani/Haifa/IBM@IBMIL Date: 17/03/2016 04:02 PM Subject: Re: [tutorials.TourGuide-App] Use official FIWARE images for the different GEs (#43)

Hi @urishani Yet, indeed our image is almost 900 MB. I know that we have in it an initialized tomcat which spans out the war of applications and that may increase the size. Working on reducing the size - at least based on that idea. Any other ideas on how to make the image smaller? Using docker history you can see which layers increase the size of the image. Keep in mind that adding something in a layer and removing later in another layer does not reduce the image size, so if you need to do that, try to do it in the same layer. Also every directive in the Dockerfile creates a new layer. If you need to download a file on the Dockerfile, you can use the ADD directive instead of having to install wget/curl to download that, i.e. in your current Dockerfile you can change this: RUN apt-get update && apt-get -y upgrade RUN apt-get install -y wget bash curl RUN wget https://raw.githubusercontent.com/ishkin/Proton/master/puppet/miscellaneous/CEP_Install_via_puppet.sh

to this: ADD https://raw.githubusercontent.com/ishkin/Proton/master/puppet/miscellaneous/CEP_Install_via_puppet.sh /CEP_Install_via_puppet.sh

Instead of installing java and tomcat you could use the official tomcat image that already has the neccesary packages installed and automatically starts tomcat when creating a container (that's the one we used for our docker image). Btw, I tried building the image with the current Dockerfile and, though it does not fail to build the image, puppet does not seem to install any of the required packages. I've opened an issue about this (ishkin/Proton#34). ? You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

urishani commented 8 years ago

Hi Uri, These steps are correct and are the best practice of Docker files, however in our case it is reduced few MB (tried it) as the main job of the build is the BASH script that used NON docker configuration, we used it to avoid of updates and any changes in few places. Regarding the Tomcan image, the image is a Debian based image and its required the CEP... script customization and more packages installed on the Tomcat image that will not reduce the space for much.   As you can see in the current image the space is taken by the directories  278M    var ---128M    lib ----95M     tomcat7 537M    usr ---114M    share---393M    lib ---298M    jvm  Almost 50% of the image taken by the libraries JVM and tomcat (Data), so as these directories needed by the application any extreme optimization will not reduce much the image size. In my opinion that do the whole process in the Dockerfile will reduce no more than 50-60M (If even) that doesnt worth the maintenance and the work to create the Dockerfile. Thanks. Regards, Pavel Druyan Technologies Team Phone: +972-48281444 | Mobile: +972-54-7612998 | Fax: +972-4-8296111 Haifa University, Mount Carmel Haifa, 31905 Israel
      ----- Original message -----From: Uri Shani/Haifa/IBMTo: "Fiware/tutorials.TourGuide-App" reply@reply.github.comCc: Amir Malki/Haifa/IBM@IBMIL, Pavel Druyan/Haifa/IBM@IBMIL, Inna Skarbovsky1/Haifa/IBM@IBMILSubject: Re: [tutorials.TourGuide-App] Use official FIWARE images for the different GEs (#43)Date: Sat, Mar 19, 2016 12:03 AM Thanks for all these tips. We will apply them, and work out the other wrinkles of this component.Thanks,- URidmabtrg ---17/03/2016 04:02:09 PM---Hi @urishani > Yet, indeed our image is almost 900 MB. I know that we have in it an initialized tomcFrom: dmabtrg notifications@github.comTo: "Fiware/tutorials.TourGuide-App" tutorials.TourGuide-App@noreply.github.comCc: Uri Shani/Haifa/IBM@IBMILDate: 17/03/2016 04:02 PMSubject: Re: [tutorials.TourGuide-App] Use official FIWARE images for the different GEs (#43) Hi @urishani Yet, indeed our image is almost 900 MB. I know that we have in it an initialized tomcat which spans out the war of applications and that may increase the size. Working on reducing the size - at least based on that idea. Any other ideas on how to make the image smaller?Using docker history  you can see which layers increase the size of the image. Keep in mind that adding something in a layer and removing later in another layer does not reduce the image size, so if you need to do that, try to do it in the same layer. Also every directive in the Dockerfile creates a new layer. If you need to download a file on the Dockerfile, you can use the ADD directive instead of having to install wget/curl to download that, i.e. in your current Dockerfile you can change this: RUN apt-get update && apt-get -y upgradeRUN apt-get install -y wget bash curlRUN wget https://raw.githubusercontent.com/ishkin/Proton/master/puppet/miscellaneous/CEP_Install_via_puppet.sh to this: ADD https://raw.githubusercontent.com/ishkin/Proton/master/puppet/miscellaneous/CEP_Install_via_puppet.sh /CEP_Install_via_puppet.sh Instead of installing java and tomcat you could use the official tomcat image that already has the neccesary packages installed and automatically starts tomcat when creating a container (that's the one we used for our docker image). Btw, I tried building the image with the current Dockerfile and, though it does not fail to build the image, puppet does not seem to install any of the required packages. I've opened an issue about this (ishkin/Proton#34). —You are receiving this because you were mentioned.Reply to this email directly or view it on GitHub    

urishani commented 8 years ago

Hi Albert, Changing the dockerfile to do ADD as you suggest indeed reduces the image to below 200MB. Yet, running it fails to create the tomcat service we need.

My developers do not know how to do that differently. Please explain what is your suggestion or how the resulting docker image can still be used at its much reduced size. Thanks for your help here.

The modified Dockerfile is:

Update Ubuntu

FROM ubuntu:latest RUN apt-get update && apt-get -y upgrade RUN apt-get install -y wget bash curl ADD https://raw.githubusercontent.com/ishkin/Proton/master/puppet/miscellaneous/CEP_Install_via_puppet.sh \ /CEP_Install_via_puppet.sh RUN apt-get clean EXPOSE 8080 RUN mkdir -p /root/Proton/ RUN wget https://raw.githubusercontent.com/ishkin/Proton/master/docker/docker_smoketest.sh -q -O root/Proton/docker_smoketest.sh `

  • Uri
dmabtrg commented 8 years ago

This is the status of the official images of the GEs used in the Fiware TourGuide Application:

dmabtrg commented 8 years ago

The old IDAS IoT Agent is being replaced with a new IoT Agent. This is the status of the official image:

I'm currently testing this image.

jmcanterafonseca commented 8 years ago

wrt to the IDM image, I think we need to use the official one. The automation of provisioning of users, authorization etc. should be done by a configuration script, or even manually by users themselves, to be run after the service is launched. This is an important point for the tour guide Application. We don't want to hide our potential users all the configuration details, all the opposite, we want them to learn how to do things.

This is an important change in the philosophy of the Tour Guide Application. 0 magic

albertinisg commented 8 years ago

This issue was fixed in https://github.com/Fiware/tutorials.TourGuide-App/pull/108 and https://github.com/Fiware/tutorials.TourGuide-App/pull/118 by @dmabtrg