Closed nachfuellbar closed 6 years ago
atomicturtle
commented
6 years ago Ah, thats good to know, I think I follow this. Do you think you could send a PR to the Dockerfile (presumably its just in that?) or if you prefer just dump it in a comment here. Thanks again!
nachfuellbar
commented
6 years ago I will look into this Some quick things should be easy to change but i'm pretty sure i'm not experienced enough for a perfect image ;) but at least we could get rid of yum cache and maybe more unnecessary things
to get rid of your cache in every layer, you could add "&& rm -rf /var/cache/yum/*" (or at least "&& yum clean all") to the end of each RUN with an yum command
your RUN for syncing scap, nvt and cert isn't that good either in your current image you include pretty old definitons which get even older with every passing day maybe only update at runtime would be better (you even update this at runtime in run.sh)
atomicturtle
commented
6 years ago Oh yeah, the SCAP & CERTDATA steps in this can take hours (sometimes as long as 12!) to run based on the speed of the upstream mirror, time of day, connection failures, etc. Thats the single biggest bottleneck on this.
So back to the RUN parts, this could all be consolidated into a single script that runs all these steps, and possibly all but one of the ADD lines (IE for the script). That'd leave it looking something like:
ADD openvas-docker-setup.sh RUN openvas-docker-setup.sh CMD /run.sh EXPOSE 443
What do you think?
nachfuellbar
commented
6 years ago That would be a pretty good idea, if you could load everything in this RUN script from external resources (e.g. http uri's) But i assume that's what you wanted to do ;)
but if you are planing to embed many things in this script (hopefully none base64 encoded things) you would add a pretty huge ADD layer
Another question which came to my mind last night; i happend to run into the error that i couldn't update definitions at all (behind firewall and proxy) maybe an note in readme for setting RSYNC_PROXY in container would be good
atomicturtle
commented
6 years ago Nah, it'd just be all those commands as separate RUN lines rolled up into a single script. Yum does all the real work here
atomicturtle
commented
6 years ago Ok first test is done, TLDR: 252MB or 6.91% reduction in size:
Before:
d8389ae0956c 8 months ago /bin/sh -c #(nop) EXPOSE 443/tcp 0 B
428cb57c660a 8 months ago /bin/sh -c #(nop) CMD ["/bin/sh" "-c" "/r... 0 B
4ab8135bc1ad 8 months ago |2 HTTPPROXY=http://172.17.0.1:3128 http... 0 B
dccff1d9c7c1 8 months ago |2 HTTPPROXY=http://172.17.0.1:3128 http... 1.87 GB
1f84fe46358d 9 months ago |2 HTTPPROXY=http://172.17.0.1:3128 http... 490 MB
9729051ea078 9 months ago |2 HTTPPROXY=http://172.17.0.1:3128 http... 236 MB
a914782f4ea5 9 months ago |2 HTTPPROXY=http://172.17.0.1:3128 http... 692 kB
2cc79e3ceacf 9 months ago /bin/sh -c #(nop) ADD file:e2237e1b3510830... 10.2 kB
6e5629b0067c 9 months ago |2 HTTPPROXY=http://172.17.0.1:3128 http... 0 B
7d0a91d24d93 9 months ago |2 HTTPPROXY=http://172.17.0.1:3128 http... 722 MB
69eeda198097 9 months ago |2 HTTPPROXY=http://172.17.0.1:3128 http... 20.5 MB
4be5c4eaf2db 9 months ago |2 HTTPPROXY=http://172.17.0.1:3128 http... 107 MB
b2544781f85a 9 months ago /bin/sh -c #(nop) ADD file:29b40c24ac18205... 174 B
d5684fd36ab2 9 months ago /bin/sh -c #(nop) ADD file:a822076c357ca64... 46.8 kB
2929154823a0 9 months ago /bin/sh -c #(nop) ADD file:c09b9272df2a931... 177 B
4276b73a48b5 9 months ago /bin/sh -c #(nop) ADD file:dba319a507743ec... 2.37 kB
82d0a3796508 9 months ago /bin/sh -c #(nop) MAINTAINER Support <sup... 0 B
196e0ce0c9fb 10 months ago /bin/sh -c #(nop) CMD ["/bin/bash"] 0 B
nachfuellbar
commented
6 years ago Nice to see such a saving in memory
You build this with openscap data? I'm curious how much space could be saved if this wouldn't be done.
For the update timeframe, i'm pretty sure you didn't use RSYNC_PROXY in your enviroment. The updates take a really long time if no RSYNC_PROXY is used because it would wait for timeouts really long (at least for me)
With RSYNC_PROXY set, it took about 5 minutes to start this container (including updates for nvt, etc) I used a Notebook with i5-5200U and most of the time i was waiting for starting of openvas itself.
atomicturtle
commented
6 years ago Yeah SCAP feed is a requirement for the other folks that use it (.gov/.mil) The latest iteration is up on docker hub now so I'll close this one out. Thanks for the help!
nachfuellbar
commented
6 years ago Thanks for your fast response ;)
But i'm still curious why this feed should be included in docker image. Wouldn't it be better to use a Volume for this?
I don't want to annoy you, i just want to know if i hadn't thought about some things ;)
atomicturtle
commented
6 years ago Speed mainly, and maintenance. The audience it was designed for 1) needed the container to be functional immediately 2) cant touch the contents of the container (so no volumes) 3) Be FIPS-140-2 compliant. That doesnt mean you couldnt pin a volume to it like it shows with the mgr settings in the README. Rather than clutter up this issue, lets take this to a new issue where we can cover some of your rsync ideas.
atomicturtle
commented
6 years ago For reference
the Rsync idea you had: #4 Volume support for SCAP/NVT/CERT data #5
hello guys,
at first thanks for creating this project. But i have to admit, while downloading your docker image, i saw how many layers you created. You really should adjust your image file.
Each RUN and ADD adds a layer. The best layer to explain what's wrong with it is
RUN rm -rf /var/cache/yum/*In your layers before, you created "trash" in your yum cache which you wanted to delete. But the files of your cache are in your layers before. With this layer you just "delete" this files for the running container but your cache is still present in your previous layers.
You could use
docker history atomicorp/openvas --no-truncto have a further look into this (e.g. this shows you "rm -rf" layer is 0 bytes, but you have to load many huge layers before)