search

cve-search / vulnerability-lookup

Vulnerability-Lookup facilitates quick correlation of vulnerabilities from various sources, independent of vulnerability IDs, and streamlines the management of Coordinated Vulnerability Disclosure (CVD).
https://vulnerability.circl.lu
GNU Affero General Public License v3.0
136 stars 16 forks source link

quick and dirty Dockerfile #37

Open svniedner opened 7 months ago

svniedner commented 7 months ago

I have added a quick-and-dirty Dockerfile which serves well for trying the software, but is not really suitable for production use. This would require to move all directories with persistence on docker volumes. This would require your support, and I am happy to develop and share the code then. Best, Sven

Rafiot commented 7 months ago

Thank you for the work!

It probably makes sense to use the redis (or more probably valkey) and kvrocks docker images instead of building from scratch in the docker. And regarding the docker volumes, what do you need from me?

svniedner commented 7 months ago

Hi Raphael,

yes, it certainly does. This is something I wanted to discuss with you going forward, as well as how persistence is organized.

I saw that you recommend building kvrocks from the source, is this a requirement because you need some header files, specific versions, etc.?

What would be great if all data that keeps some kind of state is found in one directory (where then a docker volume needs to be mounted). If this is too complicated, I can try to work around it using symlinks to a specific /data directory.

Then the entire thing could be turned into an organized stack, which would make it more suitable for production environments.

Best,

Sven

Von: Raphaël Vinot @.> Datum: Mittwoch, 24. April 2024 um 09:20 An: cve-search/vulnerability-lookup @.> Cc: Sven Niedner @.>, Author @.> Betreff: Re: [cve-search/vulnerability-lookup] quick and dirty Dockerfile (PR #37)

Thank you for the work!

It probably makes sense to use the redis (or more probably valkey) and kvrocks docker images instead of building from scratch in the docker. And regarding the docker volumes, what do you need from me?

— Reply to this email directly, view it on GitHubhttps://github.com/cve-search/vulnerability-lookup/pull/37#issuecomment-2074247537, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AV7DNCO4Y336ZBWIU5NWKZDY65MKRAVCNFSM6AAAAABGU5ZGECVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZUGI2DONJTG4. You are receiving this because you authored the thread.Message ID: @.***>

Rafiot commented 7 months ago

we were waiting for valkey to be stable before switching to it. It seems to be the case now.

The reason the default install guide tells you to install kvrocks and redis from sources is just because the versions in the distributions are too often very outdated and/or no-existant. This is not the case if you install from docker. We mostly need relatively recent, but not a specific version.

Regarding the feeders directoires, I'm guessing the approach is to go with docker compose -> https://github.com/Lookyloo/lookyloo/blob/main/docker-compose.yml ?

I'm not a docker user, and will not be using it in production settings, so I'll try to make your life as easy as possible, but docker will not be the default approach to use the tool.

svniedner commented 7 months ago

Hi Raphael,

thank you for the information. I will then experiment with the docker images for kvrocks and redis, I think this should be feasible.

I will indeed put the entire configuration into a docker-compose file, that will then start up the containers. For me, this is a very efficient way to set up a dev environment on my macbook with almost zero hassle, so the benefit is not only in operations.

In the file you sent, the volumses section defines the persistent storage directories. In this cases, there are several, which in larger installation gives you the option to put them into different storage classes.

From what I see in the code, there currently is a storage directory for each feeder, like ./vulnerabilitylookup/feeders/cvelistv5/cve. For organization of storage it would be ideal if these could move to something like ./vulnerabiltiylookup/persistence/cve. Then I can create a docker volume for the persistence folder.

Could it be that the content of theses directories is also tracked in git? I would suggest to .gitignore this data, as this reduces the size of the repo, and can then be downloaded on first startup.

Best,

Sven

Von: Raphaël Vinot @.> Datum: Mittwoch, 24. April 2024 um 09:31 An: cve-search/vulnerability-lookup @.> Cc: Sven Niedner @.>, Author @.> Betreff: Re: [cve-search/vulnerability-lookup] quick and dirty Dockerfile (PR #37)

we were waiting for valkey to be stable before switching to it. It seems to be the case now.

The reason the default install guide tells you to install kvrocks and redis from sources is just because the versions in the distributions are too often very outdated and/or no-existant. This is not the case if you install from docker. We mostly need relatively recent, but not a specific version.

Regarding the feeders directoires, I'm guessing the approach is to go with docker compose -> https://github.com/Lookyloo/lookyloo/blob/main/docker-compose.yml ?

I'm not a docker user, and will not be using it in production settings, so I'll try to make your life as easy as possible, but docker will not be the default approach to use the tool.

— Reply to this email directly, view it on GitHubhttps://github.com/cve-search/vulnerability-lookup/pull/37#issuecomment-2074268014, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AV7DNCMLD6SDAQ4763UPMYLY65NUNAVCNFSM6AAAAABGU5ZGECVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZUGI3DQMBRGQ. You are receiving this because you authored the thread.Message ID: @.***>

Rafiot commented 7 months ago

The main reason I don't use docker by default is that it is an absolute pain when I write code on the project (and it makes debugging a nightmare).

Moving the persistent directories somewhere else is possible, but it will break the existing installations, but it will be in a docker-context much more future-proof. I'll see what I can do there.

And regarding the content tracked in git, it is for some, as submodules, so they should not be fetched by default until you run the importers.

svniedner commented 7 months ago

Hi Vinot,

a non-breaking way to deal with the situation would be to create symlinks for the persistence directories.

It could still induce problems for existing installations, but they are easier to manage.

What I will do now is looking into kvrocks/redis docker images, probably over the weekend.

Best, Sven

Von: Raphaël Vinot @.> Datum: Mittwoch, 24. April 2024 um 09:58 An: cve-search/vulnerability-lookup @.> Cc: Sven Niedner @.>, Author @.> Betreff: Re: [cve-search/vulnerability-lookup] quick and dirty Dockerfile (PR #37)

The main reason I don't use docker by default is that it is an absolute pain when I write code on the project (and it makes debugging a nightmare).

Moving the persistent directories somewhere else is possible, but it will break the existing installations, but it will be in a docker-context much more future-proof. I'll see what I can do there.

And regarding the content tracked in git, it is for some, as submodules, so they should not be fetched by default until you run the importers.

— Reply to this email directly, view it on GitHubhttps://github.com/cve-search/vulnerability-lookup/pull/37#issuecomment-2074317423, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AV7DNCNZNNXP5AYAUPBSA2LY65QY3AVCNFSM6AAAAABGU5ZGECVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZUGMYTONBSGM. You are receiving this because you authored the thread.Message ID: @.***>