Open tschaffter opened 4 years ago
I think we had synk set up at some point. I had brought this up to Aaron at the beginning of this year when we went over vulnerabilities of using docker
@tschaffter , I would leave synk out of the validation of docker images. The reason for this is that they "force" you to login through the console when using their cli (and I think the login token expires over a period of time). Therefore it makes is incredibly difficult to automate this process.
My original thought was for the possibility of all docker images in the docker registry to be scanned. We certainly should implement some service in the new challenge platform that will auto scan all the docker images upon docker push
.
My original thought was for the possibility of all docker images in the docker registry to be scanned. We certainly should implement some service in the new challenge platform that will auto scan all the docker images upon docker push.
That's definitively a feature that would be great to have in general.
An important note to consider is that the output of the scan is dependent on WHEN the scan occurs. An image that is a few weeks old will likely have CVEs. For the sake of benchmarking, the scan should occurs just before we process the image. We may also need to derogate to this scan if we want to evaluate a past submission on a new dataset that has just been connected.
For the reasons that we have both exposed, let's keep this ticket as Low: Priority and come back to it when it's needed, for example if a Data Site want to be able to add this scan to the validation process.
@tschaffter Now docker scan
is a built in command that we can potentially use to scan docker images.
docker pull docker.synapse.org/syn22277123/date-annotator-example:1.2.0
docker scan docker.synapse.org/syn22277123/date-annotator-example:1.2.0
.......
Package manager: deb
Project name: docker-image|docker.synapse.org/syn22277123/date-annotator-example
Docker image: docker.synapse.org/syn22277123/date-annotator-example:1.2.0
Platform: linux/amd64
Tested 138 dependencies for known vulnerabilities, found 133 vulnerabilities.
For more free scans that keep your images secure, sign up to Snyk at https://dockr.ly/3ePqVcp
@thomasyu888 How would you test this feature before rolling it out to production? Can we implement it for submissions that are sent to test submissions queues?
Since the validity of a submission will depends on time - i.e. a submission that was submitted 1 month ago may no longer pass the scan test - I'm afraid that it would be difficult to rerun old submission. Instead, we should consider a model where we email the results of the scan to submitters but don't block submission from being evaluated based on the output of the scan.
@tschaffter Also I just wanted you to see that our date-annotator-example has 133 vulnerabilities and around 10 of high severity.
We could email or upload results of the scan to their log folder, although this is something that they could do on their own now. Since it doesn't seem like we are going to take action on these high severity issues, I wonder if we should even implement this step.
@ahayden @thomasyu888 To further tighten the security of our evaluation environment, I'm thinking to scan Docker images for known Common Vulnerabilities and Exposures (CVE) using a service like https://snyk.io/. I experimented with Snyk a few weeks ago and I was able to scan Docker images using one command line.
A strength of Snyk is its integration to CI/CD, however I don't see how we could benefit from this. Instead, we could use the snyk client to scan images.
Go to https://app.snyk.io/account and get a token, then use it as shown below
Normally your browser should open to ask you to validate this token so it can be used on your machine. Because I'm running the above command in a Docker container for security reason, the browser page never opening. From Snyk documentation I figured out that I should go to the address https://app.snyk.io/login/cli?token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Back to the terminal:
We can use the exit code of
snyk test
to determine if the test "didn't pass" (i.e. at least one vulnerability?)