Docker Image Analysis - Githubissues

glimow commented 5 years ago

DockerHub stores millions of docker images, allowing anybody to download and run them. Among those images are databases, operating systems, web services, development environnements, applications, build tools and so on. For this reasons, DockerHub is an awesome dataset of millions of applications inside their own running environnement.

Thus, analysing this huge dataset could allow building tools to automatically generate server configuration given a code repository, or infer the cost of the cloud infrastructure needed to run a specific service.

My main goal is to explorate the feasability of such tools. It involves a lot of different subtasks:

[x] Retrieving the docker image dataset https://github.com/src-d/ml-backlog/issues/81 https://github.com/src-d/ml-backlog/issues/82
[x] Achieving an in-depth analysis of what software packages (native, python and nodejs ones) are used in each image https://github.com/src-d/ml-backlog/issues/83
[x] Use that analysis to represent docker images in a dense vector space https://github.com/src-d/ml-backlog/issues/84
[x] Evaluating if this vector space captures successfully the semantics of images by using it to solve various ML tasks gradually incrasing in complexity https://github.com/src-d/ml-backlog/issues/85 https://github.com/src-d/ml-backlog/issues/86
[ ] Try reaching an application that is accurate enough to be used in the company's product

m09 commented 5 years ago

Thanks @glimow for the great issues! Did you request a repo to infra to put your code in it? I see you have a personal repo for now, but it'd be great to have it in src-d's org :) Let me know if you need help on that.

glimow commented 5 years ago

@m09 Yup I hosted this on a personal repo after asking @vmarkovtsev, because he wanted to double check the code before hosting it under sourced's banner.

also UPDATE: we have 50,000 images since python lib update. I suspect most of the limitations come from the amount of data that is limiting the embedding's quality. I will embed again when I have ~100k images.

vmarkovtsev commented 5 years ago

@glimow Normally you run a basic frequency analysis and cut the rare libraries, that should help with embeddings.

glimow commented 5 years ago

@vmarkovtsev actually the prepscript already does that for me (by default skipping <5 occurence libs)

vmarkovtsev commented 5 years ago

Tristan's internship has ended. The artifact is https://github.com/src-d/docker-image-analysis which is going to become public.

src-d / ml-backlog

Docker Image Analysis #80