Open dOrgJelli opened 3 years ago
Hey @namesty, thank you for implementing this.
After further consideration, I think the approach I've suggested of "putting the dockerfile inside the build folder" is naive and will lead to problems in the future. Here are some additional details:
Size of Docker Image:
Given a simple test polywrapper (test-cases/cases/apis/asyncify/), the docker image.tar
file is ~450mb. If you pipe the image through gzip, it becomes ~150mb.
Diving deeper into what's causing the bloat inside the image.tar
file, it's ~80% due to the base linux image that's added in the first couple of layers. If you open up the .tar
image file, you can see this for yourself.
IPFS Storage Optimization:
In IPFS, folders & files are stored as DAGs, where each node in the DAG is a hash of a file/folder. This is to help optimize storage and reduce redundancy.
My fear is that, if we store the large image.tar
as a single file, IPFS won't properly optimize its storge. Ideally IPFS would be able to store the individual layers, which would help reduce redundancy, because common layers (such as a base linux image) would be replicated and not duplicated. More research is needed to better understand IPFS's default functionality, and how well the image.tar
file "plays nice" with this, or if it makes it harder for this de-duplication to occur.
Additional Research Needed:
We need to research the following two properties of IPFS:
image.tar
make it harder for IPFS to store these things nicely? I suspect the answer is "no, it hurts IPFS's default storage optimization". This is because, even if there are common layers, they may exist in different "slices" of the image.tar
file, meaning the DAG may look different from one image to another. This can be fixed if we store the image.tar
as a decompressed directory of sub-folders containing only the image's tar file. This is what the image.tar
has inside of it already, so no additional work is needed here. This way IPFS nodes can easily reference common layers more easily. image.tar
, which is not ideal. Ideal Future State:
IMO, in the future, the ideal state would be:
image.tar
file as a single file, we would instead extract the contents and store those in a folder. This will help IPFS optimize the storage of common docker image layers. Additionally in the future, this will help us flag a build image's layers as "audited" or "known good" to help speed up audits. For example, a "known good" layer has of 0x58... which represents a known good linux image would be marked green, and auditors could largely ignore it and move onto other "unknown" layers that are specific to the API at hand.NOTE: The "package" would link to the "source image" within its web3api.build.yaml
file.
All of this said, I think we should discuss what the ideal future state should be, and create an execution plan around that. That would probably lead to us closing this issue and opening up new ones.
Please let me know your thoughts.
Checkout: https://github.com/jvassev/image2ipfs/ It may help
I'll retake this
After a bit of research, I had an idea:
I thought that this would be compliant with the second point of @dOrgJelli's ideal scenario, but I may be missing a point regarding how the audits will be done.
Does this make sense?
Yes that makes perfect sense, I came to the same conclusion after unzipping the docker .tar file and seeing they have all the layers right there!
We could have the OS layer still there, and ideally with IPFS hashing it would de-duplicate it. Would require more investigation to make sure this is the case.
I worry that if we don't have the OS layer, we would have a hard time reproducing the build again.
@namesty after speaking with a developer who's spent the past 3 years researching & building with IPFS & IPLD, he confirmed to me that "if multiple folders have the same files, IPFS will automatically deduplicate these files due to the preservation of the content hash in the IPLD database."
This means that we can store the OS layer no problem because it will not be duplicated. Additionally uploads SHOULD be optimized as well, as I do not think a node would request a request sender to upload a file it already knows it has.
Keep the built docker image inside of a "build" folder (or some other folder). This way we're "hosting" docker images ourselves instead of relying on the docker hub.