search

opensearch-project / opensearch-ci

Enables continuous integration across OpenSearch, OpenSearch Dashboards, and plugins.
Apache License 2.0
15 stars 25 forks source link

Install pigz and unpigz to speed up docker image extraction #471

Closed peterzhuamazon closed 1 month ago

peterzhuamazon commented 1 month ago

Description

Install pigz and unpigz to speed up docker image extraction

Issues Resolved

Closes https://github.com/opensearch-project/opensearch-ci/issues/374

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

peterzhuamazon commented 1 month ago

Have double verify that pigz will also install unpigz.

prudhvigodithi commented 1 month ago

Thanks, just installing pigz would take care of parallel docker extraction? I'm curious why docker dont directly install it as pre-req. :)

peterzhuamazon commented 1 month ago

Do we need this package on windows as well? To full the images faster?

It is already installed on Windows and reduced time from 15-20min to 12min 😄 https://github.com/opensearch-project/opensearch-ci/blob/main/packer/scripts/windows/scoop-install-commons-docker-support.ps1#L76-L87

peterzhuamazon commented 1 month ago

Thanks, just installing pigz would take care of parallel docker extraction? I'm curious why docker dont directly install it as pre-req. :)

Yes, there is a long history of thread to get pigz/unpigz support in docker. As of now, docker build is still single thread and docker pull/extraction can support unpigz if detected in path. The thing is docker maintainers seems still thinking parallelism can cause trouble between, thus not suggesting it by default. Thanks.