Python Package with Dependencies

[Hujun]

As Python developers, we usually build Python package to share our work (devpi for internal & private scope and PyPI for global opensource work). Normally basic usage of setup.py is enough for these works. But there is still some specific scenarios which need more sophisticated packaging process. For example, in specific CI/CD workflow, you don't want take time to download package dependencies from PyPI (or devpi) every time when build the docker image. So how to do this?

Create wheel house

The first thing to do is to make all dependencies local packages (wheel). Pip has this function as:

pip wheel -r requirements.txt --wheel-dir <path>

The command will download all dependencies defined in requirements.txt from PyPI as usual, and make them as particular wheel package in the directory specified.

To simplify the operation, you can put it in setup.py command. The following snippet demonstrates the a package procedure with dependencies wheel making.

class PackageCommand(Command):
    """Support setup.py package"""

    description = 'Build package.'
    user_options = [
        ('with-deps', None, 'package with all wheel packages of dependencies'),
    ]

    def initialize_options(self):
        """Set default values for options"""
        self.with_deps=False

    def finalize_options(self):
        """Post-process options"""
        if self.with_deps:
            self.with_deps=True

    def run(self):
        """Run command"""
        clear_files = [
            os.path.join(pwd_path, 'build'),
            os.path.join(pwd_path, 'dist'),
            os.path.join(pwd_path, '*/*.egg-info'),
            os.path.join(pwd_path, 'Youtiao.egg-info'),
        ]
        for cf in clear_files:
            print('rm {}'.format(cf))
            subprocess.run(['rm', '-rf', cf])

        # make sure that wheel is installed
        subprocess.run(['python', 'setup.py', 'bdist', 'bdist_wheel', '--universal'])

        if self.with_deps:
            rqm_path = os.path.join(pwd_path, 'requirements.txt')
            wheels_path = os.path.join(pwd_path, 'wheels')
            subprocess.run(['rm', '-rf', wheels_path])
            subprocess.run(['mkdir', '-p', wheels_path])
            subprocess.run('pip wheel --wheel-dir={} -r {}'.format(wheels_path, rqm_path), shell=True)

        sys.exit(0)

Put it in your setup cmdclass of arguments.

Make docker image without dependency download

Normally we run simple pip install -r requirements.txt in docker build script. It needs some modification if you want to avoid download from PyPI. A simple Dockerfile example as:

FROM python:3.6

COPY . /app
WORKDIR /app

# install wheel 
# make sure that your project wheel under /app/dist
# and wheels of dependencies under /app/wheels
RUN pip install /app/dist/<project_wheel> --no-index --find-links /app/wheels

ENTRYPOINT ["python"]
CMD ["start_point_script.py"]

Voila