dagster-io / dagster-cloud-action

Apache License 2.0
12 stars 18 forks source link

Gitlab/ECR Hybrid action fails when pip installing awscli directly into the root environment #178

Open mgrinshpon opened 4 months ago

mgrinshpon commented 4 months ago

https://github.com/dagster-io/dagster-cloud-action/blob/71ea967e0ed77191bb334e463b748792e4fac786/gitlab/hybrid-ci.yml#L56-L66

These lines right here are problematic because when running pip install awscli, it installs right into root causing the following failure:

$ pip install awscli
error: externally-managed-environment
× This environment is externally managed
╰─> 
    The system-wide python installation should be maintained using the system
    package manager (apk) only.

    If the package in question is not packaged already (and hence installable via
    "apk add py3-somepackage"), please consider installing it inside a virtual
    environment, e.g.:

    python3 -m venv /path/to/venv
    . /path/to/venv/bin/activate
    pip install mypackage

    To exit the virtual environment, run:

    deactivate

    The virtual environment is not deleted, and can be re-entered by re-sourcing
    the activate file.

    To automatically manage virtual environments, consider using pipx (from the
    pipx package).

Hacking through it by allowing system packages to break yields an error when building awscrt:

$ pip install awscli --break-system-packages
....
        File "<string>", line 263, in run
        File "<string>", line 255, in _build_dependencies
        File "<string>", line 163, in _build_dependencies_impl
        File "<string>", line 121, in get_cmake_path
      Exception: CMake must be installed to build from source.
      [end of output]

An alternative could be to just use apk add aws-cli rather than installing through pypi. That lets you skip the python install entirely. Lines 62-63 then just become the following:

- apk add --no-cache curl jq aws-cli