Issue installing pycairo in Ubuntu on azure-webapp

stuaxo commented 1 year ago

Migrated from #303

https://github.com/pygobject/pycairo/issues/303#issuecomment-1530768353

@sonomirco is trying to use streamlit in Azure webapp, pycairo is failing to install - see comment above.

In the linked comment, the CI config for Azure shows a fairly standard setup: Ubuntu and a venv.

This may be possible to reproduce on plain Ubuntu, using Act, or in Azure - it's possible the venv is complicating things somehow.

sonomirco commented 1 year ago

After some tests this doesn't fails but when the app opens this is the error.

      - name: Install system packages
        run: |
          sudo apt-get update
          sudo apt-get install libpango1.0-0 libcairo2 libgtk-3-dev libpq-dev libffi-dev
          sudo apt install libcairo2-dev pkg-config python3-dev
          ldconfig -p | grep cairo

      - name: Create and start virtual environment
        run: |
          python -m venv venv
          source venv/bin/activate

      - name: Install dependencies
        run: |
          pip3 install pycairo
          pip3 install -r requirements.txt

File "/tmp/8db4af7125a58fc/antenv/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 556, in _run_script exec(code, module.dict)

stuaxo commented 1 year ago

Is the code to your app on some repo somewhere ?

sonomirco commented 1 year ago

Yes in the root I have a folder called src the code is in there.

stuaxo commented 1 year ago

I can see you are in the python environment antenv in /tmp, it looks like pycairo isn't installed there.

ghost commented 1 year ago

I suddenly got the same issue, been running the app with same dependencies for a long time, and on a redeploy today it failed with this error. Possibly something to to with Azure App Service

stuaxo commented 1 year ago

Without seeing logs it's impossible to say. This sort of thing might happen if they updated their python version to one that pycairo isn't built for yet.

feedain9 commented 1 year ago

same here, these are logs that I have : `Building wheels for collected packages: pycairo [10:42:44+0000] Building wheel for pycairo (pyproject.toml): started [10:42:45+0000] Building wheel for pycairo (pyproject.toml): finished with status 'error' error: subprocess-exited-with-error

× Building wheel for pycairo (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [15 lines of output] running bdist_wheel running build running build_py creating build creating build/lib.linux-x86_64-cpython-311 creating build/lib.linux-x86_64-cpython-311/cairo copying cairo/init.py -> build/lib.linux-x86_64-cpython-311/cairo copying cairo/init.pyi -> build/lib.linux-x86_64-cpython-311/cairo copying cairo/py.typed -> build/lib.linux-x86_64-cpython-311/cairo running build_ext Package cairo was not found in the pkg-config search path. Perhaps you should add the directory containing cairo.pc' to the PKG_CONFIG_PATH environment variable No package 'cairo' found Command '['pkg-config', '--print-errors', '--exists', 'cairo >= 1.15.10']' returned non-zero exit status 1. [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for pycairo [10:42:45+0000] Failed to build pycairo ERROR: Could not build wheels for pycairo, which is required to install pyproject.toml-based projects`

And this is my jobs on workflow : `steps:

uses: actions/checkout@v2
name: Set up Python version uses: actions/setup-python@v4 with: python-version: '3.11'
name: Install linux dependancies run: | sudo apt-get update sudo apt-get install gcc libpango1.0-0 libcairo2 libgtk-3-dev libpq-dev libffi-dev sudo apt install libcairo2-dev pkg-config python3-dev pip3 install --upgrade pip ldconfig -p | grep cairo
name: Create and start virtual environment run: | python -m venv venv source venv/bin/activate
name: Install dependencies run: | pip3 install pycairo pip3 install -r requirements.txt --no-cache-dir`

stuaxo commented 1 year ago

Package cairo was not found in the pkg-config search path.

Is the key error...

On my local ubuntu machine, I have this, and can see it was installed by libcairo2-dev

dpkg -S cairo.pc
libpango1.0-dev:amd64: /usr/lib/x86_64-linux-gnu/pkgconfig/pangocairo.pc
libcairo2-dev:amd64: /usr/lib/x86_64-linux-gnu/pkgconfig/cairo.pc

I can see you do have a line to install it in Install linux dependancies.

I'd check your logs for the build process in the Install linux dependancies for errors around apt-get. You might be able to add a bit of code and see if the cairo.pc file is present.

pkg-config --list-all

Will list all the .pc files pkg-config knows about.

feedain9 commented 1 year ago

these are my logs on github actions build process around Install linux dependancies :

Reading package lists...
Building dependency tree...
Reading state information...
libcairo2-dev is already the newest version (1.16.0-5ubuntu2).
libcairo2-dev set to manually installed.
pkg-config is already the newest version (0.29.2-1ubuntu3).
python3-dev is already the newest version (3.10.6-1~22.04).
0 upgraded, 0 newly installed, 0 to remove and 28 not upgraded.
Requirement already satisfied: pip in /opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages (22.3.1)
Collecting pip
  Downloading pip-23.1.2-py3-none-any.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 11.6 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.3.1
    Uninstalling pip-22.3.1:
      Successfully uninstalled pip-22.3.1
Successfully installed pip-23.1.2
    libpangocairo-1.0.so.0 (libc6,x86-64) => /lib/x86_64-linux-gnu/libpangocairo-1.0.so.0
    libpangocairo-1.0.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libpangocairo-1.0.so
    libcairo.so.2 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcairo.so.2
    libcairo.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libcairo.so
    libcairo-script-interpreter.so.2 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcairo-script-interpreter.so.2
    libcairo-script-interpreter.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libcairo-script-interpreter.so
    libcairo-gobject.so.2 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcairo-gobject.so.2
    libcairo-gobject.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libcairo-gobject.so

There isn't problems about installing packages, i have added this job :

- name: Checking cairo installed
        run: |
          pkg-config --list-all

and it prints this :

cairo                            cairo - Multi-platform 2D graphics library
cairo-fc                         cairo-fc - Fontconfig font backend for cairo graphics library
cairo-ft                         cairo-ft - FreeType font backend for cairo graphics library
cairo-gobject                    cairo-gobject - gobject functions for cairo graphics library
cairo-pdf                        cairo-pdf - PDF surface backend for cairo graphics library
cairo-png                        cairo-png - PNG functions for cairo graphics library
cairo-ps                         cairo-ps - PostScript surface backend for cairo graphics library
cairo-script                     cairo-script - script surface backend for cairo graphics library
cairo-svg                        cairo-svg - SVG surface backend for cairo graphics library
cairo-tee                        cairo-tee - tee surface backend for cairo graphics library
cairo-xcb                        cairo-xcb - XCB surface backend for cairo graphics library
cairo-xcb-shm                    cairo-xcb-shm - XCB/SHM functions for cairo graphics library
cairo-xlib                       cairo-xlib - Xlib surface backend for cairo graphics library
cairo-xlib-xrender               cairo-xlib-xrender - Xlib Xrender surface backend for cairo graphics library```
Are these good pkg ?
About "lib*" I have only these pkgs : ```libR                             libR - R as a library
libbrotlicommon                  libbrotlicommon - Brotli common dictionary library
libbrotlidec                     libbrotlidec - Brotli decoder library
libbrotlienc                     libbrotlienc - Brotli encoder library
libcrypt                         libxcrypt - Extended crypt library for DES, MD5, Blowfish and others
libcrypto                        OpenSSL-libcrypto - OpenSSL cryptography library
libdeflate                       libdeflate - Fast implementation of DEFLATE, zlib, and gzip
libdmmp                          libdmmp - Device mapper multipath management library
libexif                          libexif - Library for easy access to EXIF data
libffi                           libffi - Library supporting Foreign Function Interfaces
libgcrypt                        libgcrypt - General purpose cryptographic library
libgdiplus                       libgdiplus - GDI+ implementation
libglvnd                         libglvnd - Vendor-neutral OpenGL dispatch library vendor interface
libjpeg                          libjpeg - A SIMD-accelerated JPEG codec that provides the libjpeg API
liblz4                           lz4 - extremely fast lossless compression algorithm library
liblzma                          liblzma - General purpose data compression library
libmagic                         libmagic - Magic number recognition library
libnsl                           libnsl - Library containing NIS functions using TI-RPC (IPv6 enabled)
libopenjp2                       openjp2 - JPEG2000 library (Part 1 and 2)
libopenjp3d                      openjp3d - JPEG2000 Extensions for three-dimensional data (Part 10)
libopenjpip                      openjpip - JPEG2000 Interactivity tools, APIs and protocols (Part 9)
libpcre                          libpcre - PCRE - Perl compatible regular expressions C library with 8 bit character support
libpcre16                        libpcre16 - PCRE - Perl compatible regular expressions C library with 16 bit character support
libpcre2-16                      libpcre2-16 - PCRE2 - Perl compatible regular expressions C library (2nd API) with 16 bit character support
libpcre2-32                      libpcre2-32 - PCRE2 - Perl compatible regular expressions C library (2nd API) with 32 bit character support
libpcre2-8                       libpcre2-8 - PCRE2 - Perl compatible regular expressions C library (2nd API) with 8 bit character support
libpcre2-posix                   libpcre2-posix - Posix compatible interface to libpcre2-8
libpcre32                        libpcre32 - PCRE - Perl compatible regular expressions C library with 32 bit character support
libpcrecpp                       libpcrecpp - PCRECPP - C++ wrapper for PCRE
libpcreposix                     libpcreposix - PCREPosix - Posix compatible interface to libpcre
libpng                           libpng - Loads and saves PNG files
libpng16                         libpng - Loads and saves PNG files
libpq                            libpq - PostgreSQL libpq library
librsvg-2.0                      librsvg - library that renders svg files
libsecret-1                      libsecret-1 - GObject bindings for Secret Service API
libsecret-unstable               libsecret-unstable - GObject bindings for Secret Service API (Unstable)
libselinux                       libselinux - SELinux utility library
libsepol                         libsepol - SELinux policy library
libssl                           OpenSSL-libssl - Secure Sockets Layer and cryptography libraries
libthai                          libthai - Thai support library
libtiff-4                        libtiff - Tag Image File Format (TIFF) library.
libtirpc                         libtirpc - Transport Independent RPC Library
libwmf                           libwmf - A library for reading and converting Windows MetaFile vector graphics (WMF)
libxcrypt                        libxcrypt - Extended crypt library for DES, MD5, Blowfish and others
libxml-2.0                       libXML - libXML library version2.
libzstd                          zstd - fast lossless compression algorithm library

ghost commented 1 year ago

It seems like the default container Azure App Service uses has stopped including certain dependencies that are needed to build the pycairo wheel. I have found a workaround.

Accoring to the pycairo documentation you need to install the following dependencies:

sudo apt-get install libcairo2-dev pkg-config python3-dev

Option 1: Custom startup script

We can setup a custom startup script for an Azure Web App, you can find information on how to do that here Basicly we need to set this in your "Application Settings":

SMD_DO_BUILD_DURING_DEPLOY=0

The instead said a custom startup script that installs the dependencies, migrates your database, collect static files and in the end starts your wsgi application:

apt-get update &&
apt-get install -y build-essential libcairo2-dev pkg-config python3-dev &&
pip install -r requirements.txt &&
python manage.py migrate &&
python manage.py collectstatic &&
gunicorn --bind=0.0.0.0 --timeout 1800 yourporject.wsgi

It is also smart to extend the default container warmup timeout, as this installation process might be longer than the default 140 seconds. Add this to your apps configuriation "Application Settings"(1800 is the max timeout ):

WEBSITES_CONTAINER_START_TIME_LIMIT=1800

Option 2: Custom container

One other solution would be to configure a custom container and make sure the dependencies are installed. They seem to be in the default python:3.10 image that I am using for development.

feedain9 commented 1 year ago

Thanks a lot @Pettermoe I am trying it.

Is "SCM_DO_BUILD_DURING_DEPLOYMENT" different of "SMD_DO_BUILD_DURING_DEPLOY" ?

Because on my app settings I have a "SCM_DO_BUILD_DURING_DEPLOYMENT" already set.

pettermoe95 commented 1 year ago

@feedain9

It was a typo from me, SCM_DO_BUILD_DURING_DEPLOYMENT is the correct name setting :)

pettermoe95 commented 1 year ago

It seems like the default container Azure App Service uses has stopped including certain dependencies that are needed to build the pycairo wheel. I have found a workaround.

Accoring to the pycairo documentation you need to install the following dependencies:
sudo apt-get install libcairo2-dev pkg-config python3-dev
Option 1: Custom startup script

We can setup a custom startup script for an Azure Web App, you can find information on how to do that here Basicly we need to set this in your "Application Settings":
SMD_DO_BUILD_DURING_DEPLOY=0 
The instead said a custom startup script that installs the dependencies, migrates your database, collect static files and in the end starts your wsgi application:
apt-get update &&
apt-get install -y build-essential libcairo2-dev pkg-config python3-dev &&
pip install -r requirements.txt &&
python manage.py migrate &&
python manage.py collectstatic &&
gunicorn --bind=0.0.0.0 --timeout 1800 yourporject.wsgi
It is also smart to extend the default container warmup timeout, as this installation process might be longer than the default 140 seconds. Add this to your apps configuriation "Application Settings"(1800 is the max timeout ):
WEBSITES_CONTAINER_START_TIME_LIMIT=1800
Option 2: Custom container

One other solution would be to configure a custom container and make sure the dependencies are installed. They seem to be in the default python:3.10 image that I am using for development.

It should be: SCM_DO_BUILD_DURING_DEPLOYMENT and not SMD_DO_BUILD_DURING_DEPLOY

feedain9 commented 1 year ago

@feedain9

It was a typo from me, SCM_DO_BUILD_DURING_DEPLOYMENT is the correct name setting :)

Thanks

Lastly, in jobs.deploy.steps I add after deploying the gunicorn command like this : - name: Launch gunicorn run: source venv/bin/activate gunicorn --bind=0.0.0.0 --timeout 1800 exoneroi.wsgi

But I have this : "venv/bin/activate: No such file or directory" with jobs.build.steps : - name: Create and start virtual environment run: | python -m venv venv source venv/bin/activate pip install --upgrade pip

How to handle this ? I can join my workflow file if it's necessary.

pettermoe95 commented 1 year ago

I'm not sure you need to activate a virtual environment, I didn't do that atleast. If so you probably need to create a virtual environment first, then activate it before you even run the pip install command.

I would try to skip the source venv/bin/activate command and see if you manage to start your app then. After this you might explore possibility of installing the python dependencies in a virtual environment, solving one problem one step at a time.

feedain9 commented 1 year ago

If i do not activate the venvI ran into "Gunicorn command was not found" even if I install it So for you there is no need to use virtual env to deploy apps ?

Except linux dependancies installs and gunicorn run, the rest is generated by azure's workflow... This is the file.

# Docs for the Azure Web Apps Deploy action: https://github.com/Azure/webapps-deploy
# More GitHub Actions for Azure: https://github.com/Azure/actions
# More info on Python, GitHub Actions, and Azure App Service: https://aka.ms/python-webapps-actions

name: Build and deploy Python app to Azure Web App

on:
  push:
    branches:
      - main
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2

      - name: Set up Python version
        uses: actions/setup-python@v1
        with:
          python-version: '3.11'

      - name: Create and start virtual environment
         run: |
           python -m venv venv
           source venv/bin/activate

      - name: Install linux dependancies
        run: |
          sudo apt-get update 
          sudo apt-get install -y build-essential libcairo2-dev pkg-config python3-dev gunicorn

      - name: Install Python dependancies
        run: |
          pip install -r requirements.txt
          pip install --upgrade virtualenv

      # Optional: Add step to run tests here (PyTest, Django test suites, etc.)

      - name: Upload artifact for deployment jobs
        uses: actions/upload-artifact@v2
        with:
          name: python-app
          path: |
            . 
            !venv/

  deploy:
    runs-on: ubuntu-latest
    needs: build
    environment:
      name: 'Production'
      url: ${{ steps.deploy-to-webapp.outputs.webapp-url }}

    steps:
      - name: Download artifact from build job
        uses: actions/download-artifact@v2
        with:
          name: python-app
          path: .

      - name: 'Deploy to Azure Web App'
        uses: azure/webapps-deploy@v2
        id: deploy-to-webapp
        with:
          app-name: 'ExoneRoi'
          slot-name: 'Production'
          publish-profile: ${{ secrets.AZUREAPPSERVICE_PUBLISHPROFILE_8BFB8CC5667242358FF4EC6CBF0D7DEE }}

      - name: Launch gunicorn 
        run: |
          source venv/bin/activate
          gunicorn --bind=0.0.0.0 --timeout 1800 exoneroi.wsgi

pettermoe95 commented 1 year ago

I think I understand now. So you are building your python dependencies, saving it as an artifact and then you deploy it to the web app.

I ended up installing the linux dependencies on the actual azure app service, as part of the startup command, then I installed the python dependencies, inside the container that runs in the azure app service.

So what I do in short steps is: Zip my django project -> Deploy to web app -> Startup command runs inside azure web app container -> Installing linux dependencies -> Install python packages -> Then launch gunicorn.

Since you are installing your python packages as part of your build step, you might not need to do anything else than simply installing the dependencies, like you already did here:

- name: Install linux dependancies
        run: |
          sudo apt-get update 
          sudo apt-get install -y build-essential libcairo2-dev pkg-config python3-dev

I think gunicorn should be installed in the app service already.

Then you could keep the deploy step as you have always done, and hopefully the web app should run as usual. The only problem I can see with this is if the pycairo needs those linux dependencies to run or if they are just required to buiild the wheel, if so you would have to install those dependencies in the actual azure app service container as well. You can do that as part of the custom startup script, that is what I ended up doing and that has worked for me.

feedain9 commented 1 year ago

So could you think that yours is something like this :


name: Deploy Django app to Azure Web App

on:
  push:
    branches:
      - main

env:
  AZURE_WEBAPP_NAME: my-app-name       
  AZURE_WEBAPP_PACKAGE_PATH: .\app.zip  
  AZURE_WEBAPP_STARTUP_COMMAND: gunicorn appwsgi --bind=0.0.0.0:8000  Gunicorn

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout code
      uses: actions/checkout@v2

    - name: Set up Python 3.8
      uses: actions/setup-python@v2
      with:
        python-version: '3.11'

    - name: Install dependencies
      run: |
        sudo apt-get update
        sudo apt-get install -y python3-dev python3-pip build-essential
        pip install --upgrade pip
        pip install -r requirements.txt

    - name: Zip application code
      run: zip -r app.zip .

    - name: Deploy to Azure Web App
      uses: azure/webapps-deploy@v2
      with:
        app-name: ${{ env.AZURE_WEBAPP_NAME }}
        package: ${{ env.AZURE_WEBAPP_PACKAGE_PATH }}
        startup-command: ${{ env.AZURE_WEBAPP_STARTUP_COMMAND }}
```  ?

pettermoe95 commented 1 year ago

Yes. And if that won't work, it can be because pycairo still needs those dependencies to run on the webserver: For my the AZURE_WEBAPP_STARTUP_COMMAND would be this then:

apt-get install -y build-essential libcairo2-dev pkg-config python3-dev &&
python manage.py migrate &&
python manage.py collectstatic &&
gunicorn --bind=0.0.0.0 --timeout 1800 myproject.wsgi

The reason why I add this:

...
python manage.py migrate &&
python manage.py collectstatic &&
...

Is because when SCM_DO_BUILD_DURING_DEPLOYMENT is set to 1(the default), it will migrate the database and collect the static files. It will also locate the requirements.txt file and install the dependencies. Since you are installing the dependencies before deploying, the only thing you need to do is making sure to run database migrations, run collectstatic, then start gunicorn.

feedain9 commented 1 year ago

"AZURE_WEBAPP_STARTUP_COMMAND" need to be set on app settings on azure board ? And with these commands :


apt-get install -y build-essential libcairo2-dev pkg-config python3-dev &&
python manage.py migrate &&
python manage.py collectstatic &&
gunicorn --bind=0.0.0.0 --timeout 1800 myproject.wsgi
```?

sonomirco commented 1 year ago

"AZURE_WEBAPP_STARTUP_COMMAND" need to be set on app settings on azure board ? And with these commands :
apt-get install -y build-essential libcairo2-dev pkg-config python3-dev &&
python manage.py migrate &&
python manage.py collectstatic &&
gunicorn --bind=0.0.0.0 --timeout 1800 myproject.wsgi
```?

Where are you adding this command? In the GitHub action? Do you have in the requirements.txt pycairo? I'll try and let you know how it goes.

pettermoe95 commented 1 year ago

I have one deployment task that takes a .zip artifact and use the built in azure web app deployment to deploy to Azure. Int here I have set it as StartupCommand.

I have a few additional commands in my script that you won't need tho. like rm -r var/static_root and others.

steps:
- task: AzureRmWebAppDeployment@4
  displayName: 'Azure App Service Deploy: my app'
  inputs:
    azureSubscription: 'My subscription'
    appType: webAppLinux
    WebAppName: 'app-myproject'
    packageForLinux: '$(System.DefaultWorkingDirectory)/_Myproject-CI/drop/*.zip'
    RuntimeStack: 'PYTHON|3.10'
    StartupCommand: 'apt-get update && apt-get install -y build-essential libcairo2-dev pkg-config python3-dev wkhtmltopdf && pip install -r requirements.txt && python manage.py migrate && rm -r var/static_root && python manage.py collectstatic && gunicorn --bind=0.0.0.0 --timeout 1800 myproject.wsgi'

feedain9 commented 1 year ago

"AZURE_WEBAPP_STARTUP_COMMAND" need to be set on app settings on azure board ? And with these commands :
apt-get install -y build-essential libcairo2-dev pkg-config python3-dev &&
python manage.py migrate &&
python manage.py collectstatic &&
gunicorn --bind=0.0.0.0 --timeout 1800 myproject.wsgi
```?
Where are you adding this command? In the GitHub action? Do you have in the requirements.txt pycairo? I'll try and let you know how it goes.

On azure webapp board.

feedain9 commented 1 year ago

I have one deployment task that takes a .zip artifact and use the built in azure web app deployment to deploy to Azure. Int here I have set it as StartupCommand.

I have a few additional commands in my script that you won't need tho. like rm -r var/static_root and others.
steps:
- task: AzureRmWebAppDeployment@4
  displayName: 'Azure App Service Deploy: my app'
  inputs:
    azureSubscription: 'My subscription'
    appType: webAppLinux
    WebAppName: 'app-myproject'
    packageForLinux: '$(System.DefaultWorkingDirectory)/_Myproject-CI/drop/*.zip'
    RuntimeStack: 'PYTHON|3.10'
    StartupCommand: 'apt-get update && apt-get install -y build-essential libcairo2-dev pkg-config python3-dev wkhtmltopdf && pip install -r requirements.txt && python manage.py migrate && rm -r var/static_root && python manage.py collectstatic && gunicorn --bind=0.0.0.0 --timeout 1800 myproject.wsgi'

So your job is on build part or deploy part of workflow ?

PBRWJELF commented 1 year ago

Same error occurs on when running on a ubuntu machine locally. It has been working fine for months and now breaks every time with the same error message about missing dependencies as described in this issue. Its not AWS specific.

I'm going to explore the solutions discussed above, but just thought I'd point out its not AWS specific.

MatthijsBurgh commented 1 year ago

I think the issue is the pip package doesn't install without the apt package libcairo2-dev being installed. The question is whether the dependencies are not set correctly. aka a dependency is missing. Or the dependency, which should provide the .pc file, doesn't do this. So that dependency is broken. (Alternative version on the latter one, the dependency, that should provide the .pc file, does provide it, but not in a location that is searched.)

P.S. This issue is unrelated to Azure, GH actions or any other web service. I have tested in on my local machine. Therefore the title should be changed.

stuaxo commented 1 year ago

If it happens in plain Ubuntu I can easily test it when I get a moment - I don't have Azure webapp access + it would take some time to learn it.

It's probably going to be tomorrow before I can test it on a local Ubuntu.

annietllnd commented 1 year ago

I experienced the issue on local Ubuntu and resolved it by adding the apt packages as above, FYI

sudo apt-get install libcairo2-dev pkg-config python3-dev

stuaxo commented 1 year ago

I run ubuntu locally and pip install pycairo works here.

To test on a clean environment I built an ubuntu docker image using the packages @annietllnd listed (the same ones listed on the getting started page https://pycairo.readthedocs.io/en/latest/getting_started.html) and cannot reproduce the issue:

Dockerfile

FROM ubuntu:latest

RUN apt-get update && \
    apt-get install -y lsb-release python3-pip \
    libcairo2-dev pkg-config python3-dev

RUN pip3 install --upgrade pip

RUN python3 -mpip install pycairo
RUN python3 -mpip freeze > requirements.txt

CMD python3 -c 'import platform, subprocess; \
               print("Python version:", platform.python_version()); \
               print("Ubuntu version:", str.strip(open("/etc/lsb-release").readlines()[3]).split("=")[1]); \
               print("Cairo version:", subprocess.check_output(["pkg-config", "--modversion", "cairo"]).decode().strip()); \
               print("Pycairo version:", subprocess.check_output(["pip", "show", "pycairo"]).decode().split("\n")[1].split(": ")[1].strip())'

build: $ docker build -t pycairo-install-test .

run:

$ docker run pycairo-install-test       
Python version: 3.10.6
Ubuntu version: "Ubuntu 22.04.1 LTS"
Cairo version: 1.16.0
Pycairo version: 1.23.0

MatthijsBurgh commented 1 year ago

@stuaxo I think pycairo should also install without these packages installed.

I only got pycairo as a new dependency of reportlab>=4. Which I only got as a dependency of xhtml2pdf. So pip packages shouldn't depend on packages from other package managers. Since you could get a package as the N-th dependency of a package you want.

stuaxo commented 1 year ago

I agree with the sentiment for sure, pycairo comes from a time before people thought about packaging in python like that so it's not straightforward (certainly before pip, virtualenv or whl files):

@naveen521kk @lazka I can't remember where the conversation around pycairo whl files for linux got to ? Did it look like it was possible, difficult, or maybe not possible ?

naveen521kk commented 1 year ago

@naveen521kk @lazka I can't remember where the conversation around pycairo whl files for linux got to ? Did it look like it was possible, difficult, or maybe not possible ?

AFAIK, there are some reasons why there are no wheels for other platforms than Windows (Linux/macOS):

To share objects between pycairo and the C interface of cairo (using pycairo's C API) both must be using the same version of cairo. This is difficult when using wheel
it is not possible to install .pc files, correctly, when using the wheel

One of the main users are pycairo is pygobject project, and they would require the those to satisfy. On Windows, though, people usually use the MSYS2's python which has a different ABI than the wheels, so no issues there.

stuaxo commented 1 year ago

Thanks for summarising it better than I could have. While virtualenvs give us different python environments, as soon as we touch the system the isolation pretty much ends.

The distinction between python packaging and system packaging rears it's head in this situation, and maybe there will be a future where this is resolved.

What would be nice, is if there was some cooperation between the two; we would probably need something that isolates the system more than the virtualenv does as well.

As mentioned with the .pc issue: for libraries that rely on reading bindings via .pc files, or other C headers, such as pycairo or cffi based bindings like cairocffi - and this breaks another assumption of system packages; between -devel and "normal" packages.

I think the assumption was that runtime installers wouldn't need these resources, but this isn't the case here.

Of course; before virtualenv was a thing, and before there were WHLs it just wasn't that big a deal - people installed everything in the system, or would even manually change their PYTHON_PATH per project.

While we have much better practices now there are limitations, and those are a few.

Having said all that; I really hope this all gets resolved at some point; for my downstream project, github.com/shoebot/shoebot a huge amount of time has been spent trying to work around issues people have getting things installed, particularly around Gtk, Pango, and to a lesser extent Cairo.

Hopefully there will be more packaging workshops at various linux confs to make progress on everything https://hackmd.io/@python-maint/2021-flock-workshop-intro

stuaxo commented 1 year ago

I think it's possible to close this for now, solving this is probably outside of the scope of pycairo - and an issue with python packaging itself.

pygobject / pycairo