painful task of building and by consequence maintaining certain python app

[lpyparmentier]

@hfiguiere, Following my PR on flathub I will try to explain why building a python app can be painful with the tool.

According to discussion we can't use the option build-options/build-args so we have to pre-download everything as a generated module thanks to the builder-tools here.

Let's start with streamdeck-ui, I want to build the flatpak app.

1. get the poetry.lock
2. run flatpak-poetry-generator.py poetry.lock, first fail.

...
Extracting download url and hash for pyrsistent, version 0.18.0
Extracting download url and hash for pyside2, version 5.15.2
Traceback (most recent call last):
  File "/usr/local/bin/test.py", line 171, in <module>
    main()
  File "/usr/local/bin/test.py", line 160, in main
    sources = get_module_sources(parsed_lockfile, ignore_pkgs=ignore_pkgs, include_devel=include_devel)
  File "/usr/local/bin/test.py", line 91, in get_module_sources
    url, hash = get_pypi_source(
  File "/usr/local/bin/test.py", line 48, in get_pypi_source
    raise Exception("Failed to extract url and hash from {}".format(url))
Exception: Failed to extract url and hash from https://pypi.org/pypi/pyside2/json

I check the related issues, found this, one team lead at Qt explain why there is no source available discussion. To my POV two ways to solve this.: 1.1. Either I ask for each developer to add the sources 1.2. Ignore and install later as a separate module

I'll go with 1.2, no time to deal with each developer that don't put source on pypi, it looks like they have their reasons and pypi don't force them. So I change the script in order to have an option --ignore-pkgs like #217

3. re-run flatpak-poetry-generator.py poetry.lock --ignore-pkgs pyside2 and keep in mind that I will have to build pyside2 or include somewhere else as a module. second fail:

   Extracting download url and hash for pywin32, version 301
   Traceback (most recent call last):
   File "/usr/local/bin/test.py", line 172, in <module>
   main()
   File "/usr/local/bin/test.py", line 161, in main
   sources = get_module_sources(parsed_lockfile, ignore_pkgs=ignore_pkgs, include_devel=include_devel)
   File "/usr/local/bin/test.py", line 91, in get_module_sources
   url, hash = get_pypi_source(
   File "/usr/local/bin/test.py", line 48, in get_pypi_source
   raise Exception("Failed to extract url and hash from {}".format(url))
   Exception: Failed to extract url and hash from https://pypi.org/pypi/pywin32/json

4. pywin32 is for windows from what I understand, let's ignore it also (or think that i'll need to add it as a module). re-run flatpak-poetry-generator.py poetry.lock --ignore-pkgs pyside2 pywin32. third fail

   Extracting download url and hash for shiboken2, version 5.15.2
   Traceback (most recent call last):
   File "/usr/local/bin/test.py", line 172, in <module>
   main()
   File "/usr/local/bin/test.py", line 161, in main
   sources = get_module_sources(parsed_lockfile, ignore_pkgs=ignore_pkgs, include_devel=include_devel)
   File "/usr/local/bin/test.py", line 91, in get_module_sources
   url, hash = get_pypi_source(
   File "/usr/local/bin/test.py", line 48, in get_pypi_source
   raise Exception("Failed to extract url and hash from {}".format(url))
   Exception: Failed to extract url and hash from https://pypi.org/pypi/shiboken2/json

5. lets remove shiboken2, it is part of pyside2 from what i've read. flatpak-poetry-generator.py poetry.lock --ignore-pkgs pyside2 pywin32 shiboken2. Now I have a generated-poetry-sources.json. Great let's add it here and run a classic build flatpak-builder --user --install --force-clean build-dir .....yml fourth fail

   Processing ./appdirs-1.4.4-py2.py3-none-any.whl
   Processing ./appnope-0.1.2-py2.py3-none-any.whl
   Processing ./argon2-cffi-21.1.0.tar.gz
   Installing build dependencies ... error
   ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /tmp/pip-standalone-pip-plt02y52/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-g9c4fjr_/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links file:///run/build/poetry-deps -- 'setuptools>=40.6.0' wheel 'cffi>=1.0'
      cwd: None
   Complete output (3 lines):
   Looking in links: file:///run/build/poetry-deps
   ERROR: Could not find a version that satisfies the requirement setuptools>=40.6.0 (from versions: none)
   ERROR: No matching distribution found for setuptools>=40.6.0
   ----------------------------------------
   WARNING: Discarding file:///run/build/poetry-deps/argon2-cffi-21.1.0.tar.gz. Command errored out with exit status 1: /usr/bin/python3 /tmp/pip-standalone-pip-plt02y52/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-g9c4fjr_/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links file:///run/build/poetry-deps -- 'setuptools>=40.6.0' wheel 'cffi>=1.0' Check the logs for full command output.
   ERROR: Could not find a version that satisfies the requirement argon2-cffi (from versions: 21.1.0)
   ERROR: No matching distribution found for argon2-cffi

6. ok let's add setuptools, now I have to modify the generated json, very ugly, if the maintainer of the original app change dependencies, it will change the generated poetry.json and I will have to re-apply everything I'm doing from here (same issue with pip-gen btw). So let's do it anyway, and keep finger crossed that no plenty similar error happend, or worst: the stack of new dependencies dont match in the constraint of versions.

Append this as source in the poetry-generated file:

...
      {
          "type": "file",
          "url": "https://files.pythonhosted.org/packages/d0/4a/22ee76842d8ffc123d4fc48d24a623c1d206b99968fe3960039f1efc2cbc/setuptools-49.1.3.zip",
          "sha256": "ea484041dc6495c4ee74dd578470866e3043f6d1c998cfafbdb1f668f1768284"
      },
...

So I spend time to find the source, the right url/version. Then re-run the build.

7. fail no counting anymore, what I didn't want to happened, happened. Now it's wheel that miss. I have to stack other missing sources on an already generated file:

   Processing ./appnope-0.1.2-py2.py3-none-any.whl                                                                                                    [52/1822]
   Processing ./argon2-cffi-21.1.0.tar.gz                                                                                                                      
   Installing build dependencies ... error                                                                                                                   
   ERROR: Command errored out with exit status 1:                                                                                                            
   command: /usr/bin/python3 /tmp/pip-standalone-pip-ei8r27o3/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-ko_t3hl1/
   overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links file:///run/build/poetry-deps -- 'setuptools>=40.6.0' whee
   l 'cffi>=1.0'                                                                                                                                               
      cwd: None                                                                                                                                            
   Complete output (4 lines):                                                                                                                                
   Looking in links: file:///run/build/poetry-deps                                                                                                             Processing ./setuptools-49.1.3.zip                                                                                                                          ERROR: Could not find a version that satisfies the requirement wheel (from versions: none)                                                                  ERROR: No matching distribution found for wheel                                                                                                             ----------------------------------------                                                                                                                  WARNING: Discarding file:///run/build/poetry-deps/argon2-cffi-21.1.0.tar.gz. Command errored out with exit status 1: /usr/bin/python3 /tmp/pip-standalone-pip-ei8r27o3/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-ko_t3hl1/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links file:///run/build/poetry-deps -- 'setuptools>=40.6.0' wheel 'cffi>=1.0' Check the logs for full command output.                                                                                                                                                          ERROR: Could not find a version that satisfies the requirement argon2-cffi (from versions: 21.1.0)                                                          
   ERROR: No matching distribution found for argon2-cffi

8. I stack wheel on the sources of the generated file, and re-run, fail:

   Processing ./bandit-1.7.0.tar.gz
   ERROR: Command errored out with exit status 1:
    command: /usr/bin/python3 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-expggchu/bandit_a758d9fd8b6e43bb8da3cca342ec93e3/setup.py'"'"'; __file__='"'"'/tmp/pip-install-expggchu/bandit_a758d9fd8b6e43bb8da3cca342ec93e3/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-dcw5l4ue
        cwd: /tmp/pip-install-expggchu/bandit_a758d9fd8b6e43bb8da3cca342ec93e3/
   Complete output (37 lines):
   WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fcfc19b67f0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pbr/
   WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fcfc19b6820>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pbr/
   WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fcfc19480d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pbr/
   WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fcfc19483a0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pbr/
   WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fcfc1948550>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/pbr/
   ERROR: Could not find a version that satisfies the requirement pbr>=2.0.0 (from versions: none)
   ERROR: No matching distribution found for pbr>=2.0.0
   Traceback (most recent call last):
     File "/usr/lib/python3.8/site-packages/setuptools/installer.py", line 119, in fetch_build_egg
       subprocess.check_call(cmd)
     File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
       raise CalledProcessError(retcode, cmd)
   subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpxk4u0ttd', '--quiet', 'pbr>=2.0.0']' returned non-zero exit status 1.
   
   The above exception was the direct cause of the following exception:
   
   Traceback (most recent call last):
     File "<string>", line 1, in <module>
     File "/tmp/pip-install-expggchu/bandit_a758d9fd8b6e43bb8da3cca342ec93e3/setup.py", line 16, in <module>
       setuptools.setup(
     File "/usr/lib/python3.8/site-packages/setuptools/__init__.py", line 152, in setup
       _install_setup_requires(attrs)
     File "/usr/lib/python3.8/site-packages/setuptools/__init__.py", line 147, in _install_setup_requires
       dist.fetch_build_eggs(dist.setup_requires)
     File "/usr/lib/python3.8/site-packages/setuptools/dist.py", line 686, in fetch_build_eggs
       resolved_dists = pkg_resources.working_set.resolve(
     File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 766, in resolve
       dist = best[req.key] = env.best_match(
     File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1051, in best_match
       return self.obtain(req, installer)
     File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1063, in obtain
       return installer(requirement)
     File "/usr/lib/python3.8/site-packages/setuptools/dist.py", line 745, in fetch_build_egg
       return fetch_build_egg(self, req)
     File "/usr/lib/python3.8/site-packages/setuptools/installer.py", line 121, in fetch_build_egg
       raise DistutilsError(str(e)) from e
   distutils.errors.DistutilsError: Command '['/usr/bin/python3', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpxk4u0ttd', '--quiet', 'pbr>=2.0.0']' returned non-zero exit status 1.
   ----------------------------------------
   WARNING: Discarding file:///run/build/poetry-deps/bandit-1.7.0.tar.gz. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
   ERROR: Could not find a version that satisfies the requirement bandit (from versions: 1.7.0)
   ERROR: No matching distribution found for bandit

How do I deal with that ? It seems it requires internet for pbr. 8.1) remove bandit and continue ? not a good idea to remove depdencies of the projet I'm building 8.2) add build-options/build-args: -share=network, no I can't, its not available on the bulding CI/CD pipeline of flathub. 8.3) re-run flatpak-poetry-generator.py poetry-of-bandit.lock (or his requirements) and go through al the process I have been through again ? so I can get a "generated" file of bandit. No way. 8.4) let's try to add a source for pbr.

9. I go for 8.4, but it's already there in the generated file. I'm stuck.

I really hope that through this issue some high level flathub maintainers will understand how painful it is to build some python app. Just because there is a 'security' reason which seems related with how CI/CD pipeline is designed and cannot have internet connection and not with flatpak/bubblwarp concept. If there was a connection we would just have to run 'pip3 install theinitialpackage` and don't go through all this indeep process..., because it clearly works locally, but not on CI/CD flathub pipeline

So maybe there is a lot of python apps on flathub., I would definitely not be the guy that build and maintain one of them in this manner. There must be another way of doing this.

[lpyparmentier]

If i'm wrong in my process, please tell me! Maybe there is another way that I don't see

[qnixsynapse]

I am having the same problem with pyQt5 and poetry. If the "source" isn't provided in the manifest for the nested dependencies (pyqt5 needs sip for example), the build will fail. Thing is python-pip-generator script cannot create a json output with source for setuptools (because it is already installed?). And because of it, I am currently struck..

This is way more painful than maintaining an rpm for example.

Edit : This is what I get by running python flatpak-pip-generator.py setuptools

{
    "name": "python3-setuptools",
    "buildsystem": "simple",
    "build-commands": [],
    "modules": []
}

[TingPing]

@akarshanbiswas setuptools is blacklisted because its already there: https://github.com/flatpak/flatpak-builder-tools/blob/master/pip/flatpak-pip-generator#L297

But maybe it can be made smarter, like if setuptools is explicitly in the list given, ignore the blacklist.

[zocker-160]

I just went through a similar packaging hell as @lpyparmentier did when trying to package a few of my own PyQt5 based applications until I came across this (and other Flatpak and PythonQT related issues that are almost 4 years old a this point).

The packaging experience is sadly so bad, that I gave up on it.

The fact that no PyQt5 extension exists like it does for openJDK just blows my mind.

[barthalion]

The fact that no PyQt5 extension exists like it does for openJDK just blows my mind.

Feel free to submit it. Voicing your frustrations here is not going to solve any problem.

[cbm755]

A few comments above mention PyQt5: this is now much better with https://github.com/flathub/com.riverbankcomputing.PyQt.BaseApp (at least it works well for me and maintainers where very responsive to a few issues that initially cropped up)