search

flatpak / flatpak-builder-tools

Various helper tools for flatpak-builder
191 stars 106 forks source link

error with cargo generator, fail to find a file #404

Closed wiiznokes closed 4 months ago

wiiznokes commented 4 months ago

flatpak-builder version

1.4.1

Linux distribution and version

fedora 40 beta

Affected flatpak-builder tool

cargo/flatpak-cargo-generator.py

flatpak-builder tool cli args

No response

Source repository URL

https://github.com/wiiznokes/fan-control

Flatpak-builder manifest URL

No response

Description

$ python3 ./flatpak-cargo-generator.py Cargo.lock -o cargo-sources.json
INFO:root:Loading packages from https://github.com/wash2/accesskit
INFO:root:Adding package accesskit from https://github.com/wash2/accesskit
INFO:root:Adding package accesskit_consumer from https://github.com/wash2/accesskit
INFO:root:Adding package accesskit_macos from https://github.com/wash2/accesskit
INFO:root:Adding package accesskit_unix from https://github.com/wash2/accesskit
INFO:root:Adding package accesskit_windows from https://github.com/wash2/accesskit
INFO:root:Adding package accesskit_winit from https://github.com/wash2/accesskit
INFO:root:Loading packages from https://github.com/jackpot51/rust-atomicwrites
INFO:root:Adding package atomicwrites from https://github.com/jackpot51/rust-atomicwrites
INFO:root:Loading packages from https://github.com/pop-os/window_clipboard
INFO:root:Adding package clipboard_macos from https://github.com/pop-os/window_clipboard
INFO:root:Adding package clipboard_wayland from https://github.com/pop-os/window_clipboard
INFO:root:Adding package clipboard_x11 from https://github.com/pop-os/window_clipboard
INFO:root:Loading packages from https://github.com/pop-os/libcosmic
INFO:root:Loading packages from https://github.com/pop-os/libcosmic
INFO:root:Loading packages from https://github.com/pop-os/cosmic-text
INFO:root:Adding package cosmic-text from https://github.com/pop-os/cosmic-text
INFO:root:Loading packages from https://github.com/pop-os/libcosmic
INFO:root:Loading packages from https://github.com/gfx-rs/wgpu
INFO:root:Adding package d3d12 from https://github.com/gfx-rs/wgpu
INFO:root:Loading packages from https://github.com/pop-os/glyphon
INFO:root:Adding package glyphon from https://github.com/pop-os/glyphon
INFO:root:Loading packages from https://github.com/pop-os/libcosmic
INFO:root:Loading packages from https://github.com/pop-os/libcosmic
INFO:root:Loading packages from https://github.com/pop-os/libcosmic
INFO:root:Loading packages from https://github.com/pop-os/libcosmic
INFO:root:Loading packages from https://github.com/pop-os/libcosmic
INFO:root:Loading packages from https://github.com/pop-os/libcosmic
INFO:root:Loading packages from https://github.com/pop-os/libcosmic
INFO:root:Loading packages from https://github.com/pop-os/libcosmic
INFO:root:Loading packages from https://github.com/pop-os/libcosmic
INFO:root:Loading packages from https://github.com/pop-os/libcosmic
INFO:root:Loading packages from https://github.com/pop-os/libcosmic
INFO:root:Loading packages from https://github.com/pop-os/libcosmic
INFO:root:Loading packages from https://github.com/pop-os/libcosmic
INFO:root:Loading packages from https://github.com/wiiznokes/lm-sensors
INFO:root:Adding package lm-sensors from https://github.com/wiiznokes/lm-sensors
INFO:root:Adding package mime from https://github.com/pop-os/window_clipboard
INFO:root:Adding package naga from https://github.com/gfx-rs/wgpu
INFO:root:Loading packages from https://github.com/pop-os/smithay-clipboard
INFO:root:Adding package smithay-clipboard from https://github.com/pop-os/smithay-clipboard
INFO:root:Loading packages from https://github.com/pop-os/softbuffer
INFO:root:Adding package softbuffer from https://github.com/pop-os/softbuffer
INFO:root:Loading packages from https://github.com/dioxuslabs/taffy
INFO:root:Adding package taffy from https://github.com/dioxuslabs/taffy
INFO:root:Adding package wgpu from https://github.com/gfx-rs/wgpu
INFO:root:Adding package wgpu-core from https://github.com/gfx-rs/wgpu
INFO:root:Adding package wgpu-hal from https://github.com/gfx-rs/wgpu
INFO:root:Adding package wgpu-types from https://github.com/gfx-rs/wgpu
INFO:root:Adding package window_clipboard from https://github.com/pop-os/window_clipboard
INFO:root:Loading packages from https://github.com/pop-os/winit
INFO:root:Adding package winit from https://github.com/pop-os/winit
Traceback (most recent call last):
  File "/home/fedasus/Documents/fan-control/./flatpak-cargo-generator.py", line 470, in <module>
    main()
  File "/home/fedasus/Documents/fan-control/./flatpak-cargo-generator.py", line 463, in main
    generated_sources = asyncio.run(generate_sources(load_toml(args.cargo_lock),
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/asyncio/base_events.py", line 685, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/fedasus/Documents/fan-control/./flatpak-cargo-generator.py", line 417, in generate_sources
    for pkg in await asyncio.gather(*pkg_coros):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fedasus/Documents/fan-control/./flatpak-cargo-generator.py", line 367, in get_package_sources
    return await get_git_package_sources(package, git_repos)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fedasus/Documents/fan-control/./flatpak-cargo-generator.py", line 304, in get_git_package_sources
    git_repo['commits'][commit] = await get_git_repo_packages(repo_url, commit)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fedasus/Documents/fan-control/./flatpak-cargo-generator.py", line 174, in get_git_repo_packages
    packages.update(await get_cargo_toml_packages(load_toml('Cargo.toml'), '.'))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fedasus/Documents/fan-control/./flatpak-cargo-generator.py", line 227, in get_cargo_toml_packages
    await get_dep_packages(root_toml, root_dir)
  File "/home/fedasus/Documents/fan-control/./flatpak-cargo-generator.py", line 216, in get_dep_packages
    await get_dep_packages(dep_toml, dep_dir, workspace)
  File "/home/fedasus/Documents/fan-control/./flatpak-cargo-generator.py", line 214, in get_dep_packages
    dep_toml = load_toml(os.path.join(dep_dir, 'Cargo.toml'))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fedasus/Documents/fan-control/./flatpak-cargo-generator.py", line 95, in load_toml
    with open(tomlfile, 'r') as f:
         ^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'iced/Cargo.toml'
zecakeh commented 4 months ago

This happens because iced is in a git submodule. There is no other way to fix that than to fetch git submodules after or while cloning a repository.

wiiznokes commented 4 months ago

So, is it me that need to fix smtg or flatpak-cargo-generator.py ? I think the solution will benefit me because i also have a git submodule i need to build in my repo

zecakeh commented 4 months ago

Ah sorry, it's flatpak-cargo-generator that needs to fetch git submodules. I am not sure if it's fine to just get the submodules automatically, we might fetch them when it's not needed. But it's also the simpler approach.

zecakeh commented 4 months ago

Can you try #406 to see if the generator works, and then if your project builds?

wiiznokes commented 4 months ago

I get this error:

python3 ./flatpak-cargo-generator.py Cargo.lock -o cargo-sources.json
INFO:root:Loading packages from https://github.com/wash2/accesskit
INFO:root:Adding package accesskit from https://github.com/wash2/accesskit
INFO:root:Adding package accesskit_consumer from https://github.com/wash2/accesskit
INFO:root:Adding package accesskit_macos from https://github.com/wash2/accesskit
INFO:root:Adding package accesskit_unix from https://github.com/wash2/accesskit
INFO:root:Adding package accesskit_windows from https://github.com/wash2/accesskit
INFO:root:Adding package accesskit_winit from https://github.com/wash2/accesskit
INFO:root:Loading packages from https://github.com/jackpot51/rust-atomicwrites
INFO:root:Adding package atomicwrites from https://github.com/jackpot51/rust-atomicwrites
INFO:root:Loading packages from https://github.com/pop-os/window_clipboard
INFO:root:Adding package clipboard_macos from https://github.com/pop-os/window_clipboard
INFO:root:Adding package clipboard_wayland from https://github.com/pop-os/window_clipboard
INFO:root:Adding package clipboard_x11 from https://github.com/pop-os/window_clipboard
INFO:root:Loading packages from https://github.com/pop-os/libcosmic
Sous-module 'examples/design-demo' (https://github.com/pop-os/cosmic-design-demo) enregistré pour le chemin 'examples/design-demo'
Sous-module 'iced' (https://github.com/pop-os/iced.git) enregistré pour le chemin 'iced'
Clonage dans '/home/fedasus/.cache/flatpak-cargo/https_github.com_pop-os_libcosmic/examples/design-demo'...
Clonage dans '/home/fedasus/.cache/flatpak-cargo/https_github.com_pop-os_libcosmic/iced'...
Chemin de sous-module 'examples/design-demo' : '493e17a0105c7523fb7ff5fd7221ec586ac9010f' extrait
Chemin de sous-module 'iced' : '1d9bb7fdff21f71c06c04e0c70087a5a8b0bc99e' extrait
INFO:root:Adding package cosmic-config from https://github.com/pop-os/libcosmic
INFO:root:Adding package cosmic-config-derive from https://github.com/pop-os/libcosmic
INFO:root:Loading packages from https://github.com/pop-os/cosmic-text
INFO:root:Adding package cosmic-text from https://github.com/pop-os/cosmic-text
INFO:root:Adding package cosmic-theme from https://github.com/pop-os/libcosmic
INFO:root:Loading packages from https://github.com/gfx-rs/wgpu
INFO:root:Adding package d3d12 from https://github.com/gfx-rs/wgpu
INFO:root:Loading packages from https://github.com/pop-os/glyphon
INFO:root:Adding package glyphon from https://github.com/pop-os/glyphon
INFO:root:Adding package iced from https://github.com/pop-os/libcosmic
INFO:root:Adding package iced_accessibility from https://github.com/pop-os/libcosmic
INFO:root:Adding package iced_core from https://github.com/pop-os/libcosmic
INFO:root:Adding package iced_futures from https://github.com/pop-os/libcosmic
INFO:root:Adding package iced_graphics from https://github.com/pop-os/libcosmic
INFO:root:Adding package iced_renderer from https://github.com/pop-os/libcosmic
INFO:root:Adding package iced_runtime from https://github.com/pop-os/libcosmic
INFO:root:Adding package iced_style from https://github.com/pop-os/libcosmic
INFO:root:Adding package iced_tiny_skia from https://github.com/pop-os/libcosmic
INFO:root:Adding package iced_wgpu from https://github.com/pop-os/libcosmic
INFO:root:Adding package iced_widget from https://github.com/pop-os/libcosmic
INFO:root:Adding package iced_winit from https://github.com/pop-os/libcosmic
INFO:root:Adding package libcosmic from https://github.com/pop-os/libcosmic
INFO:root:Loading packages from https://github.com/wiiznokes/lm-sensors
INFO:root:Adding package lm-sensors from https://github.com/wiiznokes/lm-sensors
INFO:root:Adding package mime from https://github.com/pop-os/window_clipboard
INFO:root:Adding package naga from https://github.com/gfx-rs/wgpu
INFO:root:Loading packages from https://github.com/pop-os/smithay-clipboard
INFO:root:Adding package smithay-clipboard from https://github.com/pop-os/smithay-clipboard
INFO:root:Loading packages from https://github.com/pop-os/softbuffer
INFO:root:Adding package softbuffer from https://github.com/pop-os/softbuffer
INFO:root:Loading packages from https://github.com/dioxuslabs/taffy
INFO:root:Adding package taffy from https://github.com/dioxuslabs/taffy
INFO:root:Adding package wgpu from https://github.com/gfx-rs/wgpu
INFO:root:Adding package wgpu-core from https://github.com/gfx-rs/wgpu
INFO:root:Adding package wgpu-hal from https://github.com/gfx-rs/wgpu
INFO:root:Adding package wgpu-types from https://github.com/gfx-rs/wgpu
INFO:root:Adding package window_clipboard from https://github.com/pop-os/window_clipboard
INFO:root:Loading packages from https://github.com/pop-os/winit
INFO:root:Adding package winit from https://github.com/pop-os/winit
Traceback (most recent call last):
  File "/home/fedasus/Documents/fan-control/./flatpak-cargo-generator.py", line 432, in <module>
    main()
  File "/home/fedasus/Documents/fan-control/./flatpak-cargo-generator.py", line 425, in main
    generated_sources = asyncio.run(generate_sources(load_toml(args.cargo_lock),
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/asyncio/base_events.py", line 685, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/fedasus/Documents/fan-control/./flatpak-cargo-generator.py", line 379, in generate_sources
    for pkg in await asyncio.gather(*pkg_coros):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fedasus/Documents/fan-control/./flatpak-cargo-generator.py", line 329, in get_package_sources
    return await get_git_package_sources(package, git_repos)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fedasus/Documents/fan-control/./flatpak-cargo-generator.py", line 299, in get_git_package_sources
    'contents': toml.dumps(git_pkg.normalized),
                           ^^^^^^^^^^^^^^^^^^
  File "/home/fedasus/Documents/fan-control/./flatpak-cargo-generator.py", line 170, in normalized
    update_workspace_keys(package, self.workspace)
  File "/home/fedasus/Documents/fan-control/./flatpak-cargo-generator.py", line 157, in update_workspace_keys
    update_workspace_keys(item, workspace_item)
  File "/home/fedasus/Documents/fan-control/./flatpak-cargo-generator.py", line 157, in update_workspace_keys
    update_workspace_keys(item, workspace_item)
  File "/home/fedasus/Documents/fan-control/./flatpak-cargo-generator.py", line 141, in update_workspace_keys
    if 'workspace' in item:
       ^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'bool' is not iterable
error: Recipe `flatpak-source` failed on line 130 with exit code 1
zecakeh commented 4 months ago

Alright, I believe I fixed the script in #406. At least the sources generation seems to work. All that's left to test is whether the flatpak builds.

wiiznokes commented 4 months ago

Yes, it seems to have fixed the issue. Thought, i think i will need to wait for a fix to use lfs: https://github.com/flatpak/flatpak-builder/issues/463

HEAD is now at 61a14a953d applet: Add `icon_button_from_handle` (#368)
Cloning into '/home/fedasus/Documents/fan-control/.flatpak-builder/build/fan-control-3/flatpak-cargo/git/libcosmic-61a14a9/iced'...
Submodule path 'iced': checked out '1d9bb7fdff21f71c06c04e0c70087a5a8b0bc99e'
Cloning into '/home/fedasus/Documents/fan-control/.flatpak-builder/build/fan-control-3/flatpak-cargo/git/libcosmic-61a14a9/examples/design-demo'...
Submodule path 'examples/design-demo': checked out '493e17a0105c7523fb7ff5fd7221ec586ac9010f'
git-lfs filter-process: line 1: git-lfs: command not found
fatal: the remote end hung up unexpectedly

I'm not sure where the git lfs pull need to be but maybe the script could detect lfs repo somehow, and execute this command ?

Also, can i ask you, why cargo fetch is not enough ? I think this is because it require internet, but in this case, how can i clone my submodule in my repo ? I'm currently using 

{
    "type": "shell",
    "commands": [
        "git submodule update --init hardware/libsensors"
    ]
},

but i don't think it will works.

zecakeh commented 4 months ago

If I'm not mistaken, sources like shell only run during source extraction, so after the download, when there is no more internet connection. You can't run any random command that will download stuff. It means that you can't run commands like cargo fetch, git lfs pull or git submodule update with this kind of source. That's why flatpak-builder probably needs proper handling of git-lfs for it to work.

That's why that script exists in the first place. Its goal is to convert the Cargo.lock file into a list of git and archive sources so the crates can be downloaded from git repositories and crates.io, and to generate extra sources to have the proper file tree expected by cargo after extraction, as if it had run cargo fetch.

how can i clone my submodule in my repo ?

I believe that by default flatpak loads all the submodules of the git sources, doesn't it?

wiiznokes commented 4 months ago

Okay thanks. So what to you think about the git lfs pull command then ? In the issue, they seems to have solver it this way

zecakeh commented 4 months ago

I have no idea. Maybe you could ask in the flatpak room on Matrix.

zecakeh commented 4 months ago

To be clear, you closed this because the script now works for you?

wiiznokes commented 4 months ago

Yep, sorry I should I wait to the pr to be merged maybe ? I can reopen if you want

wiiznokes commented 4 months ago

The lfs problem was solved by installing git-lfs btw

hfiguiere commented 4 months ago

Reopening, we'll close it with the PR. Thanks for testing thigs.

And installing git-lfs was what I would have suggested too. Glad to see that this is working.