[投票] 增加提交前的依赖静态检查

[VOID001]

问题类型 / Type of issues

• 其它 / other

针对某些包的较多依赖不在官方仓库，需要手动填写 lilac.yaml 指定 非官方依赖的情况，因为涉及人工依赖解析，有一定配置出错的风险，目前的 lilac 不支持在本地对依赖进行检查，需要等待开发机上打包作业运行时才能知道配置是否有问题.　而这个静态依赖检查我认为可以在 commit 提交之前检查完毕，因而提议增加提交前的静态依赖检查

Vote 👍 for agree and 👎 for disagree please

-->

[VOID001]

我的想法是将这个功能集成到 pre-commit 内，在 pre-commit 的时候就对依赖进行静态检查，能够解决开发者因失误遗漏掉不在官方源的依赖包

[yan12125]

以前寫過一個從PKGBUILD求出不在官方源依賴的code

https://gitlab.com/yan12125/package-builder/blob/master/package_builder/__init__.py#L180

用的是pyalpm跟XCPF。這段code還有處理多個build_prefix的部份，如果要移植到pre-commit，得改一下。

[VOID001]

I will try to write a static package dependency check tool and test if first

[farseerfc]

現在 pre-commit 需要能脫離 Arch 環境在 Travis CI 上獨立運行。目前檢查官方源包衝突的方法是在編譯機上導出一個 json 在 Travis CI 上下載，感覺這裏可以加上包依賴關係然後做檢查。

[VOID001]

我看了下 pre-commit 如果想要使用这里的 API 完成依赖检查，需要用到 extract_srcinfo list_packages, 目前看来 extract_srcinfo 是没有对 带有 version 的 dependency 进行处理的，而且拿到的是一个 raw string, 不适合直接使用，可能考虑增加一个 load_pkg_from_srcinfo 函数，将 srcinfo parse 为结构化的数据，判断依赖的主要逻辑已经写好．目前缺少

1. 判断带有版本的依赖
2. 判断 .so 文件依赖
3. architecture specific 依赖
4. provides 的处理
5. split package 的处理

standalone 版本: https://gist.github.com/VOID001/b385311d013872fa33028adc028d70dc 目前因为需要 load 每一个 pkg 的 lilac.py 依赖 lilaclib, 对于 standalone 版本可以通过对 lilac.py 里的 import 处理后再 load 的方式去掉依赖，对于 integrated 的版本，需要确认是否有 parse lilac.py 的 API

[yan12125]

👍 👍 👍 👍

判断 .so 文件依赖

這和一般的provides判斷方式應該相同？

[VOID001]

+1 +1 +1 +1

判断 .so 文件依赖

這和一般的provides判斷方式應該相同？

根据 PKGBUILD(5) If the dependency name appears to be a library (ends with .so), makepkg will try to find a binary that depends on the library in the built package and append the needed by the binary. Appending the version yourself disables automatic detection.

另外，我忘记考虑 provides 的情况了, 感谢提醒 :smiley:

[yan12125]

If the dependency name appears to be a library (ends with .so), makepkg will try to find a binary that depends on the library in the built package and append the needed by the binary. Appending the version yourself disables automatic detection.

原來還有這個機制。我看ffmpeg寫provides=(libavcodec.so=58-64 ...)，以為.so依賴都是手動維護的(

[yan12125]

看了一下makepkg裡.so依賴的實作，需要打好的包裡的ELF檔案來決定依賴的.so版本。這個在靜態檢查的階段應該是無法實現。

Keyword: find_libdepends() in /usr/bin/makepkg

[VOID001]

目前处理了 provides, version 已经能够初步使用 一个问题是 @farseerfc pre-commit 有没有办法或者是否允许运行 pre_build hook? 按照目前的打包模式，有些包不存在 PKGBUILD 需要 pre_build 后才能 check, 还有的包 PKGBUILD out-of-date 需要 pre_build 后才能得到 latest PKGBUILD

[VOID001]

这是目前运行出来的一个全量的 package dep check 结果

fcitx5-chinese-addons-git: Cause an exception: join() argument must be str or bytes, not 'tuple'                                                                                              
gnome-perl: Cause an exception: join() argument must be str or bytes, not 'tuple'
kcm-fcitx5-git: Cause an exception: join() argument must be str or bytes, not 'tuple'
lib32-ffmpeg: Failed dependency check because the following dependency not satisfied: ['lib32-gsm', 'lib32-lame', 'lib32-libass', 'lib32-libbluray', 'lib32-opencore-amr', 'lib32-openjpeg', 'lib32-openjpeg2', 'lib32-schroedinger', 'lib32-xvidcore', 'lib32-libx264', 'hardening-wrapper']
libbonoboui: Cause an exception: join() argument must be str or bytes, not 'tuple'
libgnomeui: Cause an exception: join() argument must be str or bytes, not 'tuple'
libui: Failed dependency check because the following dependency not satisfied: ['gtk3>100']
makehuman-hg: Failed dependency check because the following dependency not satisfied: ['python2-pyqt4']                                                                                       
mingw-w64-zlib: Failed dependency check because the following dependency not satisfied: ['mingw-w64-crt', 'mingw-w64-gcc']                                                                    
octave-statistics: Failed dependency check because the following dependency not satisfied: ['octave-io>=1.0.18']                                                                              
pushbullet-cli: Failed dependency check because the following dependency not satisfied: ['python-pushbullet.py']                                                                              
pypy-stm-hg: Cause an exception: join() argument must be str or bytes, not 'tuple'
qterminal-git: Failed dependency check because the following dependency not satisfied: ['lxqt-build-tools-git']                                                                               
shutter: Cause an exception: join() argument must be str or bytes, not 'tuple'
wecase: Failed dependency check because the following dependency not satisfied: ['pyqt>=4']
zfs-dkms: Cause an exception: join() argument must be str or bytes, not 'dict'

关于 Exception: 是因为没有处理 split package 导致的可以忽略, 麻烦大家 check 下有没有误报

[VOID001]

这是目前运行出来的一个全量的 package dep check 结果

fcitx5-chinese-addons-git: Cause an exception: join() argument must be str or bytes, not 'tuple'                                                                                              
gnome-perl: Cause an exception: join() argument must be str or bytes, not 'tuple'
kcm-fcitx5-git: Cause an exception: join() argument must be str or bytes, not 'tuple'
lib32-ffmpeg: Failed dependency check because the following dependency not satisfied: ['lib32-gsm', 'lib32-lame', 'lib32-libass', 'lib32-libbluray', 'lib32-opencore-amr', 'lib32-openjpeg', 'lib32-openjpeg2', 'lib32-schroedinger', 'lib32-xvidcore', 'lib32-libx264', 'hardening-wrapper']
libbonoboui: Cause an exception: join() argument must be str or bytes, not 'tuple'
libgnomeui: Cause an exception: join() argument must be str or bytes, not 'tuple'
libui: Failed dependency check because the following dependency not satisfied: ['gtk3>100']
makehuman-hg: Failed dependency check because the following dependency not satisfied: ['python2-pyqt4']                                                                                       
mingw-w64-zlib: Failed dependency check because the following dependency not satisfied: ['mingw-w64-crt', 'mingw-w64-gcc']                                                                    
octave-statistics: Failed dependency check because the following dependency not satisfied: ['octave-io>=1.0.18']                                                                              
pushbullet-cli: Failed dependency check because the following dependency not satisfied: ['python-pushbullet.py']                                                                              
pypy-stm-hg: Cause an exception: join() argument must be str or bytes, not 'tuple'
qterminal-git: Failed dependency check because the following dependency not satisfied: ['lxqt-build-tools-git']                                                                               
shutter: Cause an exception: join() argument must be str or bytes, not 'tuple'
wecase: Failed dependency check because the following dependency not satisfied: ['pyqt>=4']
zfs-dkms: Cause an exception: join() argument must be str or bytes, not 'dict'

关于 Exception: 是因为没有处理 split package 导致的可以忽略, 麻烦大家 check 下有没有误报

lib32-ffmpeg: Failed dependency check because the following dependency not satisfied: ['lib32-gsm', 'lib32-lame', 'lib32-libass', 'lib32-libbluray', 'lib32-opencore-amr', 'lib32-openjpeg', 'lib32-openjpeg2', 'lib32-schroedinger', 'lib32-xvidcore', 'lib32-libx264', 'hardening-wrapper']
makehuman-hg: Failed dependency check because the following dependency not satisfied: ['python2-pyqt4']
mingw-w64-zlib: Failed dependency check because the following dependency not satisfied: ['mingw-w64-crt', 'mingw-w64-gcc']
octave-statistics: Failed dependency check because the following dependency not satisfied: ['octave-io>=1.0.18']
pushbullet-cli: Failed dependency check because the following dependency not satisfied: ['python-pushbullet.py']
qterminal-git: Failed dependency check because the following dependency not satisfied: ['lxqt-build-tools-git']
wecase: Failed dependency check because the following dependency not satisfied: ['pyqt>=4']

去掉 tuple Exception 的版本

[VOID001]

目前可处理 split pacakge 了, 我将 code cleanup 后贴个 gist

[VOID001]

https://gist.github.com/VOID001/4038eb3e70a0019fa0b592d2a56996e8 目前可以处理

• package with version string
• split package
• provides
• architecture specified dependency

目前不支持

• versioned dependency from cn source repo
• run pre_build hook before check [DISMISS]

关于如何移植到 travis CI 上 大概需要做两部分事情

1. 我后续会把 lilac 的依赖完全移除 。目前因为 需要解析 lilac.py 文件, 需要 load lilac.py 文件，后续我将通过替换文件内字符串的方式去掉这个依赖
2. 需要评估下是在 CI 上建立一个最小的能让 pyalpm 运行的环境，还是将 pyalpm 的依赖去掉，使用目前的方式，将 repo 信息存为结构化数据同步到 CI. @farseerfc 这个要请教下 fc 老师

[yan12125]

之前在Ubuntu 16.04 compile過pacman，沒啥問題，想來把pyalpm弄上CI應該也不難(

[VOID001]

https://github.com/archlinuxcn/repo/tree/feature/static-dep-check 移植开始 ww 现在需要考虑是要移植完整的 pacman 还是只移植　libalpm + pyalpm

[VOID001]

https://travis-ci.org/archlinuxcn/repo/builds/473283447 pyalpm & pycman 都扔到 CI 里了，不过目前看来 pre-commit 脚本还有点问题，以前是判断是否能够 import pycman 来决定是在 travis 机器上还是在本地，现在这样会出问题

[a-wing]

travis-ci 不能用 docker 吗 ？（

[yuyichao]

可以 https://github.com/mikkeloscar/arch-travis

[VOID001]

我 port 了 pacman (libalpm) 到 debian, 并且做了一些必要的设置，目前可以使用 libalpm & pycman 了

[VOID001]

Done #979

[VOID001]

可以 https://github.com/mikkeloscar/arch-travis

这个 overhead 过大，pre-commit 还是越快执行完毕越好

[VOID001]

/poke # This is a reminder for this issue

[yan12125]

Here's something interesting, which may help this work: https://github.com/jelly/pacman-deb.

PS. jelly is a TU & Developer.

[yan12125]

I tried a Docker-based approach. It works but runs for almost 3 minutes https://travis-ci.org/yan12125/repo/builds/621561412.

[lilydjwg]

What if you cache the docker image and reuse the same container everytime?

[yan12125]

I tried caching it but the build time is not less :/

[lilydjwg]

Well, it called too many times of bash....

[yan12125]

Yep. On my machine, running pre-commit of master (e7cd6f30af49fe858b7b9b5b0e03283e3109ed3a) takes 79.37 seconds, and with static-dep-check (yan12125/repo@607402813d), pre-commit takes 111.48 seconds.

[yan12125]

Another try with Docker on GitLab: https://gitlab.com/yan12125/repo/-/jobs/372472446. Without a Ubuntu base image like builds on Travis CI, it's 10 seconds faster. I think it is the fatest possible container-based approach.

Update: I removed the testing GitLab repo.

[yan12125]

With the recent removal of gconf, there are 20+ additional packages that fail the test [1]. There are other failures caused by removal of official packages (e.g., python2-xlib of acestream-engine). If this feature is going to be merged, a way is needed to filter dependency errors not caused by Arch Linux CN contributors.

[1] https://travis-ci.org/yan12125/repo/builds/632308580

[yan12125]

需要等待开发机上打包作业运行时才能知道配置是否有问题.

I propose to run lilac upon git pushes: https://github.com/orgs/archlinuxcn/teams/repo-maintainers/discussions/15