search

openSUSE / obs-service-go_modules

OBS Source Service to download, verify, and vendor Go module dependency sources
GNU General Public License v2.0
19 stars 18 forks source link

should provide an --exclude parameter to avoid shipping test-data and other binary blobs #44

Open dirkmueller opened 7 months ago

dirkmueller commented 7 months ago

in the case of "vcpkg", we seem to include files that should not be needed for building, and could raise concerns. we should be supporting an exclude parameter to strip those files from the vendor.tar.gz tarball:


tar tvf vendor.tar.zst|grep -i -E '(lzm|xz)'
-rw-r--r-- 0/0             981 1970-01-01 01:00 vendor/vcpkg/test-data/no-status/installed/vcpkg/info/liblzma_5.2.3-2_x64-windows.list
-rw-r--r-- 0/0               0 1970-01-01 01:00 vendor/vcpkg/test-data/no-status/installed/x64-windows/bin/lzma.dll
-rw-r--r-- 0/0               0 1970-01-01 01:00 vendor/vcpkg/test-data/no-status/installed/x64-windows/lib/lzma.lib
-rw-r--r-- 0/0            1012 1970-01-01 01:00 vendor/vcpkg/test-data/normalized/installed/vcpkg/info/liblzma_5.2.3-2_x64-windows-static.list
-rw-r--r-- 0/0             981 1970-01-01 01:00 vendor/vcpkg/test-data/normalized/installed/vcpkg/info/liblzma_5.2.3-2_x64-windows.list
-rw-r--r-- 0/0               0 1970-01-01 01:00 vendor/vcpkg/test-data/normalized/installed/x64-windows/bin/lzma.dll
-rw-r--r-- 0/0               0 1970-01-01 01:00 vendor/vcpkg/test-data/normalized/installed/x64-windows/debug/bin/lzma.dll
-rw-r--r-- 0/0               0 1970-01-01 01:00 vendor/vcpkg/test-data/normalized/installed/x64-windows/debug/lib/lzma.lib
-rw-r--r-- 0/0               0 1970-01-01 01:00 vendor/vcpkg/test-data/normalized/installed/x64-windows/lib/lzma.lib
-rw-r--r-- 0/0               0 1970-01-01 01:00 vendor/vcpkg/test-data/normalized/installed/x64-windows-static/debug/lib/lzma.lib
-rw-r--r-- 0/0               0 1970-01-01 01:00 vendor/vcpkg/test-data/normalized/installed/x64-windows-static/lib/lzma.lib````
jfkw commented 7 months ago

Agreed. I'd like to investigate their layout and the mechanism. I'm not familliar with vcpkg other than the Microsoft C++ package manager. Do you have a link in OBS or upstream Github?

Normally test-only Go packages are not imported by go mod vendor. The heuristic is to vendor only packages which are transitively imported by the Go module. This sometimes trips up packages which have a subdirectory with e.g. C code and no Go code reachable from main(). In those cases the subdirectory is omitted from the generated vendor/.

jfkw commented 7 months ago

As we plan an --exclude parameter, here is a sample of unique file extensions present in the vendor/ directory of Hugo, a Go application with a large number of Go dependencies, including linking to C libraries:

$ find ./hugo/vendor -type f | perl -ne 'print $1 if m/\.([^.\/]+)$/' | sort -u
abnf
badjson
badxml
bash
bicep
c
cfg
conf
contributebot
cpp
der
dockerignore
editorconfig
gitattributes
gitignore
gitmodules
go
h
hpp
ini
jpg
json
json5
libyaml
lock
mailmap
markdown
md
pem
png
proto
ps1
py
replay
s
sh
template
tif
tmpl
toml
txt
vim
xml
yaml
yml
zsh

Given the number of file extensions one might enounter, we would want any --exclude parameter be capable of multi-invocations and/or wildcards. I'd like to align with conventions used by other OBS Source Services.

hennevogel commented 7 months ago

JFYI: For ruby we ship an RPM macro/script to do this in the spec file...

https://build.opensuse.org/projects/devel:languages:ruby/packages/ruby-common/files/gem_build_cleanup?expand=1