search

openSUSE / obs-service-source_validator

The default source validator as used by openSUSE:Factory distribution
GNU General Public License v2.0
3 stars 28 forks source link

20-files-present-and-referenced: Check for files in *.obscpio + bonus fix #127

Closed Vogtinator closed 1 year ago

Vogtinator commented 1 year ago

.obscpio files are extracted before build and the contents can be used as sources. This is not the only use case for obscpio though: most of the time they carry the (large) source code, which is converted to a tarball by a buildtime service. We can't easily tell those cases apart, so just don't bother.

It would be possible to avoid false negatives here by checking against the cpio listing as fallback, but that might take very long for larger .obscpio archives containing source code.

AdamMajer commented 1 year ago

I'm not sure it would take a very long time, if the cpio contents are listed once and the list is stored. But it's additional feature though.

For context, 100MB cpio,

time cpio -ti < node_modules.obscpio

[.... snip ....]
yargs-parser-20.2.9.tgz
yocto-queue-0.1.0.tgz
218928 blocks

real    0m0.064s
user    0m0.007s
sys     0m0.057s
Vogtinator commented 1 year ago

I'm not sure it would take a very long time, if the cpio contents are listed once and the list is stored. But it's additional feature though.

It's about 0.4s here for libqt5-qtbase. Might be possible to do this lazily only when first needed, then there's only the overhead of possibly scanning through >50k filenames. I'll have a try, maybe it's simple enough.

AdamMajer commented 1 year ago

You can also use something like systemsmanagement:cockpit/cockpit as a test case. It also includes quite a large number of sources in cpio, so if search is not scaling, it will show up there.

Vogtinator commented 1 year ago

You can also use something like systemsmanagement:cockpit/cockpit as a test case. It also includes quite a large number of sources in cpio, so if search is not scaling, it will show up there.

Took ~2.3s. For some reason cockpit-tukit has a larger node_modules (142MiB vs. 107MiB).

AdamMajer commented 1 year ago

I tried it locally, and there is basically no difference (in runtimes) between the version that reports errors by not scanning cpio and the current version that actually scans those files lists. I guess all the grep is not that heavy!

Runtime for me is also ~2.0s

LGTM! Thanks!