Firstyear / obs-service-cargo

OBS Source Service and utilities for Rust software packaging
Mozilla Public License 2.0
16 stars 9 forks source link

Vendoring may fail on first attempt #12

Closed glaubitz closed 3 years ago

glaubitz commented 3 years ago

I have been using obs-service-cargo_vendor for the first time this week and I ran into the problem that vendoring didn't work correctly on the first attempt with cargo being unable to find the vendored sources during build. I was able to fix the problem by just running osc service ra a second time.

Trying to build the package after the first call to osc service ra fails with cargo being unable to find the vendored sources:

glaubitz@suse-laptop:~/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust> osc service ra
Cloning into '/home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust/aws-sdk-rust'...
b0049d2f04e70614c78f210f86e423facc271726
Already up to date.
b0049d2f04e70614c78f210f86e423facc271726
Compressed aws-sdk-rust-0.0.6+git+b0049d2f.tar to aws-sdk-rust-0.0.6+git+b0049d2f.tar.xz
INFO:obs-service-cargo_vendor:Running OBS Source Service: obs-service-cargo_vendor
INFO:obs-service-cargo_vendor:Searching for Cargo.toml in aws-sdk-rust/sdk
INFO:obs-service-cargo_vendor:Current work dir /home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust
aws-sdk-rust/sdk/Cargo.toml
INFO:obs-service-cargo_vendor:Detected Rust app directory: aws-sdk-rust/sdk
INFO:obs-service-cargo_vendor:Vendoring Cargo.toml deps to aws-sdk-rust/sdk/vendor
INFO:obs-service-cargo_vendor:Running cargo vendor in directory: aws-sdk-rust/sdk
warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
package:   /home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust/aws-sdk-rust/sdk/examples/s3-helloworld/Cargo.toml
workspace: /home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust/aws-sdk-rust/sdk/Cargo.toml
    Updating crates.io index
    Updating git repository `https://github.com/rcoh/sigv4`
   Vendoring aho-corasick v0.7.18 (/home/glaubitz/.cargo/registry/src/github.com-1ecc6299db9ec823/aho-corasick-0.7.18) to vendor/aho-corasick
   Vendoring ansi_term v0.12.1 (/home/glaubitz/.cargo/registry/src/github.com-1ecc6299db9ec823/ansi_term-0.12.1) to vendor/ansi_term
   Vendoring assert-json-diff v1.1.0 (/home/glaubitz/.cargo/registry/src/github.com-1ecc6299db9ec823/assert-json-diff-1.1.0) to vendor/assert-json-diff
   Vendoring async-stream v0.3.2 (/home/glaubitz/.cargo/registry/src/github.com-1ecc6299db9ec823/async-stream-0.3.2) to vendor/async-stream
   Vendoring async-stream-impl v0.3.2 (/home/glaubitz/.cargo/registry/src/github.com-1ecc6299db9ec823/async-stream-impl-0.3.2) to vendor/async-stream-impl
   Vendoring atty v0.2.14 (/home/glaubitz/.cargo/registry/src/github.com-1ecc6299db9ec823/atty-0.2.14) to vendor/atty
   Vendoring autocfg v1.0.1 (/home/glaubitz/.cargo/registry/src/github.com-1ecc6299db9ec823/autocfg-1.0.1) to vendor/autocfg
(...)
INFO:obs-service-cargo_vendor:
Your spec file should be modified per the following example:

---BEGIN---
%global rustflags '-Clink-arg=-Wl,-z,relro,-z,now'

Source1:    vendor.tar.xz
Source2:    cargo_config

%prep
%setup -qa1
mkdir .cargo
cp %{SOURCE2} .cargo/config

%build
RUSTFLAGS=%{rustflags} cargo build --release

%install
RUSTFLAGS=%{rustflags} cargo install --root=%{buildroot}%{_prefix} --path .
---END---

WARNING: To avoid cargo install rebuilding the binary in the install stage
         all environment variables must be the same as in the build stage.

glaubitz@suse-laptop:~/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust> osc addremove
A    cargo_config
A    aws-sdk-rust-0.0.6+git+b0049d2f.obscpio
A    aws-sdk-rust-0.0.6+git+b0049d2f.tar.xz
glaubitz@suse-laptop:~/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust> ls
aws-sdk-rust                             aws-sdk-rust-0.0.6+git+b0049d2f.tar.xz  aws-sdk-rust.obsinfo  cargo_config  _servicedata
aws-sdk-rust-0.0.6+git+b0049d2f.obscpio  aws-sdk-rust.changes                    aws-sdk-rust.spec     _service      vendor.tar.xz
glaubitz@suse-laptop:~/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust> osc build --no-verify --clean
Building aws-sdk-rust.spec for openSUSE_Tumbleweed/x86_64
Getting buildconfig from server and store to /home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust/.osc/_buildconfig-openSUSE_Tumbleweed-x86_64
Getting buildinfo from server and store to /home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust/.osc/_buildinfo-openSUSE_Tumbleweed-x86_64.xml
Updating cache of required packages
0.0% cache miss. 206/206 dependencies cached.

Skipping verification of package signatures
Writing build configuration
Running build
logging output to /var/tmp/build-root/openSUSE_Tumbleweed-x86_64/.build.log...
[    0s] Memory limit set to 32561246KB
[    0s] Using BUILD_ROOT=/var/tmp/build-root/openSUSE_Tumbleweed-x86_64
[    0s] Using BUILD_ARCH=x86_64:i686:i586:i486:i386
[    0s] 
(...)
[   19s] Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.XnnvZk
[   19s] + umask 022
[   19s] + cd /home/abuild/rpmbuild/BUILD
[   19s] + cd /home/abuild/rpmbuild/BUILD
[   19s] + rm -rf aws-sdk-rust-0.0.6+git+b0049d2f
[   19s] + /usr/bin/xz -dc /home/abuild/rpmbuild/SOURCES/aws-sdk-rust-0.0.6+git+b0049d2f.tar.xz
[   19s] + /usr/bin/tar -xof -
[   19s] + STATUS=0
[   19s] + '[' 0 -ne 0 ']'
[   19s] + cd aws-sdk-rust-0.0.6+git+b0049d2f
[   19s] + /usr/bin/xz -dc /home/abuild/rpmbuild/SOURCES/vendor.tar.xz
[   19s] + /usr/bin/tar -xof -
[   20s] + STATUS=0
[   20s] + '[' 0 -ne 0 ']'
[   20s] + /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w .
[   20s] + mkdir .cargo
[   20s] + cp /home/abuild/rpmbuild/SOURCES/cargo_config .cargo/config
[   20s] + RPM_EC=0
[   20s] ++ jobs -p
[   20s] + exit 0
[   20s] Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.61G2aF
[   20s] + umask 022
[   20s] + cd /home/abuild/rpmbuild/BUILD
[   20s] + /usr/bin/rm -rf /home/abuild/rpmbuild/BUILDROOT/aws-sdk-rust-0.0.6+git+b0049d2f-0.x86_64
[   20s] ++ dirname /home/abuild/rpmbuild/BUILDROOT/aws-sdk-rust-0.0.6+git+b0049d2f-0.x86_64
[   20s] + /usr/bin/mkdir -p /home/abuild/rpmbuild/BUILDROOT
[   20s] + /usr/bin/mkdir /home/abuild/rpmbuild/BUILDROOT/aws-sdk-rust-0.0.6+git+b0049d2f-0.x86_64
[   20s] + cd aws-sdk-rust-0.0.6+git+b0049d2f
[   20s] + cd sdk
[   20s] + RUSTFLAGS='-Clink-arg=-Wl,-z,relro,-z,now -C debuginfo=2'
[   20s] + cargo build --release
[   20s] warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
[   20s] package:   /home/abuild/rpmbuild/BUILD/aws-sdk-rust-0.0.6+git+b0049d2f/sdk/examples/s3-helloworld/Cargo.toml
[   20s] workspace: /home/abuild/rpmbuild/BUILD/aws-sdk-rust-0.0.6+git+b0049d2f/sdk/Cargo.toml
[   20s] error: failed to get `aws-sigv4` as a dependency of package `aws-sig-auth v0.1.0 (/home/abuild/rpmbuild/BUILD/aws-sdk-rust-0.0.6+git+b0049d2f/sdk/aws-sig-auth)`
[   20s] 
[   20s] Caused by:
[   20s]   failed to load source for dependency `aws-sigv4`
[   20s] 
[   20s] Caused by:
[   20s]   Unable to update https://github.com/rcoh/sigv4?rev=1854c5f5728c80b0970fcca86c2431bf288f6997
[   20s] 
[   20s] Caused by:
[   20s]   the source https://github.com/rcoh/sigv4 requires a lock file to be present first before it can be
[   20s]   used against vendored source code
[   20s] 
[   20s]   remove the source replacement configuration, generate a lock file, and then
[   20s]   restore the source replacement configuration to continue the build
[   20s] error: Bad exit status from /var/tmp/rpm-tmp.61G2aF (%build)
[   20s] 
[   20s] 
[   20s] RPM build errors:
[   20s]     Bad exit status from /var/tmp/rpm-tmp.61G2aF (%build)
[   20s] 
[   20s] suse-laptop failed "build aws-sdk-rust.spec" at Wed Jun  2 11:10:30 UTC 2021.
[   20s] 

The buildroot was: /var/tmp/build-root/openSUSE_Tumbleweed-x86_64
glaubitz@suse-laptop:~/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust>

Just re-running osc service ra fixes the problem:

glaubitz@suse-laptop:~/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust> osc service ra
Already up to date.
b0049d2f04e70614c78f210f86e423facc271726
Compressed aws-sdk-rust-0.0.6+git+b0049d2f.tar to aws-sdk-rust-0.0.6+git+b0049d2f.tar.xz
INFO:obs-service-cargo_vendor:Running OBS Source Service: obs-service-cargo_vendor
INFO:obs-service-cargo_vendor:Searching for Cargo.toml in aws-sdk-rust/sdk
INFO:obs-service-cargo_vendor:Current work dir /home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust
aws-sdk-rust/sdk/Cargo.toml
INFO:obs-service-cargo_vendor:Detected Rust app directory: aws-sdk-rust/sdk
INFO:obs-service-cargo_vendor:Vendoring Cargo.toml deps to aws-sdk-rust/sdk/vendor
INFO:obs-service-cargo_vendor:Running cargo vendor in directory: aws-sdk-rust/sdk
warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
package:   /home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust/aws-sdk-rust/sdk/examples/s3-helloworld/Cargo.toml
workspace: /home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust/aws-sdk-rust/sdk/Cargo.toml
   Vendoring aho-corasick v0.7.18 (/home/glaubitz/.cargo/registry/src/github.com-1ecc6299db9ec823/aho-corasick-0.7.18) to /home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust/aws-sdk-rust/sdk/vendor/aho-corasick
   Vendoring ansi_term v0.12.1 (/home/glaubitz/.cargo/registry/src/github.com-1ecc6299db9ec823/ansi_term-0.12.1) to /home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust/aws-sdk-rust/sdk/vendor/ansi_term
   Vendoring assert-json-diff v1.1.0 (/home/glaubitz/.cargo/registry/src/github.com-1ecc6299db9ec823/assert-json-diff-1.1.0) to /home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust/aws-sdk-rust/sdk/vendor/assert-json-diff
   Vendoring async-stream v0.3.2 (/home/glaubitz/.cargo/registry/src/github.com-1ecc6299db9ec823/async-stream-0.3.2) to /home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust/aws-sdk-rust/sdk/vendor/async-stream
   Vendoring async-stream-impl v0.3.2 (/home/glaubitz/.cargo/registry/src/github.com-1ecc6299db9ec823/async-stream-impl-0.3.2) to /home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust/aws-sdk-rust/sdk/vendor/async-stream-impl
   Vendoring atty v0.2.14 (/home/glaubitz/.cargo/registry/src/github.com-1ecc6299db9ec823/atty-0.2.14) to /home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust/aws-sdk-rust/sdk/vendor/atty
   Vendoring autocfg v1.0.1 (/home/glaubitz/.cargo/registry/src/github.com-1ecc6299db9ec823/autocfg-1.0.1) to /home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust/aws-sdk-rust/sdk/vendor/autocfg
(...)
   Vendoring winapi-util v0.1.5 (/home/glaubitz/.cargo/registry/src/github.com-1ecc6299db9ec823/winapi-util-0.1.5) to /home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust/aws-sdk-rust/sdk/vendor/winapi-util
   Vendoring winapi-x86_64-pc-windows-gnu v0.4.0 (/home/glaubitz/.cargo/registry/src/github.com-1ecc6299db9ec823/winapi-x86_64-pc-windows-gnu-0.4.0) to /home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust/aws-sdk-rust/sdk/vendor/winapi-x86_64-pc-windows-gnu
   Vendoring xmlparser v0.13.3 (/home/glaubitz/.cargo/registry/src/github.com-1ecc6299db9ec823/xmlparser-0.13.3) to /home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust/aws-sdk-rust/sdk/vendor/xmlparser
   Vendoring zeroize v1.3.0 (/home/glaubitz/.cargo/registry/src/github.com-1ecc6299db9ec823/zeroize-1.3.0) to /home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust/aws-sdk-rust/sdk/vendor/zeroize
To use vendored sources, add this to your .cargo/config.toml for this project:

INFO:obs-service-cargo_vendor:
Your spec file should be modified per the following example:

---BEGIN---
%global rustflags '-Clink-arg=-Wl,-z,relro,-z,now'

Source1:    vendor.tar.xz
Source2:    cargo_config

%prep
%setup -qa1
mkdir .cargo
cp %{SOURCE2} .cargo/config

%build
RUSTFLAGS=%{rustflags} cargo build --release

%install
RUSTFLAGS=%{rustflags} cargo install --root=%{buildroot}%{_prefix} --path .
---END---

WARNING: To avoid cargo install rebuilding the binary in the install stage
         all environment variables must be the same as in the build stage.

glaubitz@suse-laptop:~/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust> osc build --no-verify --clean
Building aws-sdk-rust.spec for openSUSE_Tumbleweed/x86_64
Getting buildconfig from server and store to /home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust/.osc/_buildconfig-openSUSE_Tumbleweed-x86_64
Getting buildinfo from server and store to /home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust/.osc/_buildinfo-openSUSE_Tumbleweed-x86_64.xml
Updating cache of required packages
0.0% cache miss. 206/206 dependencies cached.

Skipping verification of package signatures
Writing build configuration
Running build
logging output to /var/tmp/build-root/openSUSE_Tumbleweed-x86_64/.build.log...
[    0s] Memory limit set to 32561246KB
[    0s] Using BUILD_ROOT=/var/tmp/build-root/openSUSE_Tumbleweed-x86_64
[    0s] Using BUILD_ARCH=x86_64:i686:i586:i486:i386
[    0s] 
[    0s] 
[    0s] suse-laptop started "build aws-sdk-rust.spec" at Wed Jun  2 11:12:51 UTC 2021.
[    0s] 
[    0s] 
[    0s] processing recipe /home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust/aws-sdk-rust.spec ...
[    0s] running changelog2spec --target rpm --file /home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust/aws-sdk-rust.spec
[    0s] init_buildsystem --configdir /usr/lib/build/configs --cachedir /var/cache/build --clean --rpmlist /tmp/rpmlist.ewyjhlz0 /home/glaubitz/suse/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust/aws-sdk-rust.spec ...
[    1s] cycle: rpm-config-SUSE -> rpm
[    1s]   breaking dependency rpm-config-SUSE -> rpm
[    1s] [1/33] preinstalling filesystem...
[    1s] [2/33] preinstalling permissions...
[    1s] [3/33] preinstalling glibc...
[    2s] [4/33] preinstalling fillup...
[    2s] [5/33] preinstalling libacl1...
[    2s] [6/33] preinstalling libattr1...
[    2s] [7/33] preinstalling libbz2-1...
[    2s] [8/33] preinstalling libcap2...
[    3s] [9/33] preinstalling libgcc_s1...
[    3s] [10/33] preinstalling libgpg-error0...
(...)
[   21s]    Compiling autocfg v1.0.1
[   21s]    Compiling libc v0.2.95
[   21s]    Compiling cfg-if v1.0.0
[   21s]    Compiling log v0.4.14
[   21s]    Compiling once_cell v1.7.2
[   21s]    Compiling memchr v2.4.0
[   21s]    Compiling itoa v0.4.7
[   21s]    Compiling bytes v1.0.1
[   21s]    Compiling pin-project-lite v0.2.6
[   21s]    Compiling lazy_static v1.4.0
[   21s]    Compiling smallvec v1.6.1
[   21s]    Compiling futures-core v0.3.15
[   21s]    Compiling scopeguard v1.1.0
[   21s]    Compiling proc-macro-hack v0.5.19
[   21s]    Compiling fnv v1.0.7
(...)
Firstyear commented 3 years ago

Hey mate, glad to hear your using cargo-vendor,

I don't think this is a problem with cargo-vendor specifically - it sounds like the culprit is the lack of the lock file.

Let's imagine your _service file is similar to the example https://en.opensuse.org/Packaging_Rust_Software

<services>
  <service mode="disabled" name="obs_scm">
  <service mode="disabled" name="tar" />
  <service mode="disabled" name="recompress">
  <service mode="disabled" name="set_version"/>
  <service name="cargo_audit" mode="disabled">
  <service name="cargo_vendor" mode="disabled">
</services>

Let's focus on the order of these modules.

When you first run osc service ra, you get a clone of the source into projectname. This is then compressed into projectname.tar.xz. This initial checkout does NOT have a Cargo.lock file.

Then later in the process, you run cargo_vendor. Since this generates the Cargo.lock, it now creates packagename/Cargo.lock. But remember, this doesn't cause the the projectname.tar.xz to be updated, which still is MISSING the Cargo.lock.

Then you do your build - it fails, because you can't generate a Cargo.lock in --offline.

So you re-run the osc service. What happens is you already hae a checkout of projectname and it now also contains the Cargo.lock file that was generated by the previous executions cargo_vendor call. This is re-compressed to projectname.tar.xz now containing Cargo.lock, so your subsequent build works.

Honestly, I think this is an artifact of how the module order is processed, and Ithink it's a potentially good reason to update and change the examples and how we present them.

I think if we change the example _service to:

<services>
  <service mode="disabled" name="obs_scm">

  <service name="cargo_audit" mode="disabled">
  <service name="cargo_vendor" mode="disabled">

  <service mode="disabled" name="tar" />
  <service mode="disabled" name="recompress">
  <service mode="disabled" name="set_version"/>
</services>

This should mean the Cargo.lock is created before you call tar.

A serious issue though, is if we cleanup the projectname/vendor folder - if we aren't, then we are effectively double vendoring here in this process, so I think we should check that.

Additionally, if this application is a binary, it's a recomendadiotn from rust as a language that it should commit it's Cargo.lock, so you could also ask upstream to consider commiting that, at least into their release tags.

Does that help?

glaubitz commented 3 years ago

Thanks a lot for the very detailed explanation so I could learn something in the process ;).

I think if we change the example _service to:

<services>
  <service mode="disabled" name="obs_scm">

  <service name="cargo_audit" mode="disabled">
  <service name="cargo_vendor" mode="disabled">

  <service mode="disabled" name="tar" />
  <service mode="disabled" name="recompress">
  <service mode="disabled" name="set_version"/>
</services>

This should mean the Cargo.lock is created before you call tar.

Thanks, this fixes the problem.

A serious issue though, is if we cleanup the projectname/vendor folder - if we aren't, then we are effectively double vendoring here in this process, so I think we should check that.

Hmm, unless I'm missing something, it actually just puts the vendor folder below the sdk directory where it should be.

Additionally, if this application is a binary, it's a recomendadiotn from rust as a language that it should commit it's Cargo.lock, so you could also ask upstream to consider commiting that, at least into their release tags.

I think it's currently not intended to be a binary application.

Does that help?

Yep, a lot. Thanks!

Firstyear commented 3 years ago

Thanks a lot for the very detailed explanation so I could learn something in the process ;).

I think if we change the example _service to:

<services>
  <service mode="disabled" name="obs_scm">

  <service name="cargo_audit" mode="disabled">
  <service name="cargo_vendor" mode="disabled">

  <service mode="disabled" name="tar" />
  <service mode="disabled" name="recompress">
  <service mode="disabled" name="set_version"/>
</services>

This should mean the Cargo.lock is created before you call tar.

Thanks, this fixes the problem.

A serious issue though, is if we cleanup the projectname/vendor folder - if we aren't, then we are effectively double vendoring here in this process, so I think we should check that.

Hmm, unless I'm missing something, it actually just puts the vendor folder below the sdk directory where it should be.

Yes, but then it stores that vendor folder inside the source tar and a duplicate copy in vendor.tar.xz. So either we need to:

The question becomes about the workflows people will use to create these packages. If we want the vendor in the primary tar, we are really limiting ourselves to the obs_scm git module in almost all cases. In the later if we clean up sdk/vendor, then we can support alternate behaviours like tar downloads, unpack and then vendoring etc.

Additionally, if this application is a binary, it's a recomendadiotn from rust as a language that it should commit it's Cargo.lock, so you could also ask upstream to consider commiting that, at least into their release tags.

I think it's currently not intended to be a binary application.

If it builds a binary that you install, then it's a binary application and should have Cargo.lock

If it's a library, then you shouldn't package it because the sources should come from crates.io for binaries that want to consume it.

Does that help?