"Failed to check manifest file" when Cargo.toml is in a subdir

[marcosbc]

We are trying the new version of obs-service-cargo that was recently released, re-written in Rust, but we are unable to get it to work when the Cargo.toml file is located in a subfolder.

According to the README it should be possible by using the --cargotoml option, but it is not working for us.

For example, when trying to build flux, where the Cargo.toml file is located in the libflux folder, it fails with the following errors:

$ /usr/lib/obs/service/cargo_vendor --src flux-master.tar.gz --outdir $(pwd) --cargotoml libflux/Cargo.toml
2023-11-08T16:57:43.862963Z  INFO cargo_vendor: 🎢 Starting OBS Service Cargo Vendor.
2023-11-08T16:57:43.864152Z  INFO obs_service_cargo::utils: 🍿 Vendoring for src 'flux-master.tar.gz'
2023-11-08T16:57:43.986692Z ERROR obs_service_cargo::cli: err=kind: cargo vendor process failed
reason: failed to check manifest file at path libflux/Cargo.toml
2023-11-08T16:57:43.999147Z ERROR cargo_vendor: kind: cargo vendor process failed
reason: kind: cargo vendor process failed
reason: failed to check manifest file at path libflux/Cargo.toml
Error: kind: cargo vendor process failed
reason: kind: cargo vendor process failed
reason: failed to check manifest file at path libflux/Cargo.toml

You can easily reproduce this issue by downloading the previous repository's master branch in a tarball, and executing the command shown above.

[pallaswept]

I'm having the same experience. I have a previously working package, here is the relevant section of the _service file:

  <service name="cargo_vendor" mode="disabled">
     <param name="update">false</param>
     <param name="srcdir">mouse-actions</param>
  </service>
  <service name="cargo_vendor" mode="disabled">
     <param name="tag">gui</param>
     <param name="update">false</param>
     <param name="srcdir">mouse-actions</param>
     <param name="cargotoml">config-editor/src-tauri/Cargo.toml</param>
  </service>

This used to work, as can be seen from the directory listing of the existing checkout, there were two separate vendor archives created (the last two files here):

ls
-rw-r--r-- 1 pallaswept pallaswept      100 Oct 29 00:26 cargo_config
-rw-r--r-- 1 pallaswept pallaswept      100 Oct 29 00:26 cargo_config_gui
-rw-r--r-- 1 pallaswept pallaswept     3628 Oct 29 00:26 mouse-actions.changes
-rw-r--r-- 1 pallaswept pallaswept      477 Oct 29 00:26 mouse-actions-gui.desktop
-rw-r--r-- 1 pallaswept pallaswept      236 Oct 29 00:26 mouse-actions.service
-rw-r--r-- 1 pallaswept pallaswept     4230 Oct 29 00:26 mouse-actions.spec
-rw-r--r-- 1 pallaswept pallaswept   416659 Oct 29 00:26 mouse-actions.tar.zst
-rw-r--r-- 1 pallaswept pallaswept 70371494 Oct 29 00:28 node_modules.tar.gz
-rw-r--r-- 1 pallaswept pallaswept     1617 Oct 29 00:26 _service
-rw-r--r-- 1 pallaswept pallaswept      482 Oct 29 00:28 tauri.conf.nobundle.diff
-rw-r--r-- 1 pallaswept pallaswept      807 Oct 29 00:28 update.sh
-rw-r--r-- 1 pallaswept pallaswept 62945713 Oct 29 00:28 vendor-gui.tar.zst
-rw-r--r-- 1 pallaswept pallaswept 15646367 Oct 29 00:29 vendor.tar.zst

I took a copy of that previously working checkout and re-ran the service, now it fails:

2023-11-16T20:48:26.337033Z  INFO cargo_vendor: 🎢 Starting OBS Service Cargo Vendor.
2023-11-16T20:48:26.337080Z  INFO obs_service_cargo::utils: 🍿 Vendoring for src 'mouse-actions'
2023-11-16T20:48:26.340345Z  INFO obs_service_cargo::utils: 📗 Project does not use a workspace!
2023-11-16T20:48:26.340385Z  WARN obs_service_cargo::utils: 😥 Disabled update of dependencies. You should enable this for security updates.
2023-11-16T20:48:26.363431Z  INFO obs_service_cargo::audit: 🎉 Cargo audit passed!
2023-11-16T20:48:27.790941Z  INFO obs_service_cargo::vendor: 📦 Archiving vendored dependencies...
2023-11-16T20:48:39.219638Z  INFO obs_service_cargo::cli: 🥳 ✨ Successfull ran OBS Service Cargo Vendor ✨
2023-11-16T20:48:39.283434Z  INFO cargo_vendor: 🎢 Starting OBS Service Cargo Vendor.
2023-11-16T20:48:39.283485Z  INFO obs_service_cargo::utils: 🍿 Vendoring for src 'mouse-actions'
2023-11-16T20:48:39.286742Z ERROR obs_service_cargo::cli: err=kind: cargo vendor process failed
reason: failed to check manifest file at path config-editor/src-tauri/Cargo.toml
2023-11-16T20:48:39.287753Z ERROR cargo_vendor: kind: cargo vendor process failed
reason: kind: cargo vendor process failed
reason: failed to check manifest file at path config-editor/src-tauri/Cargo.toml
Error: kind: cargo vendor process failed
reason: kind: cargo vendor process failed
reason: failed to check manifest file at path config-editor/src-tauri/Cargo.toml
Aborting: service call failed:  /usr/lib/obs/service/cargo_vendor --tag gui --update false --srcdir mouse-actions --cargotoml config-editor/src-tauri/Cargo.toml --outdir /tmp/mouse-actions/tmpd2wo1gw_.cargo_vendor.service

[pallaswept]

I should make a small correction to the title and description of the fault though - it works fine when the file is in a subdir, the problem only occurs when using the cargotoml parameter. It also appears that the tag parameter is being ignored.

This first vendoring works, and the second, using cargotoml param, fails:

  <service name="cargo_vendor" mode="disabled">
     <param name="tag">daemon</param>
     <param name="update">false</param>
     <param name="srcdir">coolercontrol/coolercontrold</param>
  </service>
  <service name="cargo_vendor" mode="disabled">
     <param name="tag">gui</param>
     <param name="update">false</param>
     <param name="srcdir">coolercontrol</param>
     <param name="cargotoml">coolercontrol-ui/src-tauri/Cargo.toml</param>

Both of these work, note the second one is the exact same toml file being read and vendored, but there is only a single vendor.tar.zst file created, the tag is not applied to the output file name.

  <service name="cargo_vendor" mode="disabled">
     <param name="tag">daemon</param>
     <param name="update">false</param>
     <param name="srcdir">coolercontrol/coolercontrold</param>
  </service>
  <service name="cargo_vendor" mode="disabled">
     <param name="tag">gui</param>
     <param name="update">false</param>
     <param name="srcdir">coolercontrol/coolercontrol-ui/src-tauri</param>

[pallaswept]

Additionally. the new version is not generating a cargo_config file.

I tested using the old version from https://software.opensuse.org//download.html?project=openSUSE%3AFactory&package=obs-service-cargo_vendor and it both creates the cargo_config* files and respects the tag parameter when naming all files it creates.

I also tried it on the copy of the checkout of the known-good copy and it worked as before.

[Firstyear]

Can you actually send me the link to a repo on OBS that's failing?

[pallaswept]

Can you actually send me the link to a repo on OBS that's failing?

It's on my private instance (well, it happens on my local drive before it even reaches there, because we run these services locally of course) and if I re-run the service and break the working one online that would really suck, but this is the one: https://build.opensuse.org/package/show/home:pallaswept/mouse-actions

Zypper has installed the latest one from here (0.8.4) on my local machine, I don't know if the public OBS server is using that one or still on the old versions?

rpm -qf cargo_vendor obs-service-cargo-0.8.4-1.1.x86_64

When I use the ye olde one linked above (0.4.5), it's all fine.

[Firstyear]

That service file looks really weird, why do you have cargo vendor twice?

[pallaswept]

That service file looks really weird, why do you have cargo vendor twice?

Software has a front and a back end with conflicting versions of dependencies. It's two rust apps and an NPM app in one application, it was a nightmare to package.

[Firstyear]

Anyway, I'll have a look now

[pallaswept]

I actually came across this today doing an app just like it, two rust apps and an NPM app in one application, and funnily enough, came across the same thing where the backend service had a dependency on version X of one of the rust packages and the front end had a dependency on version Y.... I've not finished packaging that yet and it's not anywhere online, but fortunately (unfortunately, more like it, lol), I have this one that's just like it.

[pallaswept]

You don't need all that fanciness tho, just try using the cargotoml and/or the tag params with the current version, neither work.

Edit: to be clear, neither work, independently. This should probably be two separate issues but it just kinda unfolded like this.....

[Firstyear]

We removed tag, because we didn't see anyone using it and we made it possible to vendor multiple cargo.tomls to one.

[Firstyear]

We also made cargo_config built-in to the vendor.tar as well.

[pallaswept]

tag is still in the current version's help file and the service file:

./cargo_vendor_new --help
OBS Source Service to vendor all crates.io and dependencies for Rust project locally

Usage: cargo_vendor_new [OPTIONS] --src <SRC> --outdir <OUTDIR>

Options:
      --src <SRC>
          Where to find sources. Source is either a directory or a source tarball AND cannot be both.

          [aliases: srctar, srcdir]

      --compression <COMPRESSION>
          What compression algorithm to use.

          [default: zst]
          [possible values: gz, xz, zst]

      --tag <TAG>
          Tag some files for multi-vendor and multi-cargo_config projects

(I have it renamed to *_new because I have the old version in its place right now so I can do stuff, but that's cargo_vendor version 0.8.4)

[pallaswept]

We also made cargo_config built-in to the vendor.tar as well.

Interesting, so that's one less file to track, seems cool, I assume the idea is to extract the cargo config to the necessary location from the vendor tar? And so the reason I wasn't seeing it created is because it was being created but it was actually inside the vendor.tar.zst?

[Firstyear]

Yep, exactly.

I'm gonna make --tag throw an error if used as well. I'm currently testing your project to check if it's working.

[pallaswept]

So, without tag, how are we supposed to handle projects like these ones where they have clashing dependencies?

[uncomfyhalomacro]

We also made cargo_config built-in to the vendor.tar as well.

Interesting, so that's one less file to track, seems cool, I assume the idea is to extract the cargo config to the necessary location from the vendor tar? And so the reason I wasn't seeing it created is because it was being created but it was actually inside the vendor.tar.zst?

Yeah. both cargo_config and Cargo.lock are now inside the vendor.tar.<ext> file

[uncomfyhalomacro]

I will update the README after a fix is merged. I need to investigate this issue first

[Firstyear]

So, without tag, how are we supposed to handle projects like these ones where they have clashing dependencies?

Vendor can have multiple versions of a crate, not just one. So it can have a single vendor there and it's fine.

[uncomfyhalomacro]

So, without tag, how are we supposed to handle projects like these ones where they have clashing dependencies?

Vendor can have multiple versions of a crate, not just one. So it can have a single vendor there and it's fine.

with the --sync option e.g. cargo vendor --manifest-path <Path to desired manifest> --sync <Path to additional manifests e.g. subcrates>. In that way, you can reuse one vendored tarball

[pallaswept]

So, without tag, how are we supposed to handle projects like these ones where they have clashing dependencies?

Vendor can have multiple versions of a crate, not just one. So it can have a single vendor there and it's fine.

with the --sync option e.g. cargo vendor --manifest-path <Path to desired manifest> --sync <Path to additional manifests e.g. subcrates>. In that way, you can reuse one vendored tarball

Too easy, I like it :) I assume that's supported in _service files too?

[uncomfyhalomacro]

I guess we forgot to reintroduce the logic to append and join the path to the temporary working directory when we refactored the whole thing.

That should be libflux/Cargo.toml instead of the full path since we use the working directory's full path and join that parameter.

[pallaswept]

Is that maybe this And you already did it but it's not released yet? (Sorry, it seemed familiar, but I don't know this codebase at all so I might be miles off)

[uncomfyhalomacro]

Is that maybe this And you already did it but it's not released yet? (Sorry, it seemed familiar, but I don't know this codebase at all so I might be miles off)

Oh that's a rewrite of bulk-updater. unrelated to this issue. i will have to investigate something

[Firstyear]

It looks like an issue had been introduce with multi-manifest services which prevented vendoring. I am going to submit a PR.

To make your service and spec work, the following changes are needed.

<services>
  ...
  <service name="cargo_vendor" mode="disabled">
     <param name="update">false</param>
     <param name="srcdir">mouse-actions</param>
     <param name="cargotoml">Cargo.toml</param>
     <param name="cargotoml">config-editor/src-tauri/Cargo.toml</param>
  </service>
  <service name="cargo_audit" mode="disabled">
     <param name="srcdir">mouse-actions</param>
  </service>
</services>

# CLI Cargo deps
%setup -a2 -D -n %{name} -T -q
mkdir -p .cargo
cp cargo_config .cargo/config

And remove the other source lines.

Im also looking to now change

• --tag to be an error
• An info notice to say that you can remove cargo_config now
• a way to automatically put the .cargo/config into the source tree without stomping other cargo configs that might be included in the project.

[pallaswept]

Thanks heaps you guys keep on killin it

[pallaswept]

Really sorry to bother you but a quick question: Is there any way for us to know when the public OBS instance is running this new version?

Obviously I'll need to do a minor update to a package or two, and I'll know when my instance will get it, but come to think of it I have no clue how to tell what version is live on openbuildservice.org

No rush or anything, just trying to edumacate myself a little :) Any tips appreciated.

[Firstyear]

OBS doesn't run any version of cargo_vendor, and if they do, it's not endorsed by us or supported by us. This simply comes down to the fact that OBS is a "black box" and we have no way to investigate, mitigate or resolve issues that arise inside OBS instances.

As a result the answer is "I have no idea" but the intent of this project is always to be used client side.

[pallaswept]

Perfect, thanks yet again, legend

[marcosbc]

I could get it to work with flux as well, thank you very much!