Add possibility to upload "transactional"

[claasd]

Hi,

I would like a feature where all binary packages and the receipt are make available on the server only if all uploads have succeeded and are finished. If one upload fails, nothing should be available on the server. Maybe a feature line --all --as_transaction.

We are currently building a lot of different configurations of one package on our CI System. When all builds are finished (currently 4, Release/Debug and x64/x86, this will get more when more compilers need to be supported), I upload them to our conan server using the --all flag.

The problem is that the receipt is publish first, and then the binary packages are uploaded. This currently takes up to 2 minutes, and sometimes an upload fails. Then, other CI builds are using the newly uploaded component and building them "on the fly", because the binary package is not available on the server. This is not desired, as some of the base components take some time to build.

Best, Claas

[lasote]

Maybe you can control the conan upload return code and delete the recipe/packages if the command failed. When you say:

Then, other CI builds are using the newly uploaded component and building them "on the fly", because the binary package is not available on the server.

That means that the upload is triggering some other builds in your CI automatically? If true, How do you do it?

Thanks

[claasd]

Hi,

the upload does not trigger other packages, but we have some thousand jobs running on our CI platfrom, serveral (up to 50) of them running at once. Every CI build artifacts is pushed on the conan server, adding the build number to the version, something like `1.8.0-alpha-b2345'. We use the conan semver mechanism to always get the newest build of the component.

As it takes up to two minutes to publish everything to the conan server, some other builds are already using the newest package receipt without getting the binary package.

If the original proposition is not easily possible or not wanted, my idea would be to publish to a "private" channel, and then use the conan copy when everything is finished, hoping that this command is faster or somehow transactional.

Best, Claas

[lasote]

I understand, the problem is that the copy command works only locally, so then you would need to upload the copied packages with the same problem again :disappointed:

@memsharded any idea?

[memsharded]

Seems a legit use case, but very complicated too. Let's think a bit about it.

[claasd]

Maybe it is possible to put some kind of .conan_ignore file in the path on the server e.g. .conan_server/data/MyPackage/0.1/demo/testing/.conan_ignore before uploading the receipt and the binaries and remove the ./conan_ignore once the upload of all is completed. Configure the DiskSearchManager to ignore those folders.

This could also be implemented as single commands:

conan begin_transaction MyPackage/0.1/demo/testing
conan upload MyPackage/0.1/demo/testing --all
conan finish_transaction MyPackage/0.1/demo/testing

or

conan rollback_transaction MyPackage/0.1/demo/testing

The problem would be to delete folders where something went wrong ...

[lasote]

Another problem is how to know if all the binaries has been uploaded. Could be 2 or more processes uploading binaries to the recipe, but each process doesn't know about the rest, so who decide if the upload is done?

[claasd]

But this could be handled with having single commands. Call conan begin_transaction MyPackage/0.1/demo/testing Let the CI server upload all packages, it does not matter if the use several processes. When all are finished, call either conan finish_transaction MyPackage/0.1/demo/testingor conan rollback_transaction MyPackage/0.1/demo/testing

You would give the responsibility of deciding when a transaction is finished to the user. I agree that this is highly advanced use, and the problem of unfinished transactions must also be tackled ...

Maybe

conan begin_transaction MyPackage/0.1/demo/testing --timeout=3600 # timeout is seconds

I am willing to put work into this feature, but I need some feedback/idea bouncing how this could be achieved.

[rbierbasz]

I think that quick, atomic move operation for remote server can solve the problem. One can upload packages from CI builders to ci channel and then "promote" them to stable:

conan move MyPackage/0.1/demo/ci demo/stable -r conan_server

The move operation can be useful also for local cache.

Regards, Romek

[claasd]

I agree, moving or copying directly on the remote would solve the problem. Artifactory can do this, but only between two conan repositories.

[michaelmaguire]

We are looking for a solution to the problem of atomic upload of coherent builds as well.

Note that for us its important that the upload support not just a transaction for all binary flavours of a given package, but also a transaction of several different packages that may have needed to be rebuilt as a result of a change to a specified package (important to prevent One Definition Rule violations, etc).

So we need something like:

conan begin_transaction some_transaction_name
conan upload MyPackageA/0.6/demo/testing --all
conan upload MyPackageBThatDependsOnA/0.2/demo/testing --all
conan upload MyPackageCThatDependsOnA/0.7/demo/testing --all
conan finish_transaction some_transaction_name

I've seen @memsharded suggest in other threads that creating separate conan repos in artifactory for each build might be a solution to this, but at the moment our company's Artifactory maintainers don't know how to grant permission to create repos on the fly to our team's build process without granting it full Administrative privileges over Artifactory, which for a large company is not acceptable.

If we go for a move operation, it too would need to support a begin_transaction, end_transaction concept.

[claasd]

Hi,

we solved it this way:

• We have a conan-ci repository in artifactory, where only ci has access.
• We build and upload all packages to this ci repo.
• once all packages have been created and uploaded successfully, we use artifactory to move the package to a different repository, where all users have access. using the JFrog CLI (calling jfrog rt move CI_REPO/user/packageName/version/channel TARGET_REPO

This artifactory move is AFAIK atomar.

[michaelmaguire]

Thanks @claasd, but if 3 packages B,C,D need to be rebuilt because we've changed package A they all depend on, how do you atomically move all 4 of those packages into TARGET_REPO from your ci repo?

[claasd]

The you would need to use fileSpecs, where you move several items at once: https://www.jfrog.com/confluence/display/CLI/CLI+for+JFrog+Artifactory#CLIforJFrogArtifactory-UsingFileSpecs

https://www.jfrog.com/confluence/display/CLI/CLI+for+JFrog+Artifactory#CLIforJFrogArtifactory-MovingFiles

[michaelmaguire]

@claasd thanks -- this could work for us!

Two questions:

1. Do you know whether access to the jfrog move and filespec commands are supported with Artifactory non-Administrator privileges?
2. Do you know of any examples of how to use the jfrog move commands from Jenkins Groovy? My attempts to Google for it only seem to show examples of uploading or downloading https://www.jfrog.com/confluence/display/RTF5X/Working+With+Pipeline+Jobs+in+Jenkins -- If I just shell out to a command line, I imagine it will be non-trivial to get the credentials to work properly. Right now credentials are handled for us by our Jenkins Administrators and made available to us via the Artifactory.server object initialized in our Groovy.

[michaelmaguire]

Partially answering my own questions above,

Hello Michael,

Thank you for your reply. With regards to your repository creation request as you mentioned this process requires admin privileges, currently, only Artifactory admin users can create repositories you may want to use this REST API to create the repository via the CI server with an admin user as part of the build.    We hope this answer your question. Please feel free to contact us for any further assistance or inquiry.

Best regards, JFrog Support

So the dynamic conan repo creation mechanism isn't feasible in our organization.

I have replied asking jFrog Artifactory support whether non-admins can move files with filespec using the REST API.

[claasd]

Hi,

1. Do you know whether access to the jfrog move and filespec commands are supported with Artifactory non-Administrator privileges?

Yes, they are. You need create and maybe overwrite permissions on one repo and delete permissions on the other.

2. Do you know of any examples of how to use the jfrog move commands from Jenkins Groovy? My attempts to Google for it only seem to show examples of uploading or downloading You cannot use the Jenkins-Artifactory plugin for this. You have two options: use the JFROG-CLI , or use the Artifactory API directly, for example with the [Jenkins HTTP-Request Plugin] (https://github.com/jenkinsci/http-request-plugin)

We use both, here is an example script for using JFrog CLI to login using Jenkins credentials and move a conan package.

withCredentials([credentialsId: 'artifactory-cred', passwordVariable: 'cred_pass', usernameVariable: 'cred_user')]) {
            sh("jfrog rt config --interactive=false --url=${artifactoryUrl} " +
                    "--user=\"${cred_user}\" --password=\"${cred_pass}\" --enc-password=true")
}
sh("jfrog rt move ${srcRepo}/${user}/${packageName}/${version}/${channel} ${targetRepo}")

[danimtb]

Hi! I have read the comments above and to me, there are two issues being discussed here:

1. The fact of uploading packages, even from different libraries, and having an atomic promotion of all of them to a new repo.

I this case, I think the approach taken by @claasd is the most reasonable one, use the JFrog CLI to perform any management of the packages like promotion or movement. Also having a separated repo for this uploaded packages and not promoting them to a "stable" repo seems the right thing to do and it is something fairly common when using Artifactory for development.

Considering the answers above and the possible solutions I think this issue can be considered as solved.

2. The need of having atomic uploads of recipe + metadata files AND packages + metadata files.

As the Conan mode is decoupled (you could upload just recipes but no packages), I don't see a way of having a unique transaction of all files as the current protocol updloads files one by one. Having commands like the ones proposed in https://github.com/conan-io/conan/issues/871#issuecomment-273195986 seem like a manual solution that could lead to more errors or "blocked uploads". In any case, such a solution should be automatic and managed by the Conan client.

I have been working on having more consistent uploads and consumption of packages (#4662) although this only mitigates the issue on having "non-transactional" package uploads (package + metadata files).

Anyway, I think the proposed solution also fixes this issue and there is not much Conan client can do apart from managing the upload/search/install in the most consistent way.

[villytiger]

1. The fact of uploading packages, even from different libraries, and having an atomic promotion of all of them to a new repo.

I this case, I think the approach taken by @claasd is the most reasonable one, use the JFrog CLI to perform any management of the packages like promotion or movement. Also having a separated repo for this uploaded packages and not promoting them to a "stable" repo seems the right thing to do and it is something fairly common when using Artifactory for development.

Considering the answers above and the possible solutions I think this issue can be considered as solved.

This solution works for Artifactory Pro users only. Copy and move commands are not available in Artifactory CE for C/C++. I think it would be much better to support move command in Conan client.

https://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API#ArtifactoryRESTAPI-MoveItem

[danimtb]

@villytiger I understand your issue but as you know Artifactory CE for C/C++ can be used for free, therefore it has a limited set of features. However, the "Move" button for promoting packages from one repository to another is available in the UI of Artifactory CE.

I haven't checked if it is available through the JFrog CLI but if not, we cannot embed the functionalities of Jfrog CLI inside the Conan client as this would mean coupling the client with Artifactory and Conan can also work for repositories in Bintray that don't have such feature.

---

Considering the resolution of the issue, I think the ways to solve this are explained above and the only improvement I see is documenting this in a how-to guide or similar in docs.conan.io

[villytiger]

I assume JFrog CLI just uses REST API. I can confirm that it can't move items with Artifactory CE for C/C++.

jfrog rt move conan/user/pkgname conan-test
[Info] Searching items...
[Info] Found 1 artifact.
[Info] Moving artifact: conan/user/pkgname/ to: conan-test/user/
[Error] Artifactory response: 400 Bad Request
{
  "errors": [
    {
      "status": 400,
      "message": "This REST API is available only in Artifactory Pro (see: jfrog.com/artifactory/features). If you are already running Artifactory
 Pro please make sure your server is activated with a valid license key.\n"
    }
  ]
}
[Error] Failed Moving 1 artifacts.
{
  "status": "failure",
  "totals": {
    "success": 0,
    "failure": 1
  }
}

Moving files in web UI is not possible for every CI build. I understand that Conan must not be coupled with Artifactory that way. But if Conan doesn't support moving remote files using its client, it means that Conan can be reliably used only with Artifactory Pro in cases such as described in this topic. Can this feature be implemented in terms of Conan server API?

[danimtb]

Yes, so it is only available in the API of the Artifactory Pro version. It cannot be implemented in Conan server as there are no repositories in Conan server. in case you'd like to change the user/channel in the reference you could use copy or alias commands in the Conan client.

[villytiger]

It cannot be implemented in Conan server as there are no repositories in Conan server.

Actually I don't need moving between repositories. I'd like to change a package reference within one repository.

in case you'd like to change the user/channel in the reference you could use copy or alias commands in the Conan client.

I do so now. But it isn't atomic during upload.

[schwaerz]

But... will conan alias be atomic? Is that ensured?

[memsharded]

The conan alias command has nothing to do with conan upload transactions. It is a different recipe, just the recipe that acts as a symlink or pointer to another one. In that sense, it allows to decouple a bit the upload of heavy things from thier usage.

However, plase note that conan alias command does not longer exist in Conan 2.0. While some alias functionality remains in Conan 2.0, it is mostly for some compatibility with Conan 1.X, but it will no longer be a recommended mechanism for versioning. At the moment it is not even documented.

The correct mechanism for heavy uploads, and any kind of upload from builds in general is to upload to a different repository rather than the "used" one in production or by developers and CI. Only once everything is in the server, including most likely the builds from different platforms/compilers/versions, then a copy can be done from that repo the the "develop" repo, and if that is done via APIs, it can be way more atomic than the upload from the client.

[schwaerz]

@memsharded Thanks for the fast answer. :) You wrote 'way more atomic' compared to the upload. This sounds like copy is not 100% atomic, right? Are you planning to support 100% atomic operations so that we can reference to always using the latest version in CI without facing race conditions from time to when where some package cannot be downloaded because it currently is being upload (or copied) on the other end?

[memsharded]

You wrote 'way more atomic' compared to the upload. This sounds like copy is not 100% atomic, right? Are you planning to support 100% atomic operations so that we can reference to always using the latest version in CI without facing race conditions from time to when where some package cannot be downloaded because it currently is being upload (or copied) on the other end?

Depending on the server, as atomicity must be supported server side, running a "buildInfo" based promotion in Artifactory, for example, it can be atomic. Running your own API call, not sure how it works for bulk operations. But in any case, the time window is so small compared with the upload that it should be negligible in practice, like promoting a package can take less than one second compared to a several minutes upload from the client. This is something that we don't have much control over it, if you can request it via support tickets to JFrog support, that would be useful feedback for the Artifactory team to prioritize.

[schwaerz]

Thanks, we'll do that. :)

[schwaerz]

Just realized that you wrote

Running your own API call, not sure how it works for bulk operations.

I suppose there's no support for this in Conan. Conan copy just seems to create a local copy of a package. Anything planned there?

[memsharded]

I meant server-side API calls. Artifactory for example has a rich API to do all kind of operations over packages directly in the server. So the copy repo-to-repo in the server side is almost instantaneous because of file storage de-duplication.

[schwaerz]

Thanks, I did some implementation using the Artifactory REST API. Seems to work so far. Still feels kinda like a hack, though.

[memsharded]

Closing in favor of https://github.com/conan-io/conan/issues/16703, that will be used to centralize future 2.X investigations, please track that ticket instead, thanks!